Screen share 19: Spike — Windows hardware H264 through the pre-encoded path (#166) #230
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "screenshare-19-spike-hw-encode-windows"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stacked on: screenshare-17-voice-livekit-video
What this builds
Spike for Windows hardware H.264 through the SDK's pre-encoded path (plan Task 19, spec §7.2, §16 Q2). Outcome: built, never run on Windows hardware, so the
mf-encoderfeature stays off (the plan's exit). Windows ships software-only until #229 has run it.crates/fluxer-voice/src/mf_encoder.rs(Windows only, featuremf-encoder, off by default):MfH264Encoderdrives the GPU's hardware H.264 encoder MFT (NVENC/AMF/QSV viaMFTEnumEx, hardware only, sorted). NV12 in; Baseline, CBR, low latency, no B-frames, 60 s GOP out. Keyframes come fromCODECAPI_AVEncVideoForceKeyFrameand the rate fromCODECAPI_AVEncCommonMeanBitRate. The async MFT is driven by its NeedInput/HaveOutput events. Output is Annex B, and SPS/PPS go in front of keyframes when the MFT leaves them out. Alsohardware_h264_encoders()andavailable().rtc::pre_encoded(inrtc.rs, compiled for Windows with the feature and for tests on every OS):Pump+EncoderThreadfeed the encoder from pushed frames, newest first, on one thread. They followtake_keyframe_request(also for the first frame and after a size change) andtake_rate_control_request(capped by the share's ceiling, set throughupdate_screen_encoding). A target of 0 pauses encoding and keeps any keyframe already asked for. Units go toNativeVideoSource::new_encoded+capture_encoded_frame.LiveKitRoom(Windows + feature + MFT present):codec_capabilitiesreportsCodecCap { H264, hardware: true }. H.264 shares whose Encoder path is not prefer-software are published withVideoEncoderBackend::PreEncoded.EncoderStatsreports the MFT's name withhardware: true, because the pass-through reports itself as software.docs/superpowers/research/2026-11-11-screenshare-spikes.md§6: what the SDK's pass-through does (from its source), what was observed on Linux, how the encoder is built, and what is still to check on Windows.How it was tested
cargo xtask ci): 10 tests inrtc::pre_encodedwith a scripted encoder and sink. They cover: only hardware H.264 takes the path, the MFT adds H.264 to the hardware list, Annex B / parameter-set helpers, I420→NV12 with odd-edge crop, a keyframe for the first frame and on request, rate capping, pausing at 0, a new encoder on a new size, no retry of a size that failed, and the thread's frame handoff and shutdown.tests/mf_encoder.rs(Windows,--features mf-encoder,FLUXER_HW_ENCODER_TESTS=1):mft_found_and_encodes_pattern,keyframe_on_request,rate_control_within_20_percent. Not run: no Windows machine with a GPU was available. They were seen failing (compile error, module missing) and then compiling.cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice [--features mf-encoder] --all-targets -- -D warnings(DOCS_RS=1, stubCC/CXX/AR, as in spikes §3).h264_nvencBaseline units from an RTX 5090 throughnew_encoded+PreEncodedto livekit-server 1.12.0. Keyframe requests came at the first frame after publishing and 60 ms after a late viewer subscribed. Rate requests went from 1.09 to 1.5 Mbit/s in about 1 s and never went abovemax_bitrate. The viewer's FFmpeg decoder decoded every frame with rising counters. The stats nameLiveKit pre-encoded passthroughwith power-efficient false. Details in spikes §6.2.FLUXER_SCREENSHOT_TESTS=1 cargo xtask cigreen locally. On Linux, the interop testsscreen_share_pattern_arrives_for_each_encodable_codecandrepublish_with_other_codec_reaches_viewerpass against a local server (the publish path was refactored).Deviations from the plan
--features mf-encoder.rtc.rs(pre_encoded) rather than inmf_encoder.rs, so that they are unit tested on every OS.mf_encoder.rsnames no LiveKit type.windowsis added tofluxer-voiceas an optional, Windows-only dependency (the workspace's 0.62, plus theWin32_Media_MediaFoundationandWin32_System_Olefeatures). The plan's tech stack names it; no new crate is added to the lock.42e01f), and 1080p goes beyond it. FFmpeg and browsers decode it anyway; see spikes §6.1.Closes #166
🤖 Generated with Claude Code
Independent review: Screen share 19 (spike-hw-encode-windows)
Scope:
git diff 9e7acfb..origin/screenshare-19-spike-hw-encode-windows(c8a1187), read without relying on the implementer's notes.(1) Tests the plan names.
mft_found_and_encodes_pattern,keyframe_on_requestandrate_control_within_20_percentare intests/mf_encoder.rs. The file is#![cfg(all(windows, feature = "mf-encoder"))], and each test is#[ignore]and does nothing unlessFLUXER_HW_ENCODER_TESTS=1. What they assert:That matches the plan's exit criteria. The 10
rtc::pre_encodedunit tests ran in CI and pass. The three hardware tests were never run. The plan allows that for a spike: the feature stays off, the PR says so, and follow-up #229 is open.(2) Interfaces. The plan asks for four things, and all four are there:
new_encoded+capture_encoded_frame.take_keyframe_requestandtake_rate_control_request.VideoEncoderBackend::PreEncoded.CodecCap { H264, hardware: true }viawith_mft_h264→capabilities_from, only when the feature is on andavailable().No existing signature changed.
report_encodergained a private parameter.(3) Invariants. Only
fluxer-voicechanges.mf_encoder.rsnames no LiveKit or WebRTC type.pre_encodedispub(crate), and the only public items it re-exports areEncodedUnitandEncoderConfig, which contain plain data.windowsis the workspace's 0.62, optional and Windows-only.Cargo.lockgains only that edge, andcargo denyis green.(4) Review Focus. None of the items belongs to Task 19. Focus 3 (the decode advertisement) is unchanged: the MFT only adds to the encode list.
(5) CI.
FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciin the review worktree: green (all steps passed, exit 0; LiveKit interop skipped becauseFLUXER_LIVEKIT_TESTSis unset).cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice --all-targets -- -D warningson Linux, with and without--features mf-encoder(DOCS_RS=1, target-scoped stub CC/CXX/AR): clean both times.Findings
mf_encoder.rspull_output, async format change (plausible, not checkable without hardware). OnMF_E_TRANSFORM_STREAM_CHANGEthe loop sets the new output type and callsProcessOutputagain at once (continue). An asynchronous MFT may only be asked for output after aMETransformHaveOutputevent. If a vendor's MFT answers this second call withE_UNEXPECTED, thenencodefails, the pump drops the encoder and reopens it on the next frame, and this can repeat every frame (Intel's MFT is known to change format on its first output). Suggested fix: whenself.events.is_some(), returnOk(None)after taking the new type and let the nextHaveOutputevent drive the read. Please add this to #229's checklist.available()only enumerates the MFTs. If all of them refuse to open,publish_screenstill publishes a pre-encoded H.264 track and the pump returnsStep::Failedforever: the share is live but black, and the codec advert claims hardware H.264. This could happen, for example, with a vendor MFT that wants a D3D11 device manager (spikes §6.4). This does not block while the feature is off. Before turning it on (#229), either open the encoder before publishing and fall back to the SDK's software path, or haveavailable()test-open one.42e01f(level 3.1), and 1080p goes beyond it. This is recorded in spikes §6.1 but not in #229. Add "1080p/60 decodes on a strict hardware decoder (e.g. a Windows viewer with HW decode on), or the MFT's SPS level vs SDP" to #229's checklist.The
encode_asyncevent loop otherwise looks correct:NeedInputevents are counted across frames, andProcessInputis only called when one is owed.The
ProcessOutputManuallyDrophandling is also correct:pSampleandpEventsare always taken back and dropped whatever the result, so there is no leak or double release, and a sample the MFT provides is released through the takenOption.Verdict: APPROVED. No blocking findings and CI is green. The spike exits by its fallback with the feature off. Findings 1–3 should go onto follow-up #229.