Screen share 19: Spike — Windows hardware H264 through the pre-encoded path (#166) #230

Manually merged
charles merged 3 commits from screenshare-19-spike-hw-encode-windows into screenshare 2026-11-12 08:39:52 +00:00
Owner

Stacked on: screenshare-17-voice-livekit-video

What this builds

Spike for Windows hardware H.264 through the SDK's pre-encoded path (plan Task 19, spec §7.2, §16 Q2). Outcome: built, never run on Windows hardware, so the mf-encoder feature stays off (the plan's exit). Windows ships software-only until #229 has run it.

  • crates/fluxer-voice/src/mf_encoder.rs (Windows only, feature mf-encoder, off by default): MfH264Encoder drives the GPU's hardware H.264 encoder MFT (NVENC/AMF/QSV via MFTEnumEx, hardware only, sorted). NV12 in; Baseline, CBR, low latency, no B-frames, 60 s GOP out. Keyframes come from CODECAPI_AVEncVideoForceKeyFrame and the rate from CODECAPI_AVEncCommonMeanBitRate. The async MFT is driven by its NeedInput/HaveOutput events. Output is Annex B, and SPS/PPS go in front of keyframes when the MFT leaves them out. Also hardware_h264_encoders() and available().
  • rtc::pre_encoded (in rtc.rs, compiled for Windows with the feature and for tests on every OS): Pump + EncoderThread feed the encoder from pushed frames, newest first, on one thread. They follow take_keyframe_request (also for the first frame and after a size change) and take_rate_control_request (capped by the share's ceiling, set through update_screen_encoding). A target of 0 pauses encoding and keeps any keyframe already asked for. Units go to NativeVideoSource::new_encoded + capture_encoded_frame.
  • LiveKitRoom (Windows + feature + MFT present): codec_capabilities reports CodecCap { H264, hardware: true }. H.264 shares whose Encoder path is not prefer-software are published with VideoEncoderBackend::PreEncoded. EncoderStats reports the MFT's name with hardware: true, because the pass-through reports itself as software.
  • docs/superpowers/research/2026-11-11-screenshare-spikes.md §6: what the SDK's pass-through does (from its source), what was observed on Linux, how the encoder is built, and what is still to check on Windows.

How it was tested

  • Unit tests, every OS (in cargo xtask ci): 10 tests in rtc::pre_encoded with a scripted encoder and sink. They cover: only hardware H.264 takes the path, the MFT adds H.264 to the hardware list, Annex B / parameter-set helpers, I420→NV12 with odd-edge crop, a keyframe for the first frame and on request, rate capping, pausing at 0, a new encoder on a new size, no retry of a size that failed, and the thread's frame handoff and shutdown.
  • tests/mf_encoder.rs (Windows, --features mf-encoder, FLUXER_HW_ENCODER_TESTS=1): mft_found_and_encodes_pattern, keyframe_on_request, rate_control_within_20_percent. Not run: no Windows machine with a GPU was available. They were seen failing (compile error, module missing) and then compiling.
  • Windows type-check and lint on Linux, with and without the feature: cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice [--features mf-encoder] --all-targets -- -D warnings (DOCS_RS=1, stub CC/CXX/AR, as in spikes §3).
  • SDK side observed on Linux with a throwaway probe (not committed). The probe sent ffmpeg h264_nvenc Baseline units from an RTX 5090 through new_encoded + PreEncoded to livekit-server 1.12.0. Keyframe requests came at the first frame after publishing and 60 ms after a late viewer subscribed. Rate requests went from 1.09 to 1.5 Mbit/s in about 1 s and never went above max_bitrate. The viewer's FFmpeg decoder decoded every frame with rising counters. The stats name LiveKit pre-encoded passthrough with power-efficient false. Details in spikes §6.2.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green locally. On Linux, the interop tests screen_share_pattern_arrives_for_each_encodable_codec and republish_with_other_codec_reaches_viewer pass against a local server (the publish path was refactored).

Deviations from the plan

  • The exit was hit because the encoder could not be run, not because it was unstable. The feature stays off; follow-up #229 asks for the hardware run and for a Windows CI build of --features mf-encoder.
  • The rules that do not depend on the OS (the pump, the thread, the Annex B and NV12 helpers) live in rtc.rs (pre_encoded) rather than in mf_encoder.rs, so that they are unit tested on every OS. mf_encoder.rs names no LiveKit type.
  • windows is added to fluxer-voice as an optional, Windows-only dependency (the workspace's 0.62, plus the Win32_Media_MediaFoundation and Win32_System_Ole features). The plan's tech stack names it; no new crate is added to the lock.
  • To review: H.264 in the pass-through's SDP is level 3.1 (42e01f), and 1080p goes beyond it. FFmpeg and browsers decode it anyway; see spikes §6.1.

Closes #166

🤖 Generated with Claude Code

Stacked on: screenshare-17-voice-livekit-video ## What this builds Spike for Windows hardware H.264 through the SDK's pre-encoded path (plan Task 19, spec §7.2, §16 Q2). **Outcome: built, never run on Windows hardware, so the `mf-encoder` feature stays off (the plan's exit). Windows ships software-only until #229 has run it.** - `crates/fluxer-voice/src/mf_encoder.rs` (Windows only, feature `mf-encoder`, off by default): `MfH264Encoder` drives the GPU's hardware H.264 encoder MFT (NVENC/AMF/QSV via `MFTEnumEx`, hardware only, sorted). NV12 in; Baseline, CBR, low latency, no B-frames, 60 s GOP out. Keyframes come from `CODECAPI_AVEncVideoForceKeyFrame` and the rate from `CODECAPI_AVEncCommonMeanBitRate`. The async MFT is driven by its NeedInput/HaveOutput events. Output is Annex B, and SPS/PPS go in front of keyframes when the MFT leaves them out. Also `hardware_h264_encoders()` and `available()`. - `rtc::pre_encoded` (in `rtc.rs`, compiled for Windows with the feature and for tests on every OS): `Pump` + `EncoderThread` feed the encoder from pushed frames, newest first, on one thread. They follow `take_keyframe_request` (also for the first frame and after a size change) and `take_rate_control_request` (capped by the share's ceiling, set through `update_screen_encoding`). A target of 0 pauses encoding and keeps any keyframe already asked for. Units go to `NativeVideoSource::new_encoded` + `capture_encoded_frame`. - `LiveKitRoom` (Windows + feature + MFT present): `codec_capabilities` reports `CodecCap { H264, hardware: true }`. H.264 shares whose Encoder path is not prefer-software are published with `VideoEncoderBackend::PreEncoded`. `EncoderStats` reports the MFT's name with `hardware: true`, because the pass-through reports itself as software. - `docs/superpowers/research/2026-11-11-screenshare-spikes.md` §6: what the SDK's pass-through does (from its source), what was observed on Linux, how the encoder is built, and what is still to check on Windows. ## How it was tested - Unit tests, every OS (in `cargo xtask ci`): 10 tests in `rtc::pre_encoded` with a scripted encoder and sink. They cover: only hardware H.264 takes the path, the MFT adds H.264 to the hardware list, Annex B / parameter-set helpers, I420→NV12 with odd-edge crop, a keyframe for the first frame and on request, rate capping, pausing at 0, a new encoder on a new size, no retry of a size that failed, and the thread's frame handoff and shutdown. - `tests/mf_encoder.rs` (Windows, `--features mf-encoder`, `FLUXER_HW_ENCODER_TESTS=1`): `mft_found_and_encodes_pattern`, `keyframe_on_request`, `rate_control_within_20_percent`. **Not run: no Windows machine with a GPU was available.** They were seen failing (compile error, module missing) and then compiling. - Windows type-check and lint on Linux, with and without the feature: `cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice [--features mf-encoder] --all-targets -- -D warnings` (`DOCS_RS=1`, stub `CC`/`CXX`/`AR`, as in spikes §3). - SDK side observed on Linux with a throwaway probe (not committed). The probe sent ffmpeg `h264_nvenc` Baseline units from an RTX 5090 through `new_encoded` + `PreEncoded` to livekit-server 1.12.0. Keyframe requests came at the first frame after publishing and 60 ms after a late viewer subscribed. Rate requests went from 1.09 to 1.5 Mbit/s in about 1 s and never went above `max_bitrate`. The viewer's FFmpeg decoder decoded every frame with rising counters. The stats name `LiveKit pre-encoded passthrough` with power-efficient false. Details in spikes §6.2. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green locally. On Linux, the interop tests `screen_share_pattern_arrives_for_each_encodable_codec` and `republish_with_other_codec_reaches_viewer` pass against a local server (the publish path was refactored). ## Deviations from the plan - The exit was hit because the encoder could not be run, not because it was unstable. The feature stays off; follow-up #229 asks for the hardware run and for a Windows CI build of `--features mf-encoder`. - The rules that do not depend on the OS (the pump, the thread, the Annex B and NV12 helpers) live in `rtc.rs` (`pre_encoded`) rather than in `mf_encoder.rs`, so that they are unit tested on every OS. `mf_encoder.rs` names no LiveKit type. - `windows` is added to `fluxer-voice` as an optional, Windows-only dependency (the workspace's 0.62, plus the `Win32_Media_MediaFoundation` and `Win32_System_Ole` features). The plan's tech stack names it; no new crate is added to the lock. - To review: H.264 in the pass-through's SDP is level 3.1 (`42e01f`), and 1080p goes beyond it. FFmpeg and browsers decode it anyway; see spikes §6.1. Closes #166 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Refs #166

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows hardware H.264 through the SDK's pre-encoded path, behind the
off-by-default `mf-encoder` feature: `mf_encoder::MfH264Encoder` drives
the GPU's hardware encoder MFT (NV12 in, Baseline CBR low-latency out,
keyframes on request), and `rtc::pre_encoded` feeds it from pushed
frames on its own thread, following the pass-through's keyframe and
rate requests. With the feature on and an MFT present, H.264 is
reported as hardware and published with `VideoEncoderBackend::PreEncoded`.

Not run on Windows hardware (none available): the feature stays off,
findings and the SDK-side observations on Linux are in spikes §6, and
#229 tracks the run.

Refs #166

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 19 (spike-hw-encode-windows)

Scope: git diff 9e7acfb..origin/screenshare-19-spike-hw-encode-windows (c8a1187), read without relying on the implementer's notes.

(1) Tests the plan names. mft_found_and_encodes_pattern, keyframe_on_request and rate_control_within_20_percent are in tests/mf_encoder.rs. The file is #![cfg(all(windows, feature = "mf-encoder"))], and each test is #[ignore] and does nothing unless FLUXER_HW_ENCODER_TESTS=1. What they assert:

  • SPS, PPS and IDR in the first unit; Annex B; one unit per frame in capture order.
  • An IDR with SPS on exactly the requested frames and nowhere else.
  • The bitrate within 20 % at 6, 2 and 4 Mbit/s, 1080p30.

That matches the plan's exit criteria. The 10 rtc::pre_encoded unit tests ran in CI and pass. The three hardware tests were never run. The plan allows that for a spike: the feature stays off, the PR says so, and follow-up #229 is open.

(2) Interfaces. The plan asks for four things, and all four are there:

  • new_encoded + capture_encoded_frame.
  • Requests honoured through take_keyframe_request and take_rate_control_request.
  • Publishing with VideoEncoderBackend::PreEncoded.
  • CodecCap { H264, hardware: true } via with_mft_h264 → capabilities_from, only when the feature is on and available().

No existing signature changed. report_encoder gained a private parameter.

(3) Invariants. Only fluxer-voice changes.

  • mf_encoder.rs names no LiveKit or WebRTC type. pre_encoded is pub(crate), and the only public items it re-exports are EncodedUnit and EncoderConfig, which contain plain data.
  • windows is the workspace's 0.62, optional and Windows-only. Cargo.lock gains only that edge, and cargo deny is green.
  • WebRTC's rate requests are checked: u64→u32 saturates, NaN or huge fps is clamped to 1..max_fps, and a 0 target pauses encoding. Server data has no path to a panic.
  • No secrets.

(4) Review Focus. None of the items belongs to Task 19. Focus 3 (the decode advertisement) is unchanged: the MFT only adds to the encode list.

(5) CI.

  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci in the review worktree: green (all steps passed, exit 0; LiveKit interop skipped because FLUXER_LIVEKIT_TESTS is unset).
  • I also ran cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice --all-targets -- -D warnings on Linux, with and without --features mf-encoder (DOCS_RS=1, target-scoped stub CC/CXX/AR): clean both times.

Findings

  1. minor — mf_encoder.rs pull_output, async format change (plausible, not checkable without hardware). On MF_E_TRANSFORM_STREAM_CHANGE the loop sets the new output type and calls ProcessOutput again at once (continue). An asynchronous MFT may only be asked for output after a METransformHaveOutput event. If a vendor's MFT answers this second call with E_UNEXPECTED, then encode fails, the pump drops the encoder and reopens it on the next frame, and this can repeat every frame (Intel's MFT is known to change format on its first output). Suggested fix: when self.events.is_some(), return Ok(None) after taking the new type and let the next HaveOutput event drive the read. Please add this to #229's checklist.
  2. minor — no fallback when every MFT fails to open. available() only enumerates the MFTs. If all of them refuse to open, publish_screen still publishes a pre-encoded H.264 track and the pump returns Step::Failed forever: the share is live but black, and the codec advert claims hardware H.264. This could happen, for example, with a vendor MFT that wants a D3D11 device manager (spikes §6.4). This does not block while the feature is off. Before turning it on (#229), either open the encoder before publishing and fall back to the SDK's software path, or have available() test-open one.
  3. minor — H.264 level (implementer asked for this check). The pass-through announces 42e01f (level 3.1), and 1080p goes beyond it. This is recorded in spikes §6.1 but not in #229. Add "1080p/60 decodes on a strict hardware decoder (e.g. a Windows viewer with HW decode on), or the MFT's SPS level vs SDP" to #229's checklist.

The encode_async event loop otherwise looks correct:

  • NeedInput events are counted across frames, and ProcessInput is only called when one is owed.
  • The wait is bounded (50 ms for the frame's own output, 1 s for input).
  • Late output is drained without blocking.

The ProcessOutput ManuallyDrop handling is also correct: pSample and pEvents are always taken back and dropped whatever the result, so there is no leak or double release, and a sample the MFT provides is released through the taken Option.

Verdict: APPROVED. No blocking findings and CI is green. The spike exits by its fallback with the feature off. Findings 1–3 should go onto follow-up #229.

## Independent review: Screen share 19 (spike-hw-encode-windows) Scope: `git diff 9e7acfb..origin/screenshare-19-spike-hw-encode-windows` (c8a1187), read without relying on the implementer's notes. **(1) Tests the plan names.** `mft_found_and_encodes_pattern`, `keyframe_on_request` and `rate_control_within_20_percent` are in `tests/mf_encoder.rs`. The file is `#![cfg(all(windows, feature = "mf-encoder"))]`, and each test is `#[ignore]` and does nothing unless `FLUXER_HW_ENCODER_TESTS=1`. What they assert: - SPS, PPS and IDR in the first unit; Annex B; one unit per frame in capture order. - An IDR with SPS on exactly the requested frames and nowhere else. - The bitrate within 20 % at 6, 2 and 4 Mbit/s, 1080p30. That matches the plan's exit criteria. The 10 `rtc::pre_encoded` unit tests ran in CI and pass. The three hardware tests were never run. The plan allows that for a spike: the feature stays off, the PR says so, and follow-up #229 is open. **(2) Interfaces.** The plan asks for four things, and all four are there: - `new_encoded` + `capture_encoded_frame`. - Requests honoured through `take_keyframe_request` and `take_rate_control_request`. - Publishing with `VideoEncoderBackend::PreEncoded`. - `CodecCap { H264, hardware: true }` via `with_mft_h264` → `capabilities_from`, only when the feature is on and `available()`. No existing signature changed. `report_encoder` gained a private parameter. **(3) Invariants.** Only `fluxer-voice` changes. - `mf_encoder.rs` names no LiveKit or WebRTC type. `pre_encoded` is `pub(crate)`, and the only public items it re-exports are `EncodedUnit` and `EncoderConfig`, which contain plain data. - `windows` is the workspace's 0.62, optional and Windows-only. `Cargo.lock` gains only that edge, and `cargo deny` is green. - WebRTC's rate requests are checked: u64→u32 saturates, NaN or huge fps is clamped to 1..max_fps, and a 0 target pauses encoding. Server data has no path to a panic. - No secrets. **(4) Review Focus.** None of the items belongs to Task 19. Focus 3 (the decode advertisement) is unchanged: the MFT only adds to the encode list. **(5) CI.** - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` in the review worktree: **green** (all steps passed, exit 0; LiveKit interop skipped because `FLUXER_LIVEKIT_TESTS` is unset). - I also ran `cargo clippy --target x86_64-pc-windows-msvc -p fluxer-voice --all-targets -- -D warnings` on Linux, with and without `--features mf-encoder` (`DOCS_RS=1`, target-scoped stub CC/CXX/AR): **clean** both times. ### Findings 1. **minor — `mf_encoder.rs` `pull_output`, async format change (plausible, not checkable without hardware).** On `MF_E_TRANSFORM_STREAM_CHANGE` the loop sets the new output type and calls `ProcessOutput` again at once (`continue`). An asynchronous MFT may only be asked for output after a `METransformHaveOutput` event. If a vendor's MFT answers this second call with `E_UNEXPECTED`, then `encode` fails, the pump drops the encoder and reopens it on the next frame, and this can repeat every frame (Intel's MFT is known to change format on its first output). Suggested fix: when `self.events.is_some()`, return `Ok(None)` after taking the new type and let the next `HaveOutput` event drive the read. Please add this to #229's checklist. 2. **minor — no fallback when every MFT fails to open.** `available()` only enumerates the MFTs. If all of them refuse to open, `publish_screen` still publishes a pre-encoded H.264 track and the pump returns `Step::Failed` forever: the share is live but black, and the codec advert claims hardware H.264. This could happen, for example, with a vendor MFT that wants a D3D11 device manager (spikes §6.4). This does not block while the feature is off. Before turning it on (#229), either open the encoder before publishing and fall back to the SDK's software path, or have `available()` test-open one. 3. **minor — H.264 level (implementer asked for this check).** The pass-through announces `42e01f` (level 3.1), and 1080p goes beyond it. This is recorded in spikes §6.1 but not in #229. Add "1080p/60 decodes on a strict hardware decoder (e.g. a Windows viewer with HW decode on), or the MFT's SPS level vs SDP" to #229's checklist. The `encode_async` event loop otherwise looks correct: - `NeedInput` events are counted across frames, and `ProcessInput` is only called when one is owed. - The wait is bounded (50 ms for the frame's own output, 1 s for input). - Late output is drained without blocking. The `ProcessOutput` `ManuallyDrop` handling is also correct: `pSample` and `pEvents` are always taken back and dropped whatever the result, so there is no leak or double release, and a sample the MFT provides is released through the taken `Option`. **Verdict: APPROVED.** No blocking findings and CI is green. The spike exits by its fallback with the feature off. Findings 1–3 should go onto follow-up #229.
charles manually merged commit 9ab00bc511 into screenshare 2026-11-12 08:39:51 +00:00
Sign in to join this conversation.
No description provided.