Windows hardware H.264 on: GPU order, self-test before first use, software fallback (#229) #873

Closed
charles wants to merge 0 commits from feat-229-windows-hardware-encoding into textchat
Owner

Refs #229

The MediaFoundation layer (crates/fluxer-voice/src/mf_encoder.rs) has never run on any machine. Charles decided on 2026-11-18 ("Turn it on") to ship it now, so it is turned on in every Windows build with guards for code that has never executed: each GPU's encoder transform is self-tested before it is first used, it is given up if it fails during a share, and software encoding takes over. Everything Windows-specific was type-checked and linted from Linux. None of it was linked or run. #229 stays open until someone runs it on Windows hardware (see the checklist, docs/acceptance/screenshare.md item 10).

Stacked on PR #865 (fix-857-vaapi-hardware-encoding, merged into this branch: commit d8d43bc3, because it is not in textchat yet). The diff against textchat therefore includes #865's changes (60 files). This PR's own commit is 8af95413: 41 files, +5115/−896. About 790 lines of that only move the existing rtc::pre_encoded module out of rtc.rs into rtc/pre_encoded.rs.

What changes

  1. On in Windows builds. apps/fluxer/Cargo.toml turns on fluxer-voice/mf-encoder for cfg(windows). That covers CI's cargo test/clippy --workspace and the release build. cargo xtask package also passes --features fluxer-voice/mf-encoder for Windows (unit test windows_releases_have_the_hardware_encoder). ci.yaml and release.yaml only gain comments, because the manifest already does the work. The Windows CI job therefore compiles the code, lints it with -D warnings and runs every test that needs no GPU. tests/mf_encoder.rs stays ignored unless FLUXER_HW_ENCODER_TESTS=1.
  2. Encoder order on Windows, same rule as #857 (fluxer_core::voice::encoder_order):
    • The GPUs are the DXGI adapters, with software adapters and WARP left out. Each GPU's kind comes from wgpu's DX12 adapter info, matched by PCI ids.
    • The display GPU is the adapter that drives the captured monitor. fluxer_capture::display_of finds that monitor: for a window, its nearest monitor; for a screen, the id read as an HMONITOR or else as a display index. The monitor's GDI device name is then matched to the DXGI outputs. If that fails, the primary monitor's adapter is used.
    • Encoder transforms are matched to adapters with MFTEnum2 and MFT_ENUM_ADAPTER_LUID. Where that lists nothing, MFTEnumEx results are matched by vendor, but only when exactly one adapter has that vendor.
    • Ids are vendor:device@dxgiN. The once-per-share INFO line voice: screen share encoder order … and the --debug-info lines encoder gpu: … and encoder order: … are the same as on Linux, with encoder mft. "Encoding GPU" works the same way.
    • The pure rules are in gpu::rules and are tested on Linux.
  3. Self-test before first use, per transform and GPU and per resolution class. The class is the lowest level in Table A-1 for the share's size and frame rate (selftest.rs).
    • The transform encodes 6 frames of the test pattern at the share's size, frame rate and bitrate. Keyframes are asked for at frames 0 and 3, and up to 4 extra frames flush an encoder that holds frames back. The whole test is bounded at 1.5 s.
    • The output must pass h264_check::check_stream:
      • Annex B framing and valid NAL headers;
      • SPS and PPS before the first slice, and an SPS whose cropped size is the share's size;
      • Baseline profile without CABAC (what the pass-through announces, 42e01f);
      • a level at least the Table A-1 minimum;
      • an IDR first, and an IDR with its SPS at frame 3;
      • keyframe flags that agree with the NAL units;
      • slices with sensible headers in every unit;
      • at most 4× the bitrate budget plus a quarter of a raw frame per keyframe, with a 64 KiB floor.
    • The output must also decode in FFmpeg's software decoder (fluxer_media_playback::h264::decode_annex_b) to the pattern: the right frame counters, and a mean difference of at most 12 per plane.
    • If a check fails, the encoder errors or panics, or the test times out, that transform is unusable at that class for the rest of the run. The WARN line names the check that failed. The next transform in the order takes over, and software comes last.
    • The verdicts, passes included, are kept per run (Registry).
  4. Failure during a share (rtc::pre_encoded::Pump, which now walks a list of steps):
    • The transform in use is given up for the rest of the run if it returns an error, gives no output for 2 s of capture time, or produces an access unit that h264_check::UnitChecker rejects. A rejected unit is not sent.
    • The next transform starts on a keyframe.
    • When none is left, the room unpublishes the pre-encoded track and republishes the video on WebRTC's software encoder. The session hears of the new track as LocalTrackPublished, and the store takes the new sid.
    • Each transform given up produces RoomEvent::EncoderFallback → VoiceEvent::HardwareEncoderFallback.
    • Why not libwebrtc's selector from #838/#865: a pre-encoded track has no libwebrtc encoder to switch, so the same rule is implemented here.
  5. Escape hatch. With "Encoder path" set to "Prefer software", the pre-encoded path is not used, no self-test runs and no order is installed (uses_pre_encoded, install_order). The store shows a new one-time notice, RefusalModal::ENCODER_FAILED ("Hardware encoder unavailable"). It names "Encoder path" → "Prefer software" and the settings path to it, and it appears for Auto too.
  6. Level. MF_MT_MPEG2_LEVEL is set to the Table A-1 minimum, as #764 and #857 did. If the transform refuses that output type, it is set again without a level, and the refusal is logged.
  7. Logging.
    • INFO: DXGI adapters (LUID, ids, outputs, primary monitor), transforms found (name, vendor, LUID), the order line, the steps planned for the share, the self-test result with open/encode/check/total times and profile/level/bytes, and each transform opened (friendly name, vendor, hardware flag and URL, adapter LUID, sync or async, the negotiated output and input media types).
    • WARN: every self-test failure, step skipped, step given up and software republish, with the reason.
  8. Docs. Spikes §6.5, the spec's §7.2 note, the fluxer-voice, fluxer-media-playback and fluxer-video READMEs, a webrtc-sys PATCHES.md entry, and the acceptance checklist item 10. Windows hardware encoding (what to click, which log lines to send back).

Tests (all run here, Linux, in the gate)

  • fluxer-voice h264_check::tests: real streams made here by tools/h264-fixtures/make.sh (6 pattern frames at 640×360 and 30 fps, IDRs at frames 0 and 3) from OpenH264 (h264enc), NVENC (RTX 5090) and VAAPI (AMD radeonsi). All three pass, decode and match the pattern. Broken variants are rejected: missing SPS (first unit and the later keyframe), missing PPS, wrong resolution (also off by 2 rows), no IDR first, keyframe request not honoured, lying keyframe flag, too few frames, garbage slice payloads (4 seeds × 3 encoders), the forbidden bit and non-Annex-B framing, a level too low for the size or for 60 fps, High profile or CABAC, filler far over the bitrate, and pictures swapped, missing or with wrong chroma. Other tests check UnitChecker on running streams and parameter sets written and read back.
  • minimum_level_is_the_one_webrtc_sys_uses: checks the Rust Table A-1 against webrtc-sys's C++ h264_minimum_level_idc on a grid of sizes, frame rates and bitrates. This needed webrtc-sys's h264_level module to be pub (entry in PATCHES.md).
  • selftest::tests: an encoder that replays the real streams passes, also with two frames of delay. Each failure kind fails with its named check: open error, encode error, keyframe ignored, frames out of order, panic, wrong size. A hung encoder times out within its bound. Verdicts are kept per run, per class, and for transforms given up mid-share.
  • rtc::pre_encoded::tests: open failure → next step → software. Errors, stalls and garbage mid-share → next step on a keyframe → exhausted, with no garbage sent. A step that fails its self-test is never opened. Earlier tests are kept.
  • gpu::rules::tests: a desktop with the dGPU as display GPU and an iGPU; a hybrid laptop; monitors on two GPUs (the captured one wins, then the primary); transforms without a LUID; two identical cards; GPUs without an encoder; the same order line as Linux.
  • fluxer-core stream_golive::a_hardware_encoder_given_up_is_said_once_with_the_escape_hatch, apps/fluxer debug_info::tests::report_lists_windows_encoder_gpus_the_same_way, xtask windows_releases_have_the_hardware_encoder, and fluxer-media-playback h264::tests.
  • The env-gated hardware tests (tests/mf_encoder.rs, tests/hw_encoder.rs) are unchanged.

Windows type-check (what was and was not checked)

cargo clippy --target x86_64-pc-windows-msvc --workspace --all-targets -- -D warnings was run from Linux in the workspace itself rather than an out-of-tree copy, and it passes: rc=0. The in-tree check covers more than copying the module would, including the MSVC target, the app with the feature on, and fluxer-capture's new code.

How it was set up:

  • A first run with DOCS_RS=1 skipped webrtc-sys's native build. Its build script does not rerun when environment variables change, so the result was reused.
  • CC, CXX and AR for the target pointed at stub scripts that write empty outputs.
  • FFmpeg's Windows binding was tools/ffmpeg/bindings/8.0.3-r1-win-x64.rs.

What this does and does not show:

  • Checked: type-checking and lints (the "unused import outside cfg" kind included) for the whole workspace on Windows with mf-encoder. That covers mf_encoder.rs (MFTEnum2, attributes, level, negotiated-type logging), gpu.rs's DXGI, GDI and wgpu code, rtc.rs's Windows plan, verifier and software republish, fluxer-capture's display_of, and tests/mf_encoder.rs.
  • Not checked: linking, and any run. The real C/C++ build of webrtc-sys for Windows was not done either.

Not verified

  • The whole MediaFoundation path on real hardware: whether the transforms accept NV12 in system memory, force keyframes, honour bitrate changes and the level, and what LUIDs and vendor attributes they report (§6.4).
  • MFTEnum2 on old Windows 10 builds (the MFTEnumEx fallback path).
  • How libwebrtc numbers screen sources under WGC versus DXGI (display_of handles both readings).
  • The software republish against a live server, and how Electron viewers react to the new track.

Noticed, not fixed (possible separate issues)

  • The first gate run hit the known SIGSEGV at process exit after all 96 fluxer lib tests had passed (#549 / #304). The binary then ran 15/15 clean, and a second full gate passed.
  • On Windows, H.264 is now listed as hardware whenever a transform exists. A share whose transforms all fail falls back to OpenH264 at full resolution, with no 720p30 software clamp (as on Linux before #764).
  • The self-test opens a second encoder instead of reusing the tested one, which costs one extra open per first use.

Gate

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci on this machine (no CUDA headers, so NVENC was not compiled into webrtc-sys):

==> xtask ci: livekit-interop
    skipped (FLUXER_LIVEKIT_TESTS is not 1)
==> xtask ci: all steps passed

No goldens were added or changed.

🤖 Generated with Claude Code

Refs #229 **The MediaFoundation layer (`crates/fluxer-voice/src/mf_encoder.rs`) has never run on any machine.** Charles decided on 2026-11-18 ("Turn it on") to ship it now, so it is turned on in every Windows build with guards for code that has never executed: each GPU's encoder transform is self-tested before it is first used, it is given up if it fails during a share, and software encoding takes over. Everything Windows-specific was type-checked and linted from Linux. None of it was linked or run. #229 stays open until someone runs it on Windows hardware (see the checklist, `docs/acceptance/screenshare.md` item 10). **Stacked on PR #865** (`fix-857-vaapi-hardware-encoding`, merged into this branch: commit d8d43bc3, because it is not in `textchat` yet). The diff against `textchat` therefore includes #865's changes (60 files). This PR's own commit is 8af95413: 41 files, +5115/−896. About 790 lines of that only move the existing `rtc::pre_encoded` module out of `rtc.rs` into `rtc/pre_encoded.rs`. ## What changes 1. **On in Windows builds.** `apps/fluxer/Cargo.toml` turns on `fluxer-voice/mf-encoder` for `cfg(windows)`. That covers CI's `cargo test/clippy --workspace` and the release build. `cargo xtask package` also passes `--features fluxer-voice/mf-encoder` for Windows (unit test `windows_releases_have_the_hardware_encoder`). `ci.yaml` and `release.yaml` only gain comments, because the manifest already does the work. The Windows CI job therefore compiles the code, lints it with `-D warnings` and runs every test that needs no GPU. `tests/mf_encoder.rs` stays ignored unless `FLUXER_HW_ENCODER_TESTS=1`. 2. **Encoder order on Windows, same rule as #857** (`fluxer_core::voice::encoder_order`): - The GPUs are the DXGI adapters, with software adapters and WARP left out. Each GPU's kind comes from wgpu's DX12 adapter info, matched by PCI ids. - The display GPU is the adapter that drives the captured monitor. `fluxer_capture::display_of` finds that monitor: for a window, its nearest monitor; for a screen, the id read as an `HMONITOR` or else as a display index. The monitor's GDI device name is then matched to the DXGI outputs. If that fails, the primary monitor's adapter is used. - Encoder transforms are matched to adapters with `MFTEnum2` and `MFT_ENUM_ADAPTER_LUID`. Where that lists nothing, `MFTEnumEx` results are matched by vendor, but only when exactly one adapter has that vendor. - Ids are `vendor:device@dxgiN`. The once-per-share INFO line `voice: screen share encoder order …` and the `--debug-info` lines `encoder gpu: …` and `encoder order: …` are the same as on Linux, with encoder `mft`. "Encoding GPU" works the same way. - The pure rules are in `gpu::rules` and are tested on Linux. 3. **Self-test before first use**, per transform and GPU and per resolution class. The class is the lowest level in Table A-1 for the share's size and frame rate (`selftest.rs`). - The transform encodes 6 frames of the test pattern at the share's size, frame rate and bitrate. Keyframes are asked for at frames 0 and 3, and up to 4 extra frames flush an encoder that holds frames back. The whole test is bounded at 1.5 s. - The output must pass `h264_check::check_stream`: - Annex B framing and valid NAL headers; - SPS and PPS before the first slice, and an SPS whose cropped size is the share's size; - Baseline profile without CABAC (what the pass-through announces, `42e01f`); - a level at least the Table A-1 minimum; - an IDR first, and an IDR with its SPS at frame 3; - keyframe flags that agree with the NAL units; - slices with sensible headers in every unit; - at most 4× the bitrate budget plus a quarter of a raw frame per keyframe, with a 64 KiB floor. - The output must also decode in FFmpeg's software decoder (`fluxer_media_playback::h264::decode_annex_b`) to the pattern: the right frame counters, and a mean difference of at most 12 per plane. - If a check fails, the encoder errors or panics, or the test times out, that transform is unusable at that class for the rest of the run. The WARN line names the check that failed. The next transform in the order takes over, and software comes last. - The verdicts, passes included, are kept per run (`Registry`). 4. **Failure during a share** (`rtc::pre_encoded::Pump`, which now walks a list of steps): - The transform in use is given up for the rest of the run if it returns an error, gives no output for 2 s of capture time, or produces an access unit that `h264_check::UnitChecker` rejects. A rejected unit is not sent. - The next transform starts on a keyframe. - When none is left, the room unpublishes the pre-encoded track and republishes the video on WebRTC's software encoder. The session hears of the new track as `LocalTrackPublished`, and the store takes the new sid. - Each transform given up produces `RoomEvent::EncoderFallback` → `VoiceEvent::HardwareEncoderFallback`. - Why not libwebrtc's selector from #838/#865: a pre-encoded track has no libwebrtc encoder to switch, so the same rule is implemented here. 5. **Escape hatch.** With "Encoder path" set to "Prefer software", the pre-encoded path is not used, no self-test runs and no order is installed (`uses_pre_encoded`, `install_order`). The store shows a new one-time notice, `RefusalModal::ENCODER_FAILED` ("Hardware encoder unavailable"). It names "Encoder path" → "Prefer software" and the settings path to it, and it appears for Auto too. 6. **Level.** `MF_MT_MPEG2_LEVEL` is set to the Table A-1 minimum, as #764 and #857 did. If the transform refuses that output type, it is set again without a level, and the refusal is logged. 7. **Logging.** - INFO: DXGI adapters (LUID, ids, outputs, primary monitor), transforms found (name, vendor, LUID), the order line, the steps planned for the share, the self-test result with open/encode/check/total times and profile/level/bytes, and each transform opened (friendly name, vendor, hardware flag and URL, adapter LUID, sync or async, the negotiated output and input media types). - WARN: every self-test failure, step skipped, step given up and software republish, with the reason. 8. **Docs.** Spikes §6.5, the spec's §7.2 note, the `fluxer-voice`, `fluxer-media-playback` and `fluxer-video` READMEs, a `webrtc-sys` `PATCHES.md` entry, and the acceptance checklist item **10. Windows hardware encoding** (what to click, which log lines to send back). ## Tests (all run here, Linux, in the gate) - `fluxer-voice` `h264_check::tests`: real streams made here by `tools/h264-fixtures/make.sh` (6 pattern frames at 640×360 and 30 fps, IDRs at frames 0 and 3) from OpenH264 (`h264enc`), NVENC (RTX 5090) and VAAPI (AMD radeonsi). All three pass, decode and match the pattern. Broken variants are rejected: missing SPS (first unit and the later keyframe), missing PPS, wrong resolution (also off by 2 rows), no IDR first, keyframe request not honoured, lying keyframe flag, too few frames, garbage slice payloads (4 seeds × 3 encoders), the forbidden bit and non-Annex-B framing, a level too low for the size or for 60 fps, High profile or CABAC, filler far over the bitrate, and pictures swapped, missing or with wrong chroma. Other tests check `UnitChecker` on running streams and parameter sets written and read back. - `minimum_level_is_the_one_webrtc_sys_uses`: checks the Rust Table A-1 against `webrtc-sys`'s C++ `h264_minimum_level_idc` on a grid of sizes, frame rates and bitrates. This needed `webrtc-sys`'s `h264_level` module to be `pub` (entry in `PATCHES.md`). - `selftest::tests`: an encoder that replays the real streams passes, also with two frames of delay. Each failure kind fails with its named check: open error, encode error, keyframe ignored, frames out of order, panic, wrong size. A hung encoder times out within its bound. Verdicts are kept per run, per class, and for transforms given up mid-share. - `rtc::pre_encoded::tests`: open failure → next step → software. Errors, stalls and garbage mid-share → next step on a keyframe → exhausted, with no garbage sent. A step that fails its self-test is never opened. Earlier tests are kept. - `gpu::rules::tests`: a desktop with the dGPU as display GPU and an iGPU; a hybrid laptop; monitors on two GPUs (the captured one wins, then the primary); transforms without a LUID; two identical cards; GPUs without an encoder; the same order line as Linux. - `fluxer-core` `stream_golive::a_hardware_encoder_given_up_is_said_once_with_the_escape_hatch`, `apps/fluxer` `debug_info::tests::report_lists_windows_encoder_gpus_the_same_way`, `xtask` `windows_releases_have_the_hardware_encoder`, and `fluxer-media-playback` `h264::tests`. - The env-gated hardware tests (`tests/mf_encoder.rs`, `tests/hw_encoder.rs`) are unchanged. ## Windows type-check (what was and was not checked) `cargo clippy --target x86_64-pc-windows-msvc --workspace --all-targets -- -D warnings` was run from Linux in the workspace itself rather than an out-of-tree copy, and it passes: `rc=0`. The in-tree check covers more than copying the module would, including the MSVC target, the app with the feature on, and `fluxer-capture`'s new code. How it was set up: - A first run with `DOCS_RS=1` skipped `webrtc-sys`'s native build. Its build script does not rerun when environment variables change, so the result was reused. - `CC`, `CXX` and `AR` for the target pointed at stub scripts that write empty outputs. - FFmpeg's Windows binding was `tools/ffmpeg/bindings/8.0.3-r1-win-x64.rs`. What this does and does not show: - **Checked:** type-checking and lints (the "unused import outside cfg" kind included) for the whole workspace on Windows with `mf-encoder`. That covers `mf_encoder.rs` (`MFTEnum2`, attributes, level, negotiated-type logging), `gpu.rs`'s DXGI, GDI and wgpu code, `rtc.rs`'s Windows plan, verifier and software republish, `fluxer-capture`'s `display_of`, and `tests/mf_encoder.rs`. - **Not checked:** linking, and any run. The real C/C++ build of `webrtc-sys` for Windows was not done either. ## Not verified - The whole MediaFoundation path on real hardware: whether the transforms accept NV12 in system memory, force keyframes, honour bitrate changes and the level, and what LUIDs and vendor attributes they report (§6.4). - `MFTEnum2` on old Windows 10 builds (the `MFTEnumEx` fallback path). - How libwebrtc numbers screen sources under WGC versus DXGI (`display_of` handles both readings). - The software republish against a live server, and how Electron viewers react to the new track. ## Noticed, not fixed (possible separate issues) - The first gate run hit the known SIGSEGV at process exit after all 96 `fluxer` lib tests had passed (#549 / #304). The binary then ran 15/15 clean, and a second full gate passed. - On Windows, H.264 is now listed as hardware whenever a transform exists. A share whose transforms all fail falls back to OpenH264 at full resolution, with no 720p30 software clamp (as on Linux before #764). - The self-test opens a second encoder instead of reusing the tested one, which costs one extra open per first use. ## Gate `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` on this machine (no CUDA headers, so NVENC was not compiled into `webrtc-sys`): ```text ==> xtask ci: livekit-interop skipped (FLUXER_LIVEKIT_TESTS is not 1) ==> xtask ci: all steps passed ``` No goldens were added or changed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
- fluxer-core: chat::known finds a message loaded or listed (pins, search
  results, bookmarks, mentions); the message menu, its actions and the
  lightbox use it, so a pin outside the loaded history has its menu and
  its pictures open.
- Cards read their spoilers from the message view their clicks go to.
- Search results and inbox entries route their row events like the pins
  (screens/main/cards.rs); a search result also has the message menu and
  jumps on a click on its body; the inbox has no menu, as the original.
- The thread side panel's message actions go to the screen's actions
  state, which is drawn (the panel's menu was set and never drawn).
- Headless tests in tests/pins_clicks.rs for all four, each failing on
  origin/textchat.

Closes #849

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge pull request 'Search, inbox and unloaded pin cards answer clicks; thread panel message menu is drawn (#849)' (#862) from fix-849-inert-cards into textchat
All checks were successful
ci / Windows (x64) (push) Successful in 49m39s
ci / Linux (x86_64) (push) Successful in 1h3m51s
ci / Start the release build (push) Successful in 29s
8310a6a548
Charles, 2026-11-18: "Turn it on". The MediaFoundation H.264 encoder
(fluxer-voice `mf-encoder`) is on in every Windows build (apps/fluxer,
xtask package), but it has never run on a Windows machine, so:

- GPU order as on Linux (#857): DXGI adapters, kinds from wgpu (DX12), the
  display GPU = the adapter driving the captured monitor
  (fluxer_capture::display_of), transforms per adapter LUID (MFTEnum2,
  MFT_ENUM_ADAPTER_LUID); same order line, --debug-info lines and
  "Encoding GPU" setting. Pure rules in gpu::rules, tested on Linux.
- Self-test before each transform's first use per resolution class: 6
  pattern frames, checked as a bitstream (h264_check: NAL syntax, SPS/PPS,
  size, Baseline/CAVLC, level vs Table A-1, IDR first and on request,
  slices, size vs bitrate) and decoded with FFmpeg's software decoder
  against the pattern; bounded at 1.5 s; failures remembered per run.
- Mid-share: an error, a 2 s stall or a unit the checks reject gives the
  transform up; the next one takes over on a keyframe; with none left the
  video track is republished on the software encoder.
- One-time notice naming Encoder path -> Prefer software, which bypasses
  hardware entirely. MF_MT_MPEG2_LEVEL set to the minimum level.
- Validator tests use real OpenH264/NVENC/VAAPI streams
  (tools/h264-fixtures/make.sh) and broken variants.

Refs #229

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405); the commits are in the branch.

Merged into `textchat` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-18 06:36:02 +00:00
charles referenced this pull request from a commit 2026-11-18 07:45:41 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.