Screen share 18: Linux hardware encoding (VAAPI; NVENC documented) (#165) #232

Manually merged
charles merged 5 commits from screenshare-18-hw-encode-linux into screenshare 2026-11-12 08:39:50 +00:00
Owner

Stacked on: screenshare-17-voice-livekit-video

What this builds

  • Backend selection (crates/fluxer-voice/src/rtc.rs): the codec probe now keeps its per-backend encoder lists (EncoderLists { software, hardware }), and publish_screen picks the SDK backend for the codec under "Encoder path" with EncoderLists::choose: Auto -> SDK Auto (hardware first where listed); PreferHardware -> Hardware if the hardware backend has the codec, else Software; PreferSoftware -> Software if software has it, else Hardware (H265); a codec no backend has -> Auto (the SDK publishes VP8). screen_video_options takes the chosen backend.
  • Reporting: the probe's log line names the usable hardware encoders (hardware_encoders: vaapi/nvenc, from VideoEncoderBackend::list_available(), as spikes §1.6 asks); voice: screen share published logs the Encoder path, the backend chosen and whether it is hardware. CodecCap { H264, hardware: true } keeps coming from the hardware backend's run-time list (VAAPI or NVENC).
  • CI / release (.forgejo/workflows/ci.yaml, release.yaml): install libva-dev and fail early if pkg-config cannot find libva/libva-drm, so webrtc-sys compiles its VAAPI H264 encoder. libva stays lazily loaded (dlopen("libva.so.2") after an IsSupported() check), so no new runtime dependency; the linux-clean job still proves the AppImage starts without it.
  • Docs: top-level README.md "Hardware video encoding (Linux)" (VAAPI build/run-time conditions, LIBVA_DRIVER_NAME, how to build with NVENC via CUDA_HOME + clean target dir, LIVEKIT_PREFERRED_HW_ENCODER); crates/fluxer-voice/README.md; spikes §5.

How it was tested

  • Unit: backend_mapping_from_encoder_mode (table over a VAAPI host, a host without hardware, an NVENC-like host), publish_options_from_params updated.
  • Real (FLUXER_HW_ENCODER_TESTS=1, not in CI), on the developer machine: Ryzen 9 9950X, VAAPI H264 on the Radeon 610M iGPU (Mesa 26.2.6 radeonsi, renderD129), RTX 5090 present but decode-only through VAAPI, no CUDA toolkit, livekit-server 1.12.0 --dev on loopback, LIBVA_DRIVER_NAME unset:
    • vaapi_reports_hardware_and_publishes passed: H264 reported as hardware; Prefer hardware -> VAAPI H264 Encoder (hardware=true), Prefer software -> OpenH264 (hardware=false); the viewer decodes the pattern from both.
    • vaapi_against_software_cost ran at 2 and 6 bits of noise, 1080p30/60: table in spikes §5.2. VAAPI is 0.15-0.2 core at 1080p30 against 0.6-1.0 for VP9/AV1, with the same PSNR on screen-like content; the iGPU caps frame rate (1080p60 at 40-43 fps). The machine was loaded by other agents' builds, so the numbers are noisy.
  • FLUXER_LIVEKIT_TESTS=1 interop suite (15 tests) passed against the same server.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green locally.
  • NVENC: not testable here (no CUDA headers), documented only, as the spec says.

Deviations from the plan

  • The real test passed before this change already: this machine has libva-dev, and the SDK falls back on its own when a requested backend lacks the codec. The failing step was the unit test (new types). The value of the change is choosing the backend up front from the probe (no SDK fallback warning, correct hardware reporting in the log) plus the CI/release/docs side.
  • The real test also checks "Prefer software" -> OpenH264, and a second ignored test prints the spikes §5 table (env FLUXER_HW_NOISE_BITS).
  • Top-level README.md holds the release-build docs (no separate release doc exists).
  • The spikes section is numbered §5 as the plan says, appended at the end of the file.
  • Follow-up #231 (for Screen share 12): CodecCapabilities marks H264 hardware regardless of Encoder path, so with Prefer software the ranking would still put H264 (now OpenH264) first.
  • Finding: the viewer's FFmpeg logs illegal short term buffer state detected on VAAPI streams (mostly 1080p60); frames still decode. Recorded in spikes §5.2, not acted on.

Closes #165

🤖 Generated with Claude Code

Stacked on: screenshare-17-voice-livekit-video ## What this builds - **Backend selection** (`crates/fluxer-voice/src/rtc.rs`): the codec probe now keeps its per-backend encoder lists (`EncoderLists { software, hardware }`), and `publish_screen` picks the SDK backend for the codec under "Encoder path" with `EncoderLists::choose`: `Auto` -> SDK `Auto` (hardware first where listed); `PreferHardware` -> `Hardware` if the hardware backend has the codec, else `Software`; `PreferSoftware` -> `Software` if software has it, else `Hardware` (H265); a codec no backend has -> `Auto` (the SDK publishes VP8). `screen_video_options` takes the chosen backend. - **Reporting**: the probe's log line names the usable hardware encoders (`hardware_encoders: vaapi/nvenc`, from `VideoEncoderBackend::list_available()`, as spikes §1.6 asks); `voice: screen share published` logs the Encoder path, the backend chosen and whether it is hardware. `CodecCap { H264, hardware: true }` keeps coming from the hardware backend's run-time list (VAAPI or NVENC). - **CI / release** (`.forgejo/workflows/ci.yaml`, `release.yaml`): install `libva-dev` and fail early if `pkg-config` cannot find `libva`/`libva-drm`, so `webrtc-sys` compiles its VAAPI H264 encoder. libva stays lazily loaded (`dlopen("libva.so.2")` after an `IsSupported()` check), so no new runtime dependency; the `linux-clean` job still proves the AppImage starts without it. - **Docs**: top-level `README.md` "Hardware video encoding (Linux)" (VAAPI build/run-time conditions, `LIBVA_DRIVER_NAME`, how to build with NVENC via `CUDA_HOME` + clean target dir, `LIVEKIT_PREFERRED_HW_ENCODER`); `crates/fluxer-voice/README.md`; spikes §5. ## How it was tested - Unit: `backend_mapping_from_encoder_mode` (table over a VAAPI host, a host without hardware, an NVENC-like host), `publish_options_from_params` updated. - Real (`FLUXER_HW_ENCODER_TESTS=1`, not in CI), on the developer machine: Ryzen 9 9950X, VAAPI H264 on the Radeon 610M iGPU (Mesa 26.2.6 radeonsi, renderD129), RTX 5090 present but decode-only through VAAPI, no CUDA toolkit, livekit-server 1.12.0 --dev on loopback, `LIBVA_DRIVER_NAME` unset: - `vaapi_reports_hardware_and_publishes` passed: H264 reported as hardware; Prefer hardware -> `VAAPI H264 Encoder` (hardware=true), Prefer software -> `OpenH264` (hardware=false); the viewer decodes the pattern from both. - `vaapi_against_software_cost` ran at 2 and 6 bits of noise, 1080p30/60: table in spikes §5.2. VAAPI is 0.15-0.2 core at 1080p30 against 0.6-1.0 for VP9/AV1, with the same PSNR on screen-like content; the iGPU caps frame rate (1080p60 at 40-43 fps). The machine was loaded by other agents' builds, so the numbers are noisy. - `FLUXER_LIVEKIT_TESTS=1` interop suite (15 tests) passed against the same server. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green locally. - NVENC: not testable here (no CUDA headers), documented only, as the spec says. ## Deviations from the plan - The real test passed before this change already: this machine has `libva-dev`, and the SDK falls back on its own when a requested backend lacks the codec. The failing step was the unit test (new types). The value of the change is choosing the backend up front from the probe (no SDK fallback warning, correct hardware reporting in the log) plus the CI/release/docs side. - The real test also checks "Prefer software" -> OpenH264, and a second ignored test prints the spikes §5 table (env `FLUXER_HW_NOISE_BITS`). - Top-level `README.md` holds the release-build docs (no separate release doc exists). - The spikes section is numbered §5 as the plan says, appended at the end of the file. - Follow-up #231 (for Screen share 12): `CodecCapabilities` marks H264 hardware regardless of Encoder path, so with Prefer software the ranking would still put H264 (now OpenH264) first. - Finding: the viewer's FFmpeg logs `illegal short term buffer state detected` on VAAPI streams (mostly 1080p60); frames still decode. Recorded in spikes §5.2, not acted on. Closes #165 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Refs #165

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Choose the screen share's encoder backend from the probe's per-backend
lists under "Encoder path", log the usable hardware encoders and the
backend each share uses, install libva-dev in the CI and release
images, document VAAPI and NVENC builds, and record VAAPI against
software at 1080p30/60 in spikes §5.

Refs #165

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Refs #165

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Refs #165

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 18 (hw-encode-linux)

Scope: git diff 9e7acfb8..origin/screenshare-18-hw-encode-linux (4f416a3), read cold.

Checks

  1. Plan tests: unit backend_mapping_from_encoder_mode (rtc.rs) covers Auto / PreferHardware / PreferSoftware against VAAPI, no-hardware and NVENC lists, including the H265 hardware-only and "no backend encodes it" cases. Real vaapi_reports_hardware_and_publishes (tests/hw_encoder.rs, #[ignore] + FLUXER_HW_ENCODER_TESTS=1) asserts CodecCap { codec: H264, hardware: true } is reported, that "Prefer hardware" publishes through VAAPI with the hardware flag set, that "Prefer software" does not, and that a viewer decodes the pattern both times. OK.
  2. Interfaces: VAAPI is compiled in through libva-dev in ci.yaml and release.yaml, with a pkg-config guard that fails the build early (libva libva-drm; webrtc-sys build.rs reads libva includedir and dlopens at run time). H264 is reported as hardware from the per-backend probe. "Encoder path" is honoured via EncoderLists::choose. Spikes §5 has the CPU/quality table for 1080p30 and 1080p60. NVENC is documented in the README (CUDA_HOME, clean target dir). OK.
  3. Invariants: only fluxer-voice, CI and docs change. EncoderLists, EncoderChoice and Probe are private, and no LiveKit or libwebrtc type reaches a public signature. No new dependencies. The only key in the test is the public livekit-server --dev pair, the same one livekit_interop.rs uses. No token in the logs: the new fields are the encoder path, the backend and the hardware flag. OK.
  4. Review Focus: none of the items is Task 18's. #3 (the advert claims only decodable codecs) is unaffected: capabilities_from is unchanged.
  5. FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci in a clean review worktree: all steps passed. backend_mapping_from_encoder_mode and publish_options_from_params ran and passed. The hw tests skip without the env var.

Findings

  • minor: encoder_lists() falls back to empty lists (everything Auto) only if the blocking probe task panics. In that case "Prefer software" quietly becomes the SDK's hardware-first choice. This is an unlikely path, and the warning is logged. No action needed.
  • minor (already tracked in #231): with "Prefer software", the Screen share 12 ranking still treats H264 as hardware when VAAPI is present. That belongs to Task 12's selector, not this diff.
  • note: the VAAPI rows of spikes §5.2 show a low and variable encoder frame rate at 30 fps (13 enc fps in one run) on a heavily loaded machine. The table says the numbers are noisy. Worth re-measuring on an idle machine before deciding whether "Auto" should keep hardware first for H264. Not blocking.

Verdict: approve. No blocking findings, and local CI is green.

## Independent review: Screen share 18 (hw-encode-linux) Scope: `git diff 9e7acfb8..origin/screenshare-18-hw-encode-linux` (4f416a3), read cold. **Checks** 1. Plan tests: unit `backend_mapping_from_encoder_mode` (rtc.rs) covers Auto / PreferHardware / PreferSoftware against VAAPI, no-hardware and NVENC lists, including the H265 hardware-only and "no backend encodes it" cases. Real `vaapi_reports_hardware_and_publishes` (tests/hw_encoder.rs, `#[ignore]` + `FLUXER_HW_ENCODER_TESTS=1`) asserts `CodecCap { codec: H264, hardware: true }` is reported, that "Prefer hardware" publishes through VAAPI with the hardware flag set, that "Prefer software" does not, and that a viewer decodes the pattern both times. OK. 2. Interfaces: VAAPI is compiled in through `libva-dev` in ci.yaml and release.yaml, with a pkg-config guard that fails the build early (`libva libva-drm`; webrtc-sys build.rs reads `libva` includedir and dlopens at run time). H264 is reported as hardware from the per-backend probe. "Encoder path" is honoured via `EncoderLists::choose`. Spikes §5 has the CPU/quality table for 1080p30 and 1080p60. NVENC is documented in the README (CUDA_HOME, clean target dir). OK. 3. Invariants: only fluxer-voice, CI and docs change. `EncoderLists`, `EncoderChoice` and `Probe` are private, and no LiveKit or libwebrtc type reaches a public signature. No new dependencies. The only key in the test is the public `livekit-server --dev` pair, the same one livekit_interop.rs uses. No token in the logs: the new fields are the encoder path, the backend and the hardware flag. OK. 4. Review Focus: none of the items is Task 18's. #3 (the advert claims only decodable codecs) is unaffected: `capabilities_from` is unchanged. 5. `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` in a clean review worktree: **all steps passed**. `backend_mapping_from_encoder_mode` and `publish_options_from_params` ran and passed. The hw tests skip without the env var. **Findings** - minor: `encoder_lists()` falls back to empty lists (everything `Auto`) only if the blocking probe task panics. In that case "Prefer software" quietly becomes the SDK's hardware-first choice. This is an unlikely path, and the warning is logged. No action needed. - minor (already tracked in #231): with "Prefer software", the Screen share 12 ranking still treats H264 as hardware when VAAPI is present. That belongs to Task 12's selector, not this diff. - note: the VAAPI rows of spikes §5.2 show a low and variable encoder frame rate at 30 fps (13 enc fps in one run) on a heavily loaded machine. The table says the numbers are noisy. Worth re-measuring on an idle machine before deciding whether "Auto" should keep hardware first for H264. Not blocking. **Verdict: approve.** No blocking findings, and local CI is green.
charles manually merged commit 9ab00bc511 into screenshare 2026-11-12 08:39:50 +00:00
Sign in to join this conversation.
No description provided.