Screen share: hardware H.264 above 720p30; Prefer hardware falls back instead of sending nothing (#764) #838

Closed
charles wants to merge 0 commits from fix-764-nvenc-above-720p30 into textchat
Owner

Closes #764

Measured on Charles's machine (RTX 5090, driver 620.70.06), headless, with tools/capture-consistency/run.sh --chain (#763): the real Wayland capture path, a local livekit-server --dev, a second session that decodes (FFmpeg's H.264 decoder in libwebrtc, LK_DISABLE_NVDEC=1, unless noted). The NVIDIA codecs were compiled with cuda.h from NVIDIA's cuda-cudart-dev-12-9 package unpacked into a scratch directory (CUDA_HOME), as #763 did; removed afterwards.

1. NVENC H.264 refused everything above 720p30 (cause verified)

webrtc-sys's NVENC H.264 wrapper configured the encoder with the level of the SDP format: the factory offers profile-level-id=42e01f (Constrained Baseline, level 3.1, max 108 000 macroblocks/s = 1280×720 at 30 fps), and NvidiaH264EncoderImpl passed that level to NV_ENC_CONFIG_H264.level. The init parameters, printed by the new NVIDIA H264 NVENC config: line:

  • failing (level forced back to the SDP's for this check): 1920x1080 fps=60/1 profile=baseline level=31 (sdp profile-level-id=42e01f) rc=2 avg_bps=5219000 max_bps=0 codec_max_kbps=6000 vbv=434915 gop=4294967295 idr=4294967295 p_interval=1 → nvEncInitializeEncoder … returned error 8 (NV_ENC_ERR_INVALID_PARAM)
  • working: the same with level=42.

Cross-check with FFmpeg's h264_nvenc on the same GPU: -level 3.1 at 1280×720@60 and 1920×1080@60 → InitializeEncoder failed: invalid param (8): Invalid Level; 3.2 / 4.2 / auto work (auto picks 4.2 for 1080p60, 5.1 for 1440p60); 4.2 at 1440p60 fails the same way. OpenH264 ignores the offered level (it wrote 3.2 / 4.2 / 5.1 into its SPS), which is why "Auto" quietly worked in software.

A second bug in the same constructor: the NVENC profile GUID (nv_profile_guid_) was never initialised. At 720p30, the one size that started, NVENC sent a High profile stream (ffprobe: profile=High level=31) under the Constrained Baseline negotiation.

Fix (chosen: (a), not changing the SDP): include/livekit/h264_level.h computes the lowest level of H.264 Table A-1 that admits width × height × fps and the max bitrate (MaxFS, Sqrt(8·MaxFS) per side, MaxMBPS, MaxBR × cpbBrVclFactor), as Chromium's hardware encoders do (FindValidH264Level) under the level-asymmetry-allowed=1 that the factory already offers; NVENC's automatic choice only when nothing in the table fits. The profile is the negotiated one (Constrained Baseline → NVENC Baseline). The bitstream now says Constrained Baseline with level 3.2 / 4.2 / 4.2 / 5.1.

The code is in the registry crate webrtc-sys 0.3.53, so it is vendored under third_party/webrtc-sys the way third_party/livekit is: first commit = the published .crate unchanged (checksum as in Cargo.lock), second = the patch, every change marked fluxer-native patch (PATCHES.md) and listed with file/line in third_party/webrtc-sys/PATCHES.md; [patch.crates-io], exclude, VENDORED_CRATES (its unit tests run in vendored-tests, Linux).

H.265 and AV1: not the same line of code. H.265 never sets a level (the preset's = automatic), AV1 asks for NV_ENC_LEVEL_AV1_AUTOSELECT. AV1 NVENC started and was received at 1920×1080@60 (1801 frames in 30 s) and 2560×1440@60 (1800), 0 mixed. H.265 could not be measured end to end: the test share asked for H.265 was published as VP8 (libvpx) by the session, so NVENC H.265 never ran; FFmpeg's hevc_nvenc with the automatic level works at both sizes. Unchanged.

2. "Prefer hardware" sent nothing

With a backend other than "auto", the sender gets the SDK's FixedVideoEncoderSelector, whose OnEncoderBroken asks for the same backend again. NVENC broke twice and libwebrtc stopped trying: 0 frames in 30 s (reproduced: before-720p60-hw, received frames=0). Fix: a hardware backend (hardware/nvenc/vaapi/videotoolbox) that breaks after it was asked for explicitly gives way to the software encoder for the rest of that sender (never re-requested on bitrate/resolution changes); pre-encoded stays a hard requirement as upstream. Verified by injecting the old level (local, uncommitted fault injection): Video encoder backend hardware failed for H264; falling back to the software encoder, then OpenH264, 1800/1800 frames received at 1080p60, 0 mixed.

The user is told once per run: fluxer-core shows "Hardware encoder unavailable — The hardware encoder couldn't start for this screen share, so it uses the software encoder. Prefer hardware works like Automatic for now." (the existing voice notice modal) when a "Prefer hardware" share whose codec has a hardware encoder reports a software encoder. The original has no such notice; the wording follows its "Encoder path" description (ENCODER_PATH_NO_HARDWARE_DESCRIPTOR).

Logs: voice: screen share published no longer has hardware= (it was the prediction, true while OpenH264 ran). voice: screen share encoder implementation=… hardware=… (from the statistics) gains hardware_expected, and a warning when hardware was expected and software runs.

3. Other items of #764

  • Nvidia Decoder is supported. on stdout (nvidia_decoder_factory.cpp:157) now goes to the log. The gate below ran on this NVIDIA host with the NVIDIA codecs compiled in and without LK_DISABLE_NVDEC=1, so appimage_has_flaxative_desktop_icon_and_binary passes here now.
  • README vs release.yaml: the README was stale. release.yaml (and CI) install cuda-cudart-dev-12-9, so the Linux release AppImage has NVENC/NVDEC (it reports hardware_encoders: ["nvenc", …] on an NVIDIA host). README corrected; a stale comment in release.yaml ("NVENC is not built here") too.
  • 1920×1046 received as 1920×1044: not fixed here. Measured: the frames handed to the session are 1046 tall, but the encoder is already configured with 1044 (NVIDIA H264 NVENC config: 1920x1044) and OpenH264's SPS says 1044 as well, so the two rows go in libwebrtc's frame adaptation before the encoder (not the decoder, not NVENC). Needs its own issue.

Measurements (60 s each, Auto, H.264, 6 Mbit/s)

Before = textchat + the published webrtc-sys (only the dump diagnostic added); after = this branch. "Encoder thread" is libwebrtc's VideoEncoderQue thread CPU over the 60 s; for software the same build with "Prefer software" (OpenH264, the encoder "before" fell back to). Process CPU includes the test's own row checkers (debug build, ~16–37 s of each total), capture, conversion and the viewer's decoding.

size @60 before: encoder before: sent / received / mixed before: process CPU after: encoder after: sent / received / mixed after: process CPU encoder thread: OpenH264 → NVENC SPS level before → after
1280×720 OpenH264 (NVENC error 8) 3661 / 3600 / 0 33.8 s (0.56 cores) NVENC 3635 / 3604 / 0 25.9 s (0.43) 7.8 s → 1.0 s 3.2 → 3.2
1920×1080 OpenH264 (NVENC error 8) 3664 / 3600 / 0 66.1 s (1.10) NVENC 3639 / 3603 / 0 51.0 s (0.85) 19.1 s → 1.5 s 4.2 → 4.2
1920×1046 OpenH264 (NVENC error 8) 3666 / 3600 (as 1044) / 0 62.5 s (1.04) NVENC 3653 / 3594 (as 1044) / 0 53.0 s (0.88) 17.3 s → 1.7 s 4.2 → 4.2
2560×1440 OpenH264 (NVENC error 8) 3663 / 3598 / 0 97.4 s (1.62) NVENC 3640 / 3603 / 0 81.2 s (1.35) 31.5 s → 2.0 s 5.1 → 5.1

("sent" = frames handed to the encoder, the converted stage; mixed = 0 at raw, converted and received in every run.) Levels were read with ffprobe from the dumped stream (FLUXER_H264_DUMP); before, it is OpenH264's own level, after, NVENC's. Profile before/after: Constrained Baseline (OpenH264) → Constrained Baseline (NVENC); NVENC at 720p30 before: High.

Also: "Prefer hardware" 1080p60 after: NVENC, 3601 received, 0 mixed. NVDEC receiving the level-4.2 stream (LK_DISABLE_NVDEC unset, 30 s): 1806 received, 0 mixed.

Not verified

  • Chromium/Electron and Android receivers of a level 4.2/5.1 Constrained Baseline stream under a 42e01f offer. That is what Chromium's own hardware senders produce, but nothing here decoded it except our own receive path (FFmpeg in libwebrtc, and NVDEC).
  • H.265 NVENC through the session (published as VP8 instead, see above).
  • The fallback notice was not looked at in the running app (core test only); the modal is the existing one with new text.
  • VAAPI (no VAAPI encoder on this host).

Tests

  • third_party/webrtc-sys: h264_level::tests (Table A-1 table, side limit, High bitrate factor, beyond the table), rtp_sender::tests::fixed_selector_falls_back_from_broken_hardware, …_keeps_asking_for_pre_encoded; run by vendored-tests.
  • fluxer-core stream_golive: prefer_hardware_on_the_software_encoder_says_so_once, software_encoder_is_no_news_unless_hardware_was_preferred_and_listed.
  • sender_consistency (ignored, env-gated as before; CI has no GPU): SC_CODEC, per-thread CPU, FLUXER_H264_DUMP.

Gate

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci on this machine (NVIDIA GPU), built with the CUDA headers (CUDA_HOME, as CI and release do) and without LK_DISABLE_NVDEC=1: ==> xtask ci: all steps passed (livekit-interop skipped: FLUXER_LIVEKIT_TESTS is not 1). vendored-tests now also builds and tests third_party/webrtc-sys (Linux only).

🤖 Generated with Claude Code

Closes #764 Measured on Charles's machine (RTX 5090, driver 620.70.06), headless, with `tools/capture-consistency/run.sh --chain` (#763): the real Wayland capture path, a local `livekit-server --dev`, a second session that decodes (FFmpeg's H.264 decoder in libwebrtc, `LK_DISABLE_NVDEC=1`, unless noted). The NVIDIA codecs were compiled with `cuda.h` from NVIDIA's `cuda-cudart-dev-12-9` package unpacked into a scratch directory (`CUDA_HOME`), as #763 did; removed afterwards. ## 1. NVENC H.264 refused everything above 720p30 (cause verified) `webrtc-sys`'s NVENC H.264 wrapper configured the encoder with the level of the SDP format: the factory offers `profile-level-id=42e01f` (Constrained Baseline, **level 3.1**, max 108 000 macroblocks/s = 1280×720 at 30 fps), and `NvidiaH264EncoderImpl` passed that level to `NV_ENC_CONFIG_H264.level`. The init parameters, printed by the new `NVIDIA H264 NVENC config:` line: - failing (level forced back to the SDP's for this check): `1920x1080 fps=60/1 profile=baseline level=31 (sdp profile-level-id=42e01f) rc=2 avg_bps=5219000 max_bps=0 codec_max_kbps=6000 vbv=434915 gop=4294967295 idr=4294967295 p_interval=1` → `nvEncInitializeEncoder … returned error 8` (`NV_ENC_ERR_INVALID_PARAM`) - working: the same with `level=42`. Cross-check with FFmpeg's `h264_nvenc` on the same GPU: `-level 3.1` at 1280×720@60 and 1920×1080@60 → `InitializeEncoder failed: invalid param (8): Invalid Level`; 3.2 / 4.2 / auto work (auto picks 4.2 for 1080p60, 5.1 for 1440p60); 4.2 at 1440p60 fails the same way. OpenH264 ignores the offered level (it wrote 3.2 / 4.2 / 5.1 into its SPS), which is why "Auto" quietly worked in software. A second bug in the same constructor: the NVENC profile GUID (`nv_profile_guid_`) was never initialised. At 720p30, the one size that started, NVENC sent a **High** profile stream (ffprobe: `profile=High level=31`) under the Constrained Baseline negotiation. **Fix** (chosen: (a), not changing the SDP): `include/livekit/h264_level.h` computes the lowest level of H.264 Table A-1 that admits width × height × fps and the max bitrate (MaxFS, Sqrt(8·MaxFS) per side, MaxMBPS, MaxBR × cpbBrVclFactor), as Chromium's hardware encoders do (`FindValidH264Level`) under the `level-asymmetry-allowed=1` that the factory already offers; NVENC's automatic choice only when nothing in the table fits. The profile is the negotiated one (Constrained Baseline → NVENC Baseline). The bitstream now says `Constrained Baseline` with level 3.2 / 4.2 / 4.2 / 5.1. The code is in the registry crate `webrtc-sys` 0.3.53, so it is vendored under `third_party/webrtc-sys` the way `third_party/livekit` is: first commit = the published `.crate` unchanged (checksum as in `Cargo.lock`), second = the patch, every change marked `fluxer-native patch (PATCHES.md)` and listed with file/line in `third_party/webrtc-sys/PATCHES.md`; `[patch.crates-io]`, `exclude`, `VENDORED_CRATES` (its unit tests run in `vendored-tests`, Linux). **H.265 and AV1:** not the same line of code. H.265 never sets a level (the preset's = automatic), AV1 asks for `NV_ENC_LEVEL_AV1_AUTOSELECT`. AV1 NVENC started and was received at 1920×1080@60 (1801 frames in 30 s) and 2560×1440@60 (1800), 0 mixed. H.265 could not be measured end to end: the test share asked for H.265 was published as VP8 (`libvpx`) by the session, so NVENC H.265 never ran; FFmpeg's `hevc_nvenc` with the automatic level works at both sizes. Unchanged. ## 2. "Prefer hardware" sent nothing With a backend other than "auto", the sender gets the SDK's `FixedVideoEncoderSelector`, whose `OnEncoderBroken` asks for the same backend again. NVENC broke twice and libwebrtc stopped trying: 0 frames in 30 s (reproduced: `before-720p60-hw`, received frames=0). **Fix:** a hardware backend (hardware/nvenc/vaapi/videotoolbox) that breaks after it was asked for explicitly gives way to the software encoder for the rest of that sender (never re-requested on bitrate/resolution changes); pre-encoded stays a hard requirement as upstream. Verified by injecting the old level (local, uncommitted fault injection): `Video encoder backend hardware failed for H264; falling back to the software encoder`, then OpenH264, 1800/1800 frames received at 1080p60, 0 mixed. The user is told once per run: `fluxer-core` shows "Hardware encoder unavailable — The hardware encoder couldn't start for this screen share, so it uses the software encoder. Prefer hardware works like Automatic for now." (the existing voice notice modal) when a "Prefer hardware" share whose codec has a hardware encoder reports a software encoder. The original has no such notice; the wording follows its "Encoder path" description (`ENCODER_PATH_NO_HARDWARE_DESCRIPTOR`). Logs: `voice: screen share published` no longer has `hardware=` (it was the prediction, `true` while OpenH264 ran). `voice: screen share encoder implementation=… hardware=…` (from the statistics) gains `hardware_expected`, and a warning when hardware was expected and software runs. ## 3. Other items of #764 - `Nvidia Decoder is supported.` on stdout (`nvidia_decoder_factory.cpp:157`) now goes to the log. The gate below ran on this NVIDIA host with the NVIDIA codecs compiled in and without `LK_DISABLE_NVDEC=1`, so `appimage_has_flaxative_desktop_icon_and_binary` passes here now. - README vs `release.yaml`: the README was stale. `release.yaml` (and CI) install `cuda-cudart-dev-12-9`, so the Linux release AppImage has NVENC/NVDEC (it reports `hardware_encoders: ["nvenc", …]` on an NVIDIA host). README corrected; a stale comment in `release.yaml` ("NVENC is not built here") too. - 1920×1046 received as 1920×1044: **not fixed here.** Measured: the frames handed to the session are 1046 tall, but the encoder is already configured with 1044 (`NVIDIA H264 NVENC config: 1920x1044`) and OpenH264's SPS says 1044 as well, so the two rows go in libwebrtc's frame adaptation before the encoder (not the decoder, not NVENC). Needs its own issue. ## Measurements (60 s each, Auto, H.264, 6 Mbit/s) Before = `textchat` + the published `webrtc-sys` (only the dump diagnostic added); after = this branch. "Encoder thread" is libwebrtc's `VideoEncoderQue` thread CPU over the 60 s; for software the same build with "Prefer software" (OpenH264, the encoder "before" fell back to). Process CPU includes the test's own row checkers (debug build, ~16–37 s of each total), capture, conversion and the viewer's decoding. | size @60 | before: encoder | before: sent / received / mixed | before: process CPU | after: encoder | after: sent / received / mixed | after: process CPU | encoder thread: OpenH264 → NVENC | SPS level before → after | |---|---|---|---|---|---|---|---|---| | 1280×720 | OpenH264 (NVENC error 8) | 3661 / 3600 / 0 | 33.8 s (0.56 cores) | NVENC | 3635 / 3604 / 0 | 25.9 s (0.43) | 7.8 s → 1.0 s | 3.2 → 3.2 | | 1920×1080 | OpenH264 (NVENC error 8) | 3664 / 3600 / 0 | 66.1 s (1.10) | NVENC | 3639 / 3603 / 0 | 51.0 s (0.85) | 19.1 s → 1.5 s | 4.2 → 4.2 | | 1920×1046 | OpenH264 (NVENC error 8) | 3666 / 3600 (as 1044) / 0 | 62.5 s (1.04) | NVENC | 3653 / 3594 (as 1044) / 0 | 53.0 s (0.88) | 17.3 s → 1.7 s | 4.2 → 4.2 | | 2560×1440 | OpenH264 (NVENC error 8) | 3663 / 3598 / 0 | 97.4 s (1.62) | NVENC | 3640 / 3603 / 0 | 81.2 s (1.35) | 31.5 s → 2.0 s | 5.1 → 5.1 | ("sent" = frames handed to the encoder, the converted stage; mixed = 0 at raw, converted and received in every run.) Levels were read with ffprobe from the dumped stream (`FLUXER_H264_DUMP`); before, it is OpenH264's own level, after, NVENC's. Profile before/after: Constrained Baseline (OpenH264) → Constrained Baseline (NVENC); NVENC at 720p30 before: High. Also: "Prefer hardware" 1080p60 after: NVENC, 3601 received, 0 mixed. NVDEC receiving the level-4.2 stream (`LK_DISABLE_NVDEC` unset, 30 s): 1806 received, 0 mixed. ## Not verified - Chromium/Electron and Android receivers of a level 4.2/5.1 Constrained Baseline stream under a 42e01f offer. That is what Chromium's own hardware senders produce, but nothing here decoded it except our own receive path (FFmpeg in libwebrtc, and NVDEC). - H.265 NVENC through the session (published as VP8 instead, see above). - The fallback notice was not looked at in the running app (core test only); the modal is the existing one with new text. - VAAPI (no VAAPI encoder on this host). ## Tests - `third_party/webrtc-sys`: `h264_level::tests` (Table A-1 table, side limit, High bitrate factor, beyond the table), `rtp_sender::tests::fixed_selector_falls_back_from_broken_hardware`, `…_keeps_asking_for_pre_encoded`; run by `vendored-tests`. - `fluxer-core` `stream_golive`: `prefer_hardware_on_the_software_encoder_says_so_once`, `software_encoder_is_no_news_unless_hardware_was_preferred_and_listed`. - `sender_consistency` (ignored, env-gated as before; CI has no GPU): `SC_CODEC`, per-thread CPU, `FLUXER_H264_DUMP`. ## Gate `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` on this machine (NVIDIA GPU), built with the CUDA headers (`CUDA_HOME`, as CI and release do) and **without** `LK_DISABLE_NVDEC=1`: `==> xtask ci: all steps passed` (livekit-interop skipped: FLUXER_LIVEKIT_TESTS is not 1). `vendored-tests` now also builds and tests `third_party/webrtc-sys` (Linux only). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The crates.io package webrtc-sys-0.3.53.crate (sha256
dbfe2c11c98bfd4839242abf9a4b8ddb16a7ff3cbf87416f40928a06886eb5c7, the checksum
Cargo.lock has), unpacked unchanged, plus a .gitattributes that keeps its files
byte for byte. Not used yet; the next commit patches it in.

Refs #764

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NVENC refused H.264 above 720p30 (nvEncInitializeEncoder error 8, "Invalid
Level"): webrtc-sys's NVENC wrapper configured the level of the SDP offer,
42e01f = level 3.1. It now chooses the lowest level of H.264 Table A-1 that
admits the stream (4.2 for 1080p60, 5.1 for 1440p60), as Chromium's hardware
encoders do under level-asymmetry-allowed=1, and sets the negotiated profile,
which was never set (NVENC produced High under a Constrained Baseline
negotiation). A "Prefer hardware" sender whose hardware encoder breaks falls
back to software instead of asking for it again and sending nothing; the app
says so once ("Hardware encoder unavailable"). NVDEC's probe logs instead of
printing to stdout.

webrtc-sys 0.3.53 is vendored for this (third_party/webrtc-sys/PATCHES.md).
The "screen share published" line no longer claims hardware=; "screen share
encoder" names what runs and warns when hardware was expected. README: the
Linux release packages do have NVENC.

Measured with tools/capture-consistency/run.sh --chain (sender_consistency
gains SC_CODEC, CPU time per thread and FLUXER_H264_DUMP).

Closes #764

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405); the commits are in the branch.

Merged into `textchat` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-17 23:45:20 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.