Hardware H.264 (NVENC) does not start above 720p30: Auto falls back to software silently, Prefer hardware sends no video #764

Closed
opened 2026-11-16 08:38:04 +00:00 by charles · 1 comment
Owner

Found while measuring #589 (PR #763) on an NVIDIA machine (Linux):

  1. NVENC H.264 never starts above 720p30. nvEncInitializeEncoder fails with error 8 at 1920×1080@60, 1920×1046@60 and 1280×720@60; it works at 720p30. Probably because the advertised profile-level-id 42e01f is level 3.1 (inferred, not verified).
    • With encoder "Auto", libwebrtc silently falls back to OpenH264 (software) while the client logs hardware=true. So "hardware encoding" shares at 1080p60 are software-encoded today, with the CPU cost that implies.
    • With "Prefer hardware" no video is sent at all: 0 frames received in 60 s at 720p60.
      Wanted: negotiate a level that fits the resolution and frame rate (4.2 for 1080p60), make "Prefer hardware" fall back with a notice instead of sending nothing, and log the encoder actually in use (PR #763 adds voice: screen share encoder implementation=…).
  2. Nvidia Decoder is supported. is printed to stdout by webrtc-sys (nvidia_decoder_factory.cpp:157) ahead of --debug-info output on NVIDIA hosts when the NVIDIA codecs are compiled in; it breaks the package test appimage_has_flaxative_desktop_icon_and_binary on such a host and pollutes --debug-info. release.yaml installs the CUDA headers, so release AppImages likely print it too.
  3. The top-level README says NVENC is in no release build, but release.yaml installs cuda.h: one of the two is stale.
  4. A 1920×1046 share is received as 1920×1044 (two rows cropped somewhere in encode or decode).
Found while measuring #589 (PR #763) on an NVIDIA machine (Linux): 1. **NVENC H.264 never starts above 720p30.** `nvEncInitializeEncoder` fails with error 8 at 1920×1080@60, 1920×1046@60 and 1280×720@60; it works at 720p30. Probably because the advertised `profile-level-id 42e01f` is level 3.1 (inferred, not verified). - With encoder "Auto", libwebrtc silently falls back to **OpenH264** (software) while the client logs `hardware=true`. So "hardware encoding" shares at 1080p60 are software-encoded today, with the CPU cost that implies. - With "Prefer hardware" **no video is sent at all**: 0 frames received in 60 s at 720p60. Wanted: negotiate a level that fits the resolution and frame rate (4.2 for 1080p60), make "Prefer hardware" fall back with a notice instead of sending nothing, and log the encoder actually in use (PR #763 adds `voice: screen share encoder implementation=…`). 2. **`Nvidia Decoder is supported.` is printed to stdout** by webrtc-sys (`nvidia_decoder_factory.cpp:157`) ahead of `--debug-info` output on NVIDIA hosts when the NVIDIA codecs are compiled in; it breaks the package test `appimage_has_flaxative_desktop_icon_and_binary` on such a host and pollutes `--debug-info`. `release.yaml` installs the CUDA headers, so release AppImages likely print it too. 3. The top-level README says NVENC is in no release build, but `release.yaml` installs `cuda.h`: one of the two is stale. 4. A 1920×1046 share is received as 1920×1044 (two rows cropped somewhere in encode or decode).
Author
Owner

Fixed in nightly 2026.1118.369 (PR #838). Measured on an NVIDIA GPU on Linux; not yet confirmed in a real share. Leftovers: #842.

Fixed in nightly 2026.1118.369 (PR #838). Measured on an NVIDIA GPU on Linux; not yet confirmed in a real share. Leftovers: #842.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#764
No description provided.