Screen share: VAAPI hardware encoding on AMD/Intel fixed; encoders in display-GPU order, software last (#857) #865

Closed
charles wants to merge 0 commits from fix-857-vaapi-hardware-encoding into textchat
Owner

Closes #857

Follows #764 / #838. Measured on Charles's machine with tools/capture-consistency/run.sh --chain (real Wayland capture path, local livekit-server --dev, a second session decoding with FFmpeg's H.264 decoder, LK_DISABLE_NVDEC=1). Two GPUs: an NVIDIA RTX 5090 (renderD128, drives both monitors, boot_vga=1) and the AMD Radeon 610M iGPU of the Ryzen 9 9950X (renderD129, Mesa 26.2.6 radeonsi, VCN). LIBVA_DRIVER_NAME=nvidia is set system-wide. NVENC compiled in with cuda.h from cuda-cudart-dev-12-9 unpacked into the target dir (as #838; removed afterwards). Intel (iHD/i965) is untested: same VAAPI code, another driver.

1. The VAAPI encoder (webrtc-sys, vendored; all changes marked and listed in PATCHES.md)

Read in src/vaapi/*: H.264 only (Constrained Baseline offered, 42e01f), CBR, one slice, IDR every 5 s, I420 uploaded on the CPU into an NV12 surface. Reproduced on the AMD GPU at 1280×720, 1920×1080, 1920×1046 and 2560×1440 at 60 fps (before = textchat with LIBVA_DRIVER_NAME=radeonsi LIVEKIT_PREFERRED_HW_ENCODER=vaapi, the only way to reach VAAPI before). Causes found:

  • Level: level_idc = 41 hard-coded for every stream (4.1 under 1080p60 / 1440p60, which need 4.2 / 5.1). radeonsi does not refuse it (NVENC did, #764). Now the shared h264_minimum_level_idc (#764) via vaapi_h264_level_idc (6.2 when nothing in Table A-1 fits; VAAPI has no auto level), from size, frame rate and the codec's max bitrate. Bitstream now 3.2 / 4.2 / 4.2 / 5.1.
  • Garbage every 5 s (the big one): from the 256th frame after each IDR, where the 8-bit pic_order_cnt_lsb wraps, radeonsi's stream repeated every frame_num (FFmpeg: "illegal short term buffer state detected"); the viewer saw garbage until the next IDR: 105–158 unreadable frames per minute at every size. Making the upstream constants consistent (MaxPicOrderCntLsb was 2 << 8 for an 8-bit field) did not stop it (measured); log2_max_pic_order_cnt_lsb is now 16, which the 5-s IDR period never wraps. POC state moved from function statics (shared by all encoders) into the context; frame_num wraps at MaxFrameNum.
  • CABAC under Constrained Baseline (profile violation): CAVLC now unless Main/High.
  • Rate control: frame rate never given (SPS timing said 30 fps), CBR padded every frame with filler (≈6 Mbit/s of padding on a static screen; WebRTC counted it as overshoot and dropped frames), bitrate updates reached the driver only at the next IDR. Now: frame-rate misc parameter, real time_scale, disable_bit_stuffing, rates re-sent with the next frame after SetRates.
  • Mid-share failure returned WEBRTC_VIDEO_CODEC_ERROR (frame dropped, forever); now ENCODER_FAILURE, so the sender's selector switches encoder.
  • Device / driver: upstream tried renderD128, renderD129 with libva's default driver only, so LIBVA_DRIVER_NAME=nvidia (decode-only NVDEC driver) hid the AMD encoder: on this machine VAAPI was simply "unavailable". Now every render node is tried with the default driver, then with its kernel driver's VA driver (amdgpu→radeonsi, i915/xe→iHD, i965) via vaSetDriverName; an encoder can be opened on a given node (x-fluxer-vaapi-device).
  • H.265 / AV1: webrtc-sys has no VAAPI encoder for them (factory offers H.264 only), though radeonsi here lists HEVC encode. Unchanged.

Before / after (VAAPI on the AMD iGPU, Auto, H.264, 6 Mbit/s cap, 60 s)

@60 fps before: level before: converted / received / mixed before: process CPU · encoder thread after: level after: converted / received / mixed after: process CPU · encoder thread bytes sent in 60 s
1280×720 4.1 3596 / 2168 / 115 28.7 s · 1.9 s 3.2 3682 / 3602 / 0 33.0 s · 2.7 s 45.6 MB → 1.7 MB
1920×1080 4.1 3661 / 2079 / 135 65.5 s · 3.5 s 4.2 3691 / 3572 / 0 65.0 s · 5.0 s 46.1 MB → 3.1 MB
1920×1046 4.1 3571 / 2412 (as 1044) / 158 94.6 s · 4.5 s 4.2 3709 / 3579 (as 1044) / 0 88.0 s · 5.0 s 45.6 MB → 3.0 MB
2560×1440 4.1 3118 / 2198 / 105 145.2 s · 7.2 s 5.1 3702 / 3342 / 0 119.8 s · 8.1 s 44.4 MB → 5.8 MB

Profile Constrained Baseline before and after (ffprobe of FLUXER_H264_DUMP); after: 0 decoder errors. "mixed" at the received stage = frames the viewer could not read (garbage), 0 at raw/converted in every run. Process CPU includes the test's row checkers (debug build) and capture; the machine had a load average of 10–17 from other work during all runs, so received counts and CPU are noisy. The encoder-thread time is higher after because ~1.5× more frames are actually encoded, and VAAPI's I420→NV12 upload is a per-pixel CPU loop in that thread (NVENC at 1080p60: 2.7 s, OpenH264: 27.7 s, same conditions).

2. Which encoder "Auto" uses, in which order

Before: no encoder selector for "auto"; libwebrtc creates the first encoder from the first implementation in GetImplementations() (NVENC, then VAAPI: factory order), and a broken one fell to the next negotiated codec; "hardware" re-asked the first hardware backend (#838 made it fall back to software). The order is now the user's rule, decided in Rust and handed to webrtc-sys before each publish:

  1. the hardware encoder of the GPU the display is bound to, whatever its type;
  2. then the other GPUs' encoders, discrete before (unknown before) integrated;
  3. software last, always, also when the encoder in use breaks mid-share.

Display GPU unknown → discrete, unknown, integrated, and the log says so. "Prefer hardware" follows the same order; "Software" stays software. A GPU picked in the new "Encoding GPU" setting is tried alone, then software (with the one-time notice).

  • Pure rule: fluxer_core::voice::gpu::encoder_order (+ describe_order, picked_gpu, gpu_choice_offered), unit-tested: discrete display + iGPU; hybrid laptop (iGPU display + dGPU); two discrete; display GPU without encoder; display unknown (discrete > unknown > integrated); one GPU of each kind; none → software; picked GPU alone; picked GPU gone / without encoder → automatic.
  • Detection (fluxer_voice::gpu, Linux): DRM cards from sysfs (PCI vendor:device@address = stable id, render node), kind from Vulkan via wgpu (DeviceType, matched by PCI ids; unknown if no match, never from the name), display GPU = the card with connected connectors (boot_vga among several), encoder = NVENC (NVIDIA, if compiled and CUDA starts) or VAAPI H.264 on the card's render node. On this machine: connectors connected only on card1 = RTX (DP-2, DP-3), boot_vga = RTX, libwebrtc's DMA-BUF import (#763) = renderD128 = RTX; all three say the monitors are on the RTX. Classified: RTX discrete + display + nvenc; AMD RADV RAPHAEL_MENDOCINO integrated + vaapi. Unit tests with a fake sysfs tree. Other OSes report no GPU and keep the factory order.
  • webrtc-sys: set_hardware_encoder_order (nvenc, vaapi:<node>), GetImplementations lists the order's backends first, then software, then left-out hardware; OrderedVideoEncoderSelector for "auto"/"hardware" walks the order on OnEncoderBroken, skipping encoders without the codec, then software. Unit tests: ordered_selector_*.
  • Logged once per share at INFO: voice: screen share encoder order gpus=[… kind, display, encoder …] order=[nvenc on NVIDIA GeForce RTX 5090 (display GPU) > vaapi on … (other GPU, integrated) > software (last resort)]; libwebrtc logs Hardware encoder order: … and each step (Hardware encoder nvenc failed for H264; trying vaapi /dev/dri/renderD129). --debug-info prints encoder gpu: … per GPU and encoder order: ….
  • FLUXER_ENCODER_GPU=<id or render node> picks a GPU like the setting (diagnostic, documented in the README only); SC_ENCODER_GPU in sender_consistency.

Verified end to end (1920×1080@60, H.264)

case order logged encoder that ran converted / received / mixed
Auto, default env (60 s) nvenc, vaapi:renderD129 NVENC (display GPU) 3701 / 3308 / 0
Prefer hardware (30 s) nvenc, vaapi:renderD129 NVENC 1904 / 1610 / 0
Encoding GPU = AMD (60 s, table above) vaapi:renderD129 VAAPI radeonsi 3691 / 3572 / 0
Auto, NVENC made to fail at start* (30 s) nvenc, vaapi:renderD129 nvenc failed … trying vaapi → VAAPI 1918 / 1766 / 0
AMD picked, VAAPI made to fail at start* (30 s) vaapi:renderD129 … falling back to the software encoder → OpenH264 1907 / 1780 / 0
AMD picked, VAAPI made to fail mid-share* after 600 frames (30 s) vaapi:renderD129 VAAPI, then OpenH264 1922 / 1787 / 0
Prefer software (60 s) — OpenH264 (encoder thread 27.7 s) 3632 / 3248 / 0

* local, uncommitted fault injection (an env check returning an error from InitEncode / ENCODER_FAILURE from Encode), removed before commit.

Cross-GPU copy (for information; the default follows the rule, not this)

Verified in the code: there is no zero-copy path. Wayland frames reach us from libwebrtc's PipeWire capturer either as MemFd (CPU) or as DMA-BUF imported on the first DRM device (renderD128 = RTX here) and read back with glReadPixels into system memory; fluxer-capture converts BGRA→I420 on the CPU and fluxer-voice copies it into an I420Buffer. NVENC then uploads from host memory (cuMemcpy2D), VAAPI copies into a derived image on the CPU. So encoding on the GPU that does not hold the frame costs no GPU-to-GPU copy; the difference is the upload. DMA-BUF source (run.sh --dmabuf, frames rendered on the RTX), 1080p60, 60 s:

encoder converted / received / mixed process CPU encoder thread capture thread
NVENC (RTX, same GPU as the frames) 3636 / 3456 / 0 75.6 s 2.7 s 16.2 s
VAAPI (AMD iGPU) 3743 / 3098 / 0 102.8 s 5.9 s 26.8 s

The encoder-thread difference (≈3 s/min) is VAAPI's CPU upload loop; the capture-thread difference is before the encoder and is load noise (the capture pulled 4149 vs 3817 raw frames).

Settings: "Encoding GPU" (deliberate addition)

Advanced → Media → Configure, under "Content hint", only with two GPUs or more that encode: "Automatic" or one GPU labelled with name and type ("NVIDIA GeForce RTX 5090 (discrete, display)"). Stored by id (vendor:device@pci-address) in the local settings file (screen_share_encoder_gpu), not synced; a stored GPU that is gone behaves and shows as Automatic (the stored value is kept). Help text states the rule. Recorded in docs/parity/settings.toml and docs/parity/screenshare.md as an addition the original does not have. A picked GPU that falls back says so once ("The encoder of the GPU picked under Encoding GPU couldn't start …"). Golden settings_encoder_controls_gpus (lavapipe, docker fluxer-native-shots:u2204).

3. Windows (by reading; nothing to run here)

  • webrtc-sys on Windows has no hardware encoder: NVENC is only compiled in the Linux branch of build.rs, the VAAPI-on-D3D12 files are commented out, and the prebuilt libwebrtc factory is software only (libvpx, OpenH264, libaom).
  • fluxer-voice's mf-encoder feature (MediaFoundation H.264 MFT: NVENC/AMF/QSV through the pre-encoded path) is off by default and not enabled by CI or release.yaml: Windows releases encode in software only.
  • Level bug: does not apply there. The MFT gets MF_MT_MPEG2_PROFILE Baseline but no level, so the MFT picks it.
  • GPU choice: the MFT is the first by MFTEnumEx merit, not the display GPU. Following the rule would need the captured output's adapter LUID (DXGI output → adapter) matched to MFT_ENUM_ADAPTER_LUID, then enabling mf-encoder after a run on Windows hardware (#229). Not changed (nothing shares the cause; untestable here). fluxer_voice::gpu reports no GPUs on Windows; --debug-info already lists wgpu's adapters with their type.

Not verified

  • Intel VAAPI (no Intel GPU here). Hybrid laptops (rule unit-tested only). Two NVIDIA GPUs: NVENC uses CUDA device 0, which may not be the display GPU.
  • Chromium/Electron receivers of the VAAPI stream (only our FFmpeg-in-libwebrtc receive path decoded it).
  • The "Encoding GPU" row and the new notice in the running app (headless UI test and golden only).

Follow-ups (not fixed here)

  • VAAPI upload is a per-pixel CPU loop in the encoder thread (≈5 s/min at 1080p60 vs ≈2.7 s for NVENC); libyuv I420ToNV12 or vaPutImage would cut it.
  • No VAAPI H.265/AV1 encoder in webrtc-sys although radeonsi encodes HEVC.
  • The VAAPI encoder still sends an IDR every 5 s (NVENC: infinite GOP, keyframes on request).
  • 1920×1046 still received as 1920×1044 (found in #838, frame adaptation before the encoder).

Tests

  • webrtc-sys (vendored-tests): rtp_sender::tests::ordered_selector_walks_hardware_then_software, …_skips_encoders_without_the_codec, …_without_hardware_is_software; h264_level::tests::vaapi_level_is_the_minimum_level_or_the_highest.
  • fluxer-core: voice::gpu::tests (11 ordering cases), stream_params::encoding_gpu_is_stored_by_id_and_used_only_when_present, stream_golive::a_picked_gpu_on_the_software_encoder_says_so_once.
  • fluxer-voice: gpu::linux::tests (display card rule, fake sysfs).
  • fluxer-ui: settings_screenshare::encoding_gpu_row_offers_the_gpus_that_encode, screenshot settings_encoder_controls_gpus_matches_golden.
  • fluxer (app): debug_info::tests::report_lists_the_encoder_gpus_and_their_order.
  • Opt-in, real hardware (FLUXER_HW_ENCODER_TESTS=1, ignored on CI): hw_encoder::vaapi_encodes_1080p60_on_every_vaapi_gpu (new) and vaapi_reports_hardware_and_publishes (now on the VAAPI GPU explicitly): both pass here with LIBVA_DRIVER_NAME=nvidia still set. sender_consistency prints the GPUs and takes SC_ENCODER_GPU.

Gate

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci on this machine (NVIDIA + AMD GPUs, LIBVA_DRIVER_NAME=nvidia set, built with the CUDA headers as CI and release do), last lines:

==> xtask ci: livekit-interop
    skipped (FLUXER_LIVEKIT_TESTS is not 1)
==> xtask ci: all steps passed

The new golden was rendered on lavapipe in fluxer-native-shots:u2204 (as the other goldens) and passes in the gate. Windows CI not run locally: the Windows-only paths add no code (fluxer_voice::gpu reports no GPUs there; webrtc-sys/wgpu are Linux-only dependencies of fluxer-voice; the new C++ file compiles its VAAPI/NVENC parts only under their USE_* defines).

🤖 Generated with Claude Code

Closes #857 Follows #764 / #838. Measured on Charles's machine with `tools/capture-consistency/run.sh --chain` (real Wayland capture path, local `livekit-server --dev`, a second session decoding with FFmpeg's H.264 decoder, `LK_DISABLE_NVDEC=1`). Two GPUs: an **NVIDIA RTX 5090** (`renderD128`, drives both monitors, `boot_vga=1`) and the **AMD Radeon 610M** iGPU of the Ryzen 9 9950X (`renderD129`, Mesa 26.2.6 radeonsi, VCN). `LIBVA_DRIVER_NAME=nvidia` is set system-wide. NVENC compiled in with `cuda.h` from `cuda-cudart-dev-12-9` unpacked into the target dir (as #838; removed afterwards). **Intel (iHD/i965) is untested**: same VAAPI code, another driver. ## 1. The VAAPI encoder (webrtc-sys, vendored; all changes marked and listed in `PATCHES.md`) Read in `src/vaapi/*`: H.264 only (Constrained Baseline offered, `42e01f`), CBR, one slice, IDR every 5 s, I420 uploaded on the CPU into an NV12 surface. Reproduced on the AMD GPU at 1280×720, 1920×1080, 1920×1046 and 2560×1440 at 60 fps (before = `textchat` with `LIBVA_DRIVER_NAME=radeonsi LIVEKIT_PREFERRED_HW_ENCODER=vaapi`, the only way to reach VAAPI before). Causes found: - **Level**: `level_idc = 41` hard-coded for every stream (4.1 under 1080p60 / 1440p60, which need 4.2 / 5.1). radeonsi does not refuse it (NVENC did, #764). Now the shared `h264_minimum_level_idc` (#764) via `vaapi_h264_level_idc` (6.2 when nothing in Table A-1 fits; VAAPI has no auto level), from size, frame rate and the codec's max bitrate. Bitstream now 3.2 / 4.2 / 4.2 / 5.1. - **Garbage every 5 s (the big one)**: from the 256th frame after each IDR, where the 8-bit `pic_order_cnt_lsb` wraps, radeonsi's stream repeated every `frame_num` (FFmpeg: "illegal short term buffer state detected"); the viewer saw garbage until the next IDR: **105–158 unreadable frames per minute at every size**. Making the upstream constants consistent (`MaxPicOrderCntLsb` was `2 << 8` for an 8-bit field) did not stop it (measured); `log2_max_pic_order_cnt_lsb` is now 16, which the 5-s IDR period never wraps. POC state moved from function statics (shared by all encoders) into the context; `frame_num` wraps at `MaxFrameNum`. - **CABAC under Constrained Baseline** (profile violation): CAVLC now unless Main/High. - **Rate control**: frame rate never given (SPS timing said 30 fps), CBR padded every frame with filler (≈6 Mbit/s of padding on a static screen; WebRTC counted it as overshoot and dropped frames), bitrate updates reached the driver only at the next IDR. Now: frame-rate misc parameter, real `time_scale`, `disable_bit_stuffing`, rates re-sent with the next frame after `SetRates`. - **Mid-share failure** returned `WEBRTC_VIDEO_CODEC_ERROR` (frame dropped, forever); now `ENCODER_FAILURE`, so the sender's selector switches encoder. - **Device / driver**: upstream tried `renderD128`, `renderD129` with libva's default driver only, so `LIBVA_DRIVER_NAME=nvidia` (decode-only NVDEC driver) hid the AMD encoder: on this machine VAAPI was simply "unavailable". Now every render node is tried with the default driver, then with its kernel driver's VA driver (`amdgpu`→radeonsi, `i915`/`xe`→iHD, i965) via `vaSetDriverName`; an encoder can be opened on a given node (`x-fluxer-vaapi-device`). - H.265 / AV1: webrtc-sys has **no** VAAPI encoder for them (factory offers H.264 only), though radeonsi here lists HEVC encode. Unchanged. ### Before / after (VAAPI on the AMD iGPU, Auto, H.264, 6 Mbit/s cap, 60 s) | @60 fps | before: level | before: converted / received / mixed | before: process CPU · encoder thread | after: level | after: converted / received / mixed | after: process CPU · encoder thread | bytes sent in 60 s | |---|---|---|---|---|---|---|---| | 1280×720 | 4.1 | 3596 / 2168 / **115** | 28.7 s · 1.9 s | 3.2 | 3682 / 3602 / 0 | 33.0 s · 2.7 s | 45.6 MB → 1.7 MB | | 1920×1080 | 4.1 | 3661 / 2079 / **135** | 65.5 s · 3.5 s | 4.2 | 3691 / 3572 / 0 | 65.0 s · 5.0 s | 46.1 MB → 3.1 MB | | 1920×1046 | 4.1 | 3571 / 2412 (as 1044) / **158** | 94.6 s · 4.5 s | 4.2 | 3709 / 3579 (as 1044) / 0 | 88.0 s · 5.0 s | 45.6 MB → 3.0 MB | | 2560×1440 | 4.1 | 3118 / 2198 / **105** | 145.2 s · 7.2 s | 5.1 | 3702 / 3342 / 0 | 119.8 s · 8.1 s | 44.4 MB → 5.8 MB | Profile Constrained Baseline before and after (ffprobe of `FLUXER_H264_DUMP`); after: 0 decoder errors. "mixed" at the received stage = frames the viewer could not read (garbage), 0 at raw/converted in every run. Process CPU includes the test's row checkers (debug build) and capture; the machine had a load average of 10–17 from other work during all runs, so received counts and CPU are noisy. The encoder-thread time is higher after because ~1.5× more frames are actually encoded, and VAAPI's I420→NV12 upload is a per-pixel CPU loop in that thread (NVENC at 1080p60: 2.7 s, OpenH264: 27.7 s, same conditions). ## 2. Which encoder "Auto" uses, in which order Before: no encoder selector for "auto"; libwebrtc creates the first encoder from the **first implementation** in `GetImplementations()` (NVENC, then VAAPI: factory order), and a broken one fell to the next negotiated codec; "hardware" re-asked the first hardware backend (#838 made it fall back to software). The order is now the user's rule, decided in Rust and handed to webrtc-sys before each publish: 1. the hardware encoder of the GPU the **display is bound to**, whatever its type; 2. then the other GPUs' encoders, **discrete before (unknown before) integrated**; 3. **software last, always**, also when the encoder in use breaks mid-share. Display GPU unknown → discrete, unknown, integrated, and the log says so. "Prefer hardware" follows the same order; "Software" stays software. A GPU picked in the new **"Encoding GPU"** setting is tried alone, then software (with the one-time notice). - Pure rule: `fluxer_core::voice::gpu::encoder_order` (+ `describe_order`, `picked_gpu`, `gpu_choice_offered`), unit-tested: discrete display + iGPU; hybrid laptop (iGPU display + dGPU); two discrete; display GPU without encoder; display unknown (discrete > unknown > integrated); one GPU of each kind; none → software; picked GPU alone; picked GPU gone / without encoder → automatic. - Detection (`fluxer_voice::gpu`, Linux): DRM cards from sysfs (PCI vendor:device@address = stable id, render node), **kind from Vulkan via wgpu** (`DeviceType`, matched by PCI ids; unknown if no match, never from the name), **display GPU = the card with connected connectors** (`boot_vga` among several), encoder = NVENC (NVIDIA, if compiled and CUDA starts) or VAAPI H.264 on the card's render node. On this machine: connectors connected only on card1 = RTX (`DP-2`, `DP-3`), `boot_vga` = RTX, libwebrtc's DMA-BUF import (#763) = `renderD128` = RTX; all three say the monitors are on the RTX. Classified: RTX discrete + display + nvenc; AMD `RADV RAPHAEL_MENDOCINO` integrated + vaapi. Unit tests with a fake sysfs tree. Other OSes report no GPU and keep the factory order. - webrtc-sys: `set_hardware_encoder_order` (`nvenc`, `vaapi:<node>`), `GetImplementations` lists the order's backends first, then software, then left-out hardware; `OrderedVideoEncoderSelector` for "auto"/"hardware" walks the order on `OnEncoderBroken`, skipping encoders without the codec, then software. Unit tests: `ordered_selector_*`. - Logged once per share at INFO: `voice: screen share encoder order gpus=[… kind, display, encoder …] order=[nvenc on NVIDIA GeForce RTX 5090 (display GPU) > vaapi on … (other GPU, integrated) > software (last resort)]`; libwebrtc logs `Hardware encoder order: …` and each step (`Hardware encoder nvenc failed for H264; trying vaapi /dev/dri/renderD129`). `--debug-info` prints `encoder gpu: …` per GPU and `encoder order: …`. - `FLUXER_ENCODER_GPU=<id or render node>` picks a GPU like the setting (diagnostic, documented in the README only); `SC_ENCODER_GPU` in `sender_consistency`. ### Verified end to end (1920×1080@60, H.264) | case | order logged | encoder that ran | converted / received / mixed | |---|---|---|---| | Auto, default env (60 s) | nvenc, vaapi:renderD129 | NVENC (display GPU) | 3701 / 3308 / 0 | | Prefer hardware (30 s) | nvenc, vaapi:renderD129 | NVENC | 1904 / 1610 / 0 | | Encoding GPU = AMD (60 s, table above) | vaapi:renderD129 | VAAPI radeonsi | 3691 / 3572 / 0 | | Auto, NVENC made to fail at start* (30 s) | nvenc, vaapi:renderD129 | `nvenc failed … trying vaapi` → VAAPI | 1918 / 1766 / 0 | | AMD picked, VAAPI made to fail at start* (30 s) | vaapi:renderD129 | `… falling back to the software encoder` → OpenH264 | 1907 / 1780 / 0 | | AMD picked, VAAPI made to fail mid-share* after 600 frames (30 s) | vaapi:renderD129 | VAAPI, then OpenH264 | 1922 / 1787 / 0 | | Prefer software (60 s) | — | OpenH264 (encoder thread 27.7 s) | 3632 / 3248 / 0 | \* local, uncommitted fault injection (an env check returning an error from `InitEncode` / `ENCODER_FAILURE` from `Encode`), removed before commit. ### Cross-GPU copy (for information; the default follows the rule, not this) Verified in the code: there is no zero-copy path. Wayland frames reach us from libwebrtc's PipeWire capturer either as MemFd (CPU) or as DMA-BUF imported on the **first DRM device** (`renderD128` = RTX here) and read back with `glReadPixels` into system memory; `fluxer-capture` converts BGRA→I420 on the CPU and `fluxer-voice` copies it into an `I420Buffer`. NVENC then uploads from host memory (`cuMemcpy2D`), VAAPI copies into a derived image on the CPU. So encoding on the GPU that does not hold the frame costs no GPU-to-GPU copy; the difference is the upload. DMA-BUF source (`run.sh --dmabuf`, frames rendered on the RTX), 1080p60, 60 s: | encoder | converted / received / mixed | process CPU | encoder thread | capture thread | |---|---|---|---|---| | NVENC (RTX, same GPU as the frames) | 3636 / 3456 / 0 | 75.6 s | 2.7 s | 16.2 s | | VAAPI (AMD iGPU) | 3743 / 3098 / 0 | 102.8 s | 5.9 s | 26.8 s | The encoder-thread difference (≈3 s/min) is VAAPI's CPU upload loop; the capture-thread difference is before the encoder and is load noise (the capture pulled 4149 vs 3817 raw frames). ### Settings: "Encoding GPU" (deliberate addition) Advanced → Media → Configure, under "Content hint", only with two GPUs or more that encode: "Automatic" or one GPU labelled with name and type ("NVIDIA GeForce RTX 5090 (discrete, display)"). Stored by id (`vendor:device@pci-address`) in the local settings file (`screen_share_encoder_gpu`), not synced; a stored GPU that is gone behaves and shows as Automatic (the stored value is kept). Help text states the rule. Recorded in `docs/parity/settings.toml` and `docs/parity/screenshare.md` as an addition the original does not have. A picked GPU that falls back says so once ("The encoder of the GPU picked under Encoding GPU couldn't start …"). Golden `settings_encoder_controls_gpus` (lavapipe, docker `fluxer-native-shots:u2204`). ## 3. Windows (by reading; nothing to run here) - webrtc-sys on Windows has **no hardware encoder**: NVENC is only compiled in the Linux branch of `build.rs`, the VAAPI-on-D3D12 files are commented out, and the prebuilt libwebrtc factory is software only (libvpx, OpenH264, libaom). - `fluxer-voice`'s `mf-encoder` feature (MediaFoundation H.264 MFT: NVENC/AMF/QSV through the pre-encoded path) is **off by default and not enabled by CI or `release.yaml`**: Windows releases encode in software only. - Level bug: does not apply there. The MFT gets `MF_MT_MPEG2_PROFILE` Baseline but no level, so the MFT picks it. - GPU choice: the MFT is the first by `MFTEnumEx` merit, not the display GPU. Following the rule would need the captured output's adapter LUID (DXGI output → adapter) matched to `MFT_ENUM_ADAPTER_LUID`, then enabling `mf-encoder` after a run on Windows hardware (#229). Not changed (nothing shares the cause; untestable here). `fluxer_voice::gpu` reports no GPUs on Windows; `--debug-info` already lists wgpu's adapters with their type. ## Not verified - Intel VAAPI (no Intel GPU here). Hybrid laptops (rule unit-tested only). Two NVIDIA GPUs: NVENC uses CUDA device 0, which may not be the display GPU. - Chromium/Electron receivers of the VAAPI stream (only our FFmpeg-in-libwebrtc receive path decoded it). - The "Encoding GPU" row and the new notice in the running app (headless UI test and golden only). ## Follow-ups (not fixed here) - VAAPI upload is a per-pixel CPU loop in the encoder thread (≈5 s/min at 1080p60 vs ≈2.7 s for NVENC); libyuv `I420ToNV12` or `vaPutImage` would cut it. - No VAAPI H.265/AV1 encoder in webrtc-sys although radeonsi encodes HEVC. - The VAAPI encoder still sends an IDR every 5 s (NVENC: infinite GOP, keyframes on request). - 1920×1046 still received as 1920×1044 (found in #838, frame adaptation before the encoder). ## Tests - webrtc-sys (`vendored-tests`): `rtp_sender::tests::ordered_selector_walks_hardware_then_software`, `…_skips_encoders_without_the_codec`, `…_without_hardware_is_software`; `h264_level::tests::vaapi_level_is_the_minimum_level_or_the_highest`. - fluxer-core: `voice::gpu::tests` (11 ordering cases), `stream_params::encoding_gpu_is_stored_by_id_and_used_only_when_present`, `stream_golive::a_picked_gpu_on_the_software_encoder_says_so_once`. - fluxer-voice: `gpu::linux::tests` (display card rule, fake sysfs). - fluxer-ui: `settings_screenshare::encoding_gpu_row_offers_the_gpus_that_encode`, screenshot `settings_encoder_controls_gpus_matches_golden`. - fluxer (app): `debug_info::tests::report_lists_the_encoder_gpus_and_their_order`. - Opt-in, real hardware (`FLUXER_HW_ENCODER_TESTS=1`, ignored on CI): `hw_encoder::vaapi_encodes_1080p60_on_every_vaapi_gpu` (new) and `vaapi_reports_hardware_and_publishes` (now on the VAAPI GPU explicitly): both pass here with `LIBVA_DRIVER_NAME=nvidia` still set. `sender_consistency` prints the GPUs and takes `SC_ENCODER_GPU`. ## Gate `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` on this machine (NVIDIA + AMD GPUs, `LIBVA_DRIVER_NAME=nvidia` set, built with the CUDA headers as CI and release do), last lines: ```text ==> xtask ci: livekit-interop skipped (FLUXER_LIVEKIT_TESTS is not 1) ==> xtask ci: all steps passed ``` The new golden was rendered on lavapipe in `fluxer-native-shots:u2204` (as the other goldens) and passes in the gate. Windows CI not run locally: the Windows-only paths add no code (`fluxer_voice::gpu` reports no GPUs there; `webrtc-sys`/`wgpu` are Linux-only dependencies of `fluxer-voice`; the new C++ file compiles its VAAPI/NVENC parts only under their `USE_*` defines). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
webrtc-sys (vendored, PATCHES.md): the VAAPI H.264 encoder announced level
4.1 for every stream, repeated frame_num once its 8-bit picture order count
wrapped (garbage from the 256th frame of every 5-s GOP), wrote CABAC under
Constrained Baseline, never gave the driver the frame rate and padded CBR
with filler, and dropped every frame instead of failing over when encoding
failed. Render nodes are now tried with the kernel driver's VA driver too,
so LIBVA_DRIVER_NAME=nvidia no longer hides an AMD/Intel encoder. The
application sets the hardware encoder order; "auto"/"hardware" senders walk
it (OrderedVideoEncoderSelector) and end with the software encoder.

fluxer-core: voice::gpu, the order rule (display GPU's encoder first, then
discrete before integrated, software last; a picked GPU alone), and the
local "Encoding GPU" setting (stored by PCI id). fluxer-voice: gpu, the
GPUs from DRM/sysfs with their kind from Vulkan (wgpu) and the display GPU
from the connected connectors; the order is installed and logged once per
share. fluxer-ui: "Encoding GPU" in the encoder controls dialog (two GPUs
or more). --debug-info lists the GPUs and the order.

Refs #857

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405); the commits are in the branch.

Merged into `textchat` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-18 05:24:30 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.