Hardware encoding on AMD and Intel GPUs (VAAPI): same level fix as NVENC; software only as the last resort #857

Closed
opened 2026-11-18 03:43:52 +00:00 by charles · 1 comment
Owner

Asked by Charles, 2026-11-18: "Can we apply the same fix for the AMD/Intel encoders and use CPU as the last resort?"

Follow-up of #764 (PR #838), which fixed NVENC H.264 refusing to start above 720p30 (the SDP's level 3.1 was passed to the encoder) and made "Prefer hardware" fall back to software instead of sending nothing.

Wanted.

  1. The other hardware encoders get the same treatment: on Linux the VAAPI encoder (AMD via Mesa radeonsi, Intel via iHD/i965) — check whether it takes its H.264 level/profile from the negotiated format and fails or misbehaves above 720p30, and fix it with the shared level selection (h264_level.h); same check for H.265 and AV1 where the hardware supports them. On Windows, establish what hardware encoding exists at all in the client today (NVENC? Media Foundation / AMF / QSV? the fork's hardware-encoder crate is listed in the program overview) and what would be needed; fix what shares the cause, report the rest.
  2. Encoder choice with "Auto": try every available hardware encoder first, in a sensible order (the GPU the session renders/captures on first; a discrete GPU before an integrated one unless that costs a cross-GPU copy — measure, do not assume), and use the software encoder only when none can start or one breaks mid-share; never end with no video. "Prefer hardware" and "Software" keep their meaning. The log line voice: screen share encoder implementation= and the one-time notice from #838 tell the user what is really in use.
  3. A machine with two GPUs (this one has an NVIDIA RTX and an AMD integrated Radeon) must pick deliberately, not by accident of the default VAAPI driver (here vainfo resolves to the NVDEC-only VA driver, which has no encode entrypoint, unless LIBVA_DRIVER_NAME/the render node select the AMD one).
**Asked by Charles, 2026-11-18:** "Can we apply the same fix for the AMD/Intel encoders and use CPU as the last resort?" Follow-up of #764 (PR #838), which fixed NVENC H.264 refusing to start above 720p30 (the SDP's level 3.1 was passed to the encoder) and made "Prefer hardware" fall back to software instead of sending nothing. **Wanted.** 1. The other hardware encoders get the same treatment: on Linux the VAAPI encoder (AMD via Mesa radeonsi, Intel via iHD/i965) — check whether it takes its H.264 level/profile from the negotiated format and fails or misbehaves above 720p30, and fix it with the shared level selection (`h264_level.h`); same check for H.265 and AV1 where the hardware supports them. On Windows, establish what hardware encoding exists at all in the client today (NVENC? Media Foundation / AMF / QSV? the fork's `hardware-encoder` crate is listed in the program overview) and what would be needed; fix what shares the cause, report the rest. 2. Encoder choice with "Auto": try every available hardware encoder first, in a sensible order (the GPU the session renders/captures on first; a discrete GPU before an integrated one unless that costs a cross-GPU copy — measure, do not assume), and use the software encoder only when none can start or one breaks mid-share; never end with no video. "Prefer hardware" and "Software" keep their meaning. The log line `voice: screen share encoder implementation=` and the one-time notice from #838 tell the user what is really in use. 3. A machine with two GPUs (this one has an NVIDIA RTX and an AMD integrated Radeon) must pick deliberately, not by accident of the default VAAPI driver (here `vainfo` resolves to the NVDEC-only VA driver, which has no encode entrypoint, unless `LIBVA_DRIVER_NAME`/the render node select the AMD one).
Author
Owner

In nightly 2026.1118.375 (PR #865): VAAPI hardware encoding for AMD and Intel GPUs, the encoder order (display GPU first, other GPUs next, CPU last) and the "Encoding GPU" setting. Not yet confirmed on a real AMD or Intel machine in a call.

In nightly 2026.1118.375 (PR #865): VAAPI hardware encoding for AMD and Intel GPUs, the encoder order (display GPU first, other GPUs next, CPU last) and the "Encoding GPU" setting. Not yet confirmed on a real AMD or Intel machine in a call.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#857
No description provided.