Screen share: trust a Wayland frame copy only while libwebrtc still points at it (#589) #763

Closed
charles wants to merge 0 commits from fix-589-stream-top-strip-encode into textchat
Owner

Closes #589

Second round of #589, after the first fix (#607). What was measured below comes from headless runs on Charles's machine (RTX 5090, driver 620.70, PipeWire 1.6.11), driving the real Wayland capture path. The real KWin session was not driven. Anything about it is inferred and labelled as such.

Where the strip comes from (measured)

The defect is on the sending side, in capture. It arises when the capture thread copies libwebrtc's shared frame. It is not in the conversion and not in the encoder.

libwebrtc's PipeWire capturer keeps two frames and writes the next picture but one into the frame it handed us, even while we are reading it (#607). #607 copies the frame and compares the copy with the frame. That misses a writer that stalls in the middle of the buffer. With DMA-BUF buffers, libwebrtc fills the frame with glReadPixels on the NVIDIA EGL device (renderD128). That writer is slow, and under CPU pressure it gets descheduled partway through. While it is stalled, the bytes hold still during both our copy and our comparison, so the comparison passes. The frame that is handed on then has the top rows of a picture 2–5 frames later over the rest. The conversion and the encoder carry it faithfully to every viewer. That matches "1/5 of the next frame too quickly".

Pipeline for this configuration (measured or read in source)

  • PipeWire: KWin 6.8.2 offers DMA-BUF. On NVIDIA it calls glFinish() before queueing a buffer, because it treats implicit sync as broken there (screencaststream.cpp), so buffers are complete when they are queued. libwebrtc imports them through EGL/GBM on the first DRM device, which is renderD128 = NVIDIA here. It calls glReadPixels into one of its two DesktopFrames, returns the pw_buffer right after, and shares the frame with no copy. With run.sh --dmabuf, the negotiated modifier was 0x300000000606010 (NVIDIA block-linear).
  • Capture thread: copy plus compare (stable_copy), then libyuv BGRA→I420 into a new buffer each frame, held as immutable Arc planes. fluxer-voice to_rtc_buffer copies those into a new I420Buffer for each frame.
  • Encoder: "Auto" with hardware=true picks NVENC. On this GPU, NVENC fails to start for H.264 at 1920×1080@60, 1920×1046@60 and 1280×720@60 (nvEncInitializeEncoder error 8). It works at 1280×720@30, which suggests the level 3.1 in the advertised profile-level-id 42e01f (inferred). libwebrtc then switches the encoder, and the share runs on OpenH264. So Charles's 1080p/1046p shares are encoded in software, even though the log says hardware=true. NVENC itself copies the I420 into its surface with a synchronous cuMemcpy2D and encodes with an output delay of 0. Measured at 720p30, it delivered 0 mixed frames out of 1801.

Measurements: tools/capture-consistency/run.sh --chain [--dmabuf]

--chain is new. The source's every row carries its frame number (RowCounter, Gray-coded, 24 cells). fluxer-voice tests/sender_consistency.rs checks every frame row by row at three stages:

  • raw: our copy of libwebrtc's frame, every copy including the ones we drop.
  • converted: the I420 handed to the session and the encoder.
  • received: frames decoded by a second session through a local livekit-server --dev, with LK_DISABLE_NVDEC=1, so FFmpeg's H.264 decoder inside libwebrtc.

--dmabuf replaces the MemFd source with dmabuf_source.c. It renders the frames with GL into DMA-BUFs and calls glFinish before queueing, as KWin does on NVIDIA. H.264 at 6 Mbit/s, 60 fps.

configuration fix raw mixed (torn / moved on) converted mixed received mixed
MemFd 1080p60, 60 s ×2 (Auto, software) #607 only 0 0 / 3675, 0 / 3675 0 / 3456, 0 / 3358
MemFd 1046p60, 60 s ×2 #607 only 0 0 / 3653, 0 / 3610 0 / 3569, 0 / 3254 (received as 1920×1044)
NVENC 720p30, 60 s #607 only 0 0 / 1856 0 / 1801
DMA-BUF 1080p60, idle-ish, 120 s #607 only 17 (all torn, caught) 0 / 7348 0 / 7073
DMA-BUF 1080p60, pinned to 3 CPUs, 120 s #607 only 53 6 / 7118 6 / 6436
DMA-BUF 1080p60, all cores oversubscribed, 300 s #607 only 38 7 / 18076 5 / 16376
DMA-BUF 1080p60, pinned to 3 CPUs, 120 s this PR 108 (148 torn, 646 moved on) 0 / 6676 0 / 6023
DMA-BUF 1046p60, pinned to 3 CPUs, 120 s this PR 38 (48 / 575) 0 / 7281 0 / 6405
DMA-BUF 1080p60, no synthetic load, 120 s ×2 this PR 51, 16 0 / 7086, 0 / 7255 0 / 5929, 0 / 6847
DMA-BUF 1046p60, no synthetic load, 120 s this PR 16 0 / 7220 0 / 6522
MemFd 1080p60, no synthetic load, 120 s this PR 0 0 / 7169 0 / 6217

"#607 only" means this branch with FLUXER_CAPTURE_VERIFY_LATEST=0, which is the code path of #607. The mixed frames are clean row splits, for example 0..100=2842 100..1080=2840, and the top picture is always the later one (distance 2 in most cases, up to 5). Every mixed frame found at the converted stage reached the viewer unchanged.

Load dependence: mixed frames got past the comparison only with DMA-BUF and CPU pressure. On a quiet machine the comparison caught all of them. With MemFd at 1080p, none appeared at all. The 1046 height behaves like 1080.

Fix

pull_stable gains a second look on the Wayland portal path. A copy counts only if libwebrtc's latest frame is still the buffer it was copied from once the comparison is done. libwebrtc publishes a buffer as latest only after writing it. While it writes into ours, the latest frame is its other buffer, so a stalled writer is always seen. The rejected copy is pulled again, at most 3 retries, as for torn copies.

  • The check is a second capture_frame() in "peek" mode. On Wayland that only shares the latest frame. The conversion now happens after the decision, outside the callback.
  • X11 and Windows skip the second look, because there a second call would capture a second frame.
  • Diagnostics (trace, probe) run only after the decision, so nothing widens the window between the comparison and the second look. A first version ran the probe inside that window: 9 mixed frames still got through in 300 s. With the probe moved after the decision, the window closed.
  • Cost: the dropped copies are re-pulled. At 1080p60 the converted rate stayed at 59–60 fps without load, and was about 56 fps on 3 pinned CPUs.

Unit test a_copy_of_a_buffer_the_writer_stalls_in_is_pulled_again pins the rule with a scripted two-buffer queue whose writer stalls after 3 rows. With the second look the result is the whole picture. Without it, the mixed copy goes through.

Diagnostics for the real session (FLUXER_FRAME_CHECK=1)

  • Capture, per pull (FLUXER_LOG=info,fluxer_capture=debug): frame check: pull with slot (which libwebrtc buffer), verdict (Trusted / Torn / MovedOn), repeat (no new picture since the last pull) and copy_us. Totals every 600 pulls: side="pulled" pulls repeats torn moved_on slots.
  • Splice check (SpliceCheck: top rows equal to a later frame's, rest not) and the black-top check on frames as captured (side="captured"), as handed to the encoder (side="encoder-input") and as decoded (side="received"). Totals every 600 frames include splices.
  • libwebrtc's own lines at info (frame check: …): the PipeWire format (a Modifier: line means DMA-BUF), dropped or failed buffers, NVENC/VAAPI start-up and failures, encoder switches.
  • voice: screen share encoder implementation=… hardware=… is logged whenever the actually running encoder changes. This is always on.
  • FLUXER_CAPTURE_VERIFY_LATEST=0 turns the fix off for an A/B run.

Not covered

  • The real KWin compositor was not driven. KWin's DMA-BUF path is reproduced by dmabuf_source.c (GPU render, glFinish, queue).
  • Hardware H.264 at the user's sizes could not be measured, because NVENC refuses them on this GPU (see above). NVENC at 720p30 was clean.

Gate

Run on Charles's machine, which has an NVIDIA GPU, with LK_DISABLE_NVDEC=1 FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci. Result: ==> xtask ci: all steps passed (livekit-interop skipped).

Without LK_DISABLE_NVDEC=1, xtask --test package appimage_has_flaxative_desktop_icon_and_binary fails on this host. webrtc-sys's NVDEC probe prints Nvidia Decoder is supported. with std::cout (nvidia_decoder_factory.cpp:157), ahead of the --debug-info report. That happens whenever the build has the NVIDIA codecs and the host has an NVIDIA GPU. It is unrelated to this change; filed below as a follow-up.

The new measurement test (sender_consistency) is #[ignore] and env-gated. The capture unit tests (desktop::tests, 33) and the check tests (9) run in CI.

🤖 Generated with Claude Code

Closes #589 Second round of #589, after the first fix (#607). **What was measured** below comes from headless runs on Charles's machine (RTX 5090, driver 620.70, PipeWire 1.6.11), driving the real Wayland capture path. The real KWin session was **not** driven. Anything about it is **inferred** and labelled as such. ## Where the strip comes from (measured) The defect is on the **sending side, in capture**. It arises when the capture thread copies libwebrtc's shared frame. It is not in the conversion and not in the encoder. libwebrtc's PipeWire capturer keeps two frames and writes the next picture but one into the frame it handed us, even while we are reading it (#607). #607 copies the frame and compares the copy with the frame. That misses a writer that **stalls in the middle of the buffer**. With DMA-BUF buffers, libwebrtc fills the frame with `glReadPixels` on the NVIDIA EGL device (renderD128). That writer is slow, and under CPU pressure it gets descheduled partway through. While it is stalled, the bytes hold still during both our copy and our comparison, so the comparison passes. The frame that is handed on then has the top rows of a picture 2–5 frames later over the rest. The conversion and the encoder carry it faithfully to every viewer. That matches "1/5 of the next frame too quickly". ### Pipeline for this configuration (measured or read in source) - **PipeWire:** KWin 6.8.2 offers DMA-BUF. On NVIDIA it calls `glFinish()` before queueing a buffer, because it treats implicit sync as broken there (`screencaststream.cpp`), so buffers are complete when they are queued. libwebrtc imports them through EGL/GBM on the first DRM device, which is `renderD128` = NVIDIA here. It calls `glReadPixels` into one of its two `DesktopFrame`s, returns the `pw_buffer` right after, and shares the frame with no copy. With `run.sh --dmabuf`, the negotiated modifier was `0x300000000606010` (NVIDIA block-linear). - **Capture thread:** copy plus compare (`stable_copy`), then libyuv BGRA→I420 into a new buffer each frame, held as immutable `Arc` planes. `fluxer-voice` `to_rtc_buffer` copies those into a new `I420Buffer` for each frame. - **Encoder:** "Auto" with `hardware=true` picks NVENC. On this GPU, **NVENC fails to start for H.264 at 1920×1080@60, 1920×1046@60 and 1280×720@60** (`nvEncInitializeEncoder` error 8). It works at 1280×720@30, which suggests the level 3.1 in the advertised `profile-level-id 42e01f` (inferred). libwebrtc then switches the encoder, and the share runs on **OpenH264**. So Charles's 1080p/1046p shares are encoded in software, even though the log says `hardware=true`. NVENC itself copies the I420 into its surface with a synchronous `cuMemcpy2D` and encodes with an output delay of 0. Measured at 720p30, it delivered 0 mixed frames out of 1801. ## Measurements: `tools/capture-consistency/run.sh --chain [--dmabuf]` `--chain` is new. The source's every row carries its frame number (`RowCounter`, Gray-coded, 24 cells). `fluxer-voice` `tests/sender_consistency.rs` checks every frame row by row at three stages: - **raw:** our copy of libwebrtc's frame, every copy including the ones we drop. - **converted:** the I420 handed to the session and the encoder. - **received:** frames decoded by a second session through a local `livekit-server --dev`, with `LK_DISABLE_NVDEC=1`, so FFmpeg's H.264 decoder inside libwebrtc. `--dmabuf` replaces the MemFd source with `dmabuf_source.c`. It renders the frames with GL into DMA-BUFs and calls `glFinish` before queueing, as KWin does on NVIDIA. H.264 at 6 Mbit/s, 60 fps. | configuration | fix | raw mixed (torn / moved on) | converted mixed | received mixed | |---|---|---|---|---| | MemFd 1080p60, 60 s ×2 (Auto, software) | #607 only | 0 | 0 / 3675, 0 / 3675 | 0 / 3456, 0 / 3358 | | MemFd 1046p60, 60 s ×2 | #607 only | 0 | 0 / 3653, 0 / 3610 | 0 / 3569, 0 / 3254 (received as 1920×1044) | | NVENC 720p30, 60 s | #607 only | 0 | 0 / 1856 | 0 / 1801 | | DMA-BUF 1080p60, idle-ish, 120 s | #607 only | 17 (all torn, caught) | **0** / 7348 | 0 / 7073 | | DMA-BUF 1080p60, pinned to 3 CPUs, 120 s | #607 only | 53 | **6** / 7118 | **6** / 6436 | | DMA-BUF 1080p60, all cores oversubscribed, 300 s | #607 only | 38 | **7** / 18076 | **5** / 16376 | | DMA-BUF 1080p60, pinned to 3 CPUs, 120 s | this PR | 108 (148 torn, 646 moved on) | **0** / 6676 | **0** / 6023 | | DMA-BUF 1046p60, pinned to 3 CPUs, 120 s | this PR | 38 (48 / 575) | **0** / 7281 | **0** / 6405 | | DMA-BUF 1080p60, no synthetic load, 120 s ×2 | this PR | 51, 16 | 0 / 7086, 0 / 7255 | 0 / 5929, 0 / 6847 | | DMA-BUF 1046p60, no synthetic load, 120 s | this PR | 16 | 0 / 7220 | 0 / 6522 | | MemFd 1080p60, no synthetic load, 120 s | this PR | 0 | 0 / 7169 | 0 / 6217 | "#607 only" means this branch with `FLUXER_CAPTURE_VERIFY_LATEST=0`, which is the code path of #607. The mixed frames are clean row splits, for example `0..100=2842 100..1080=2840`, and the top picture is always the **later** one (distance 2 in most cases, up to 5). Every mixed frame found at the converted stage reached the viewer unchanged. **Load dependence:** mixed frames got past the comparison only with **DMA-BUF and CPU pressure**. On a quiet machine the comparison caught all of them. With MemFd at 1080p, none appeared at all. The 1046 height behaves like 1080. ## Fix `pull_stable` gains a **second look** on the Wayland portal path. A copy counts only if libwebrtc's latest frame is still *the buffer it was copied from* once the comparison is done. libwebrtc publishes a buffer as latest only after writing it. While it writes into ours, the latest frame is its other buffer, so a stalled writer is always seen. The rejected copy is pulled again, at most 3 retries, as for torn copies. - The check is a second `capture_frame()` in "peek" mode. On Wayland that only shares the latest frame. The conversion now happens after the decision, outside the callback. - X11 and Windows skip the second look, because there a second call would capture a second frame. - Diagnostics (trace, probe) run only after the decision, so nothing widens the window between the comparison and the second look. A first version ran the probe inside that window: 9 mixed frames still got through in 300 s. With the probe moved after the decision, the window closed. - Cost: the dropped copies are re-pulled. At 1080p60 the converted rate stayed at 59–60 fps without load, and was about 56 fps on 3 pinned CPUs. Unit test `a_copy_of_a_buffer_the_writer_stalls_in_is_pulled_again` pins the rule with a scripted two-buffer queue whose writer stalls after 3 rows. With the second look the result is the whole picture. Without it, the mixed copy goes through. ## Diagnostics for the real session (`FLUXER_FRAME_CHECK=1`) - **Capture, per pull** (`FLUXER_LOG=info,fluxer_capture=debug`): `frame check: pull` with `slot` (which libwebrtc buffer), `verdict` (Trusted / Torn / MovedOn), `repeat` (no new picture since the last pull) and `copy_us`. Totals every 600 pulls: `side="pulled" pulls repeats torn moved_on slots`. - **Splice check** (`SpliceCheck`: top rows equal to a later frame's, rest not) and the black-top check on frames as captured (`side="captured"`), as handed to the encoder (`side="encoder-input"`) and as decoded (`side="received"`). Totals every 600 frames include `splices`. - **libwebrtc's own lines** at `info` (`frame check: …`): the PipeWire format (a `Modifier:` line means DMA-BUF), dropped or failed buffers, NVENC/VAAPI start-up and failures, encoder switches. - `voice: screen share encoder implementation=… hardware=…` is logged whenever the actually running encoder changes. This is always on. - `FLUXER_CAPTURE_VERIFY_LATEST=0` turns the fix off for an A/B run. ## Not covered - The real KWin compositor was not driven. KWin's DMA-BUF path is reproduced by `dmabuf_source.c` (GPU render, `glFinish`, queue). - Hardware H.264 at the user's sizes could not be measured, because NVENC refuses them on this GPU (see above). NVENC at 720p30 was clean. ## Gate Run on Charles's machine, which has an NVIDIA GPU, with `LK_DISABLE_NVDEC=1 FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci`. Result: `==> xtask ci: all steps passed` (livekit-interop skipped). Without `LK_DISABLE_NVDEC=1`, `xtask --test package appimage_has_flaxative_desktop_icon_and_binary` fails on this host. webrtc-sys's NVDEC probe prints `Nvidia Decoder is supported.` with `std::cout` (`nvidia_decoder_factory.cpp:157`), ahead of the `--debug-info` report. That happens whenever the build has the NVIDIA codecs and the host has an NVIDIA GPU. It is unrelated to this change; filed below as a follow-up. The new measurement test (`sender_consistency`) is `#[ignore]` and env-gated. The capture unit tests (`desktop::tests`, 33) and the `check` tests (9) run in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Measured where the top strip of a stream from another frame comes from: libwebrtc's
PipeWire capture rewrites the shared frame two buffers later; the copy-and-compare of
#607 misses a writer that stalls inside the buffer (DMA-BUF readback under CPU
pressure). A copy now counts only if libwebrtc's latest frame is still the buffer it
was copied from once the comparison is done (pull_stable's second look, Wayland only).

- tools/capture-consistency: --chain (whole sender, row-counter frames), --dmabuf
  (GPU-rendered DMA-BUF source, dmabuf_source.c), --cpus (taskset), nice'd --load.
- fluxer-voice tests/sender_consistency.rs: per-stage row-by-row check (raw copy,
  converted, decoded by a viewer after the H.264 encoder), opt-in.
- fluxer-video check: RowCounter/read_rows, row_hashes/SpliceCheck, StreamCheck.
- FLUXER_FRAME_CHECK: per-pull trace, splice check on captured/encoder-input/received
  frames, libwebrtc's PipeWire-format and encoder lines, encoder implementation log;
  FLUXER_CAPTURE_VERIFY_LATEST=0 as an A/B switch.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405); the commits are in the branch.

Merged into `textchat` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-16 08:38:02 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.