Screen share: MemFd capture buffers on NVIDIA, no black band in Linux shares (#589) #945

Closed
charles wants to merge 0 commits from fix-589-black-band-linux-sender into textchat
Owner

Closes #589

The black band across Linux shares

Cause (sender, Linux Wayland, NVIDIA driver). libwebrtc's PipeWire capturer offers the compositor DMA-BUF modifiers whenever EGL works and reads each DMA-BUF (EGL import, glReadPixels) as soon as PipeWire hands it over. It negotiates no SPA_META_SyncTimeline. KWin (ScreenCastStream::record, current master) queues a DMA-BUF right after drawing into it; the render's sync file goes only to a SyncTimeline consumer, so for libwebrtc the ordering is left to implicit sync, which the NVIDIA driver does not give. The capture then reads a frame the compositor is still drawing: black where it has cleared and not yet painted, picture elsewhere. That frame is encoded, so every viewer sees the band; it is black (BGRA zeros), not green or stale picture, which also rules out the encoder and the I420 path. Windows senders never take this path. The earlier fixes (#607, #763) were about a different race (libwebrtc rewriting the frame we read) and their harness source called glFinish before queueing, so it could not show this one.

Fix.

  • webrtc-sys (vendored, PATCHES.md "MemFd instead of DMA-BUF for the PipeWire screen capturer"): pipewire_capture::set_pipewire_dmabuf(bool). When it is off, new_desktop_capturer gives libwebrtc a SharedScreenCastStream made with an EglDmaBuf that has no device, so it lists no modifier and the compositor negotiates MemFd. With MemFd the compositor reads its picture back itself once it has finished drawing, and libwebrtc copies finished pixels. egl_dmabuf.h needs EGL, GL and GBM headers, which the build hosts don't have, so build.rs copies them from libwebrtc's own sysroot into OUT_DIR.
  • fluxer-capture: MemFd while the NVIDIA kernel driver is loaded (/sys/module/nvidia_drm), DMA-BUF elsewhere (Mesa gives implicit sync). FLUXER_CAPTURE_DMABUF=1/0 forces either one for an A/B test. The choice is logged at info when the portal capture opens. fluxer-capture now depends on the vendored webrtc-sys, which was already in the build through libwebrtc.
  • FLUXER_FRAME_CHECK=1 also reports a band of black rows inside a frame (fluxer_video::check::BandCheck: first row and height, logged when it appears; banded in the totals) at captured, encoder-input and received. The old check only looked for a black top fifth, so it could not see this band.

How it was verified. tools/capture-consistency/run.sh --dmabuf, headless (private D-Bus, private PipeWire, stand-in portal, the real libwebrtc portal path). Nothing touched the desktop session. dmabuf_source.c gains DMABUF_SOURCE_SYNC=none, which shares buffers the way KWin shares them with a consumer that has no SyncTimeline: a clear to black, then the picture drawn by a slow shader (DMABUF_SOURCE_WORK), then queued at once. A consumer that offers no modifier now gets MemFd buffers that are read back after drawing. Without --chain, portal_consistency reads every row of every captured frame (FLUXER_PORTAL_ROWS=1) and counts frames with black rows as banded. RTX 5090, driver 620.70, 60 fps source, 10 s per run:

Source → target DMA-BUF (before, or FLUXER_CAPTURE_DMABUF=1) MemFd (this PR)
3440×1440 → 1920×802 127/601 banded (before); 41/601 (A/B, same build) 0/601
1460×1254 → source 2/601 (A/B) 0/601
3440×1440 → 2560×1070 — 0/601

The bands that were found were full-width runs of black rows inside the frame, with the frame's own picture above them, e.g. 0..550=30 550..802=black. In this source the unpainted part is at the bottom. In KWin it is whatever region was still being painted. With DMABUF_SOURCE_WORK=200 (a fast draw), DMA-BUF showed nothing, so the band needs a draw that is still running when the frame is read, which a busy compositor provides.

Unit tests: desktop::linux::tests::portal_buffers_are_memfd_on_the_nvidia_driver, check::tests::a_black_band_inside_a_frame_is_found, check::tests::stream_check_reports_a_black_band_when_it_appears.

Not verified: a real KWin session (I drove no compositor) or a real share from the reporter's RTX 4080. The black band is shown only with the stand-in source. To confirm on an affected sender, run FLUXER_FRAME_CHECK=1 FLUXER_LOG=info,fluxer_capture=debug ./Flaxative-*.AppImage. Its log should show screen capture: PipeWire buffers the portal stream accepts (#589) buffers="MemFd" nvidia_driver=true and no frame check: a band of black rows inside the frame lines with side="captured". Running the same share once with FLUXER_CAPTURE_DMABUF=1 should bring the band back.

Encoder: a Linux + NVIDIA sender in a release build (CI sets CUDA_HOME) uses NVENC H.264, because the display GPU's encoder comes first in the #857 order. The encoder is not the cause: the band is black in RGB, the measured encoder runs of #879 at these sizes were clean, and the band appears before encoding in the harness.

Gate: FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: fmt, clippy, deny, core-isolation and parity pass. The test step fails only on fluxer-ui --test channel_settings (channel_invites_rows_and_create) and fluxer --test serveradmin_acceptance (bans_and_invites_as_owner). Both fail the same way on unmodified origin/textchat (4afed6390); they are unrelated to this change: #943 (the invite fixture Hq7mPz2K expired on 2026-11-19). The remaining steps were run by hand: vendored tests (livekit 78, webrtc-sys 19) pass, and the screenshot step (cargo test --workspace -- --ignored, 298 tests including 256 screenshots) passes.

🤖 Generated with Claude Code

Closes #589 ## The black band across Linux shares **Cause (sender, Linux Wayland, NVIDIA driver).** libwebrtc's PipeWire capturer offers the compositor DMA-BUF modifiers whenever EGL works and reads each DMA-BUF (EGL import, `glReadPixels`) as soon as PipeWire hands it over. It negotiates no `SPA_META_SyncTimeline`. KWin (`ScreenCastStream::record`, current master) queues a DMA-BUF right after drawing into it; the render's sync file goes only to a SyncTimeline consumer, so for libwebrtc the ordering is left to implicit sync, which the NVIDIA driver does not give. The capture then reads a frame the compositor is still drawing: **black where it has cleared and not yet painted**, picture elsewhere. That frame is encoded, so every viewer sees the band; it is black (BGRA zeros), not green or stale picture, which also rules out the encoder and the I420 path. Windows senders never take this path. The earlier fixes (#607, #763) were about a different race (libwebrtc rewriting the frame we read) and their harness source called `glFinish` before queueing, so it could not show this one. **Fix.** - `webrtc-sys` (vendored, PATCHES.md "MemFd instead of DMA-BUF for the PipeWire screen capturer"): `pipewire_capture::set_pipewire_dmabuf(bool)`. When it is off, `new_desktop_capturer` gives libwebrtc a `SharedScreenCastStream` made with an `EglDmaBuf` that has no device, so it lists no modifier and the compositor negotiates MemFd. With MemFd the compositor reads its picture back itself once it has finished drawing, and libwebrtc copies finished pixels. `egl_dmabuf.h` needs EGL, GL and GBM headers, which the build hosts don't have, so `build.rs` copies them from libwebrtc's own sysroot into `OUT_DIR`. - `fluxer-capture`: MemFd while the NVIDIA kernel driver is loaded (`/sys/module/nvidia_drm`), DMA-BUF elsewhere (Mesa gives implicit sync). `FLUXER_CAPTURE_DMABUF=1`/`0` forces either one for an A/B test. The choice is logged at `info` when the portal capture opens. `fluxer-capture` now depends on the vendored `webrtc-sys`, which was already in the build through `libwebrtc`. - `FLUXER_FRAME_CHECK=1` also reports a **band of black rows inside a frame** (`fluxer_video::check::BandCheck`: first row and height, logged when it appears; `banded` in the totals) at `captured`, `encoder-input` and `received`. The old check only looked for a black top fifth, so it could not see this band. **How it was verified.** `tools/capture-consistency/run.sh --dmabuf`, headless (private D-Bus, private PipeWire, stand-in portal, the real libwebrtc portal path). Nothing touched the desktop session. `dmabuf_source.c` gains `DMABUF_SOURCE_SYNC=none`, which shares buffers the way KWin shares them with a consumer that has no SyncTimeline: a clear to black, then the picture drawn by a slow shader (`DMABUF_SOURCE_WORK`), then queued at once. A consumer that offers no modifier now gets MemFd buffers that are read back after drawing. Without `--chain`, `portal_consistency` reads every row of every captured frame (`FLUXER_PORTAL_ROWS=1`) and counts frames with black rows as `banded`. RTX 5090, driver 620.70, 60 fps source, 10 s per run: | Source → target | DMA-BUF (before, or `FLUXER_CAPTURE_DMABUF=1`) | MemFd (this PR) | |---|---|---| | 3440×1440 → 1920×802 | 127/601 banded (before); 41/601 (A/B, same build) | 0/601 | | 1460×1254 → source | 2/601 (A/B) | 0/601 | | 3440×1440 → 2560×1070 | — | 0/601 | The bands that were found were full-width runs of black rows inside the frame, with the frame's own picture above them, e.g. `0..550=30 550..802=black`. In this source the unpainted part is at the bottom. In KWin it is whatever region was still being painted. With `DMABUF_SOURCE_WORK=200` (a fast draw), DMA-BUF showed nothing, so the band needs a draw that is still running when the frame is read, which a busy compositor provides. Unit tests: `desktop::linux::tests::portal_buffers_are_memfd_on_the_nvidia_driver`, `check::tests::a_black_band_inside_a_frame_is_found`, `check::tests::stream_check_reports_a_black_band_when_it_appears`. **Not verified:** a real KWin session (I drove no compositor) or a real share from the reporter's RTX 4080. The black band is shown only with the stand-in source. To confirm on an affected sender, run `FLUXER_FRAME_CHECK=1 FLUXER_LOG=info,fluxer_capture=debug ./Flaxative-*.AppImage`. Its log should show `screen capture: PipeWire buffers the portal stream accepts (#589) buffers="MemFd" nvidia_driver=true` and no `frame check: a band of black rows inside the frame` lines with `side="captured"`. Running the same share once with `FLUXER_CAPTURE_DMABUF=1` should bring the band back. **Encoder:** a Linux + NVIDIA sender in a release build (CI sets `CUDA_HOME`) uses NVENC H.264, because the display GPU's encoder comes first in the #857 order. The encoder is not the cause: the band is black in RGB, the measured encoder runs of #879 at these sizes were clean, and the band appears before encoding in the harness. **Gate:** `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci`: fmt, clippy, deny, core-isolation and parity pass. The test step fails only on `fluxer-ui --test channel_settings` (`channel_invites_rows_and_create`) and `fluxer --test serveradmin_acceptance` (`bans_and_invites_as_owner`). Both fail the same way on unmodified `origin/textchat` (4afed6390); they are unrelated to this change: #943 (the invite fixture `Hq7mPz2K` expired on 2026-11-19). The remaining steps were run by hand: vendored tests (livekit 78, webrtc-sys 19) pass, and the screenshot step (`cargo test --workspace -- --ignored`, 298 tests including 256 screenshots) passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
libwebrtc's PipeWire capturer reads a DMA-BUF (EGL import, glReadPixels)
as soon as PipeWire hands it over and negotiates no SPA_META_SyncTimeline.
KWin then queues the buffer right after drawing into it, without waiting
for its GPU, and leaves the ordering to implicit sync, which the NVIDIA
driver does not give: the capture reads a frame the compositor is still
drawing, black where it had cleared and not yet painted. That frame is
encoded, so every viewer sees a black band across a Linux share; Windows
senders never take this path.

- webrtc-sys (vendored, PATCHES.md): pipewire_capture, a switch that gives
  the portal capturer a ScreenCast stream with no EGL device, so it offers
  no modifier and takes MemFd buffers only (the compositor then reads its
  picture back itself, after drawing). EGL/GL/GBM headers come from
  libwebrtc's own sysroot.
- fluxer-capture: MemFd while the NVIDIA kernel driver is loaded,
  DMA-BUF elsewhere; FLUXER_CAPTURE_DMABUF=1/0 forces either (A/B);
  the choice is logged when the capture opens.
- fluxer-video: FLUXER_FRAME_CHECK also reports a band of black rows
  inside a frame (BandCheck), at capture, encoder input and receive.
- tools/capture-consistency: dmabuf_source DMABUF_SOURCE_SYNC=none shares
  buffers as KWin does for a consumer without SyncTimeline, and serves
  MemFd consumers; run.sh --dmabuf without --chain checks every row of
  every captured frame (portal_consistency FLUXER_PORTAL_ROWS=1).

Measured (RTX 5090, 3440x1440@60 source, 10 s runs): DMA-BUF 127/601
and 41/601 frames with black rows (1460x1254: 2/601); with the fix
(MemFd) 0/601 at 1080p, 1440p and source sizes.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405), in 8a5a29193; the combined tree passed the full local gate.

Merged into textchat by hand (the merge API returns 405), in 8a5a29193; the combined tree passed the full local gate.
charles closed this pull request 2026-11-19 10:45:37 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.