Screen share: copy Wayland frames before libwebrtc overwrites them (#589) #607
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/589-stream-top-strip"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Refs #589. This deliberately says Refs, not Closes: the "part of another frame" strip is found and fixed on the Linux/Wayland sender, but the black strip was not reproduced (see "Not covered"). Close #589 once
FLUXER_FRAME_CHECK=1confirms it on the real machines.Root cause (measured)
On Wayland, libwebrtc's PipeWire capturer hands
fluxer-capturea share of one of the two frames in its stream's queue, and its PipeWire thread writes a new picture into that same frame two buffers later whether or not the frame is still being read. It only logs a debug warning (shared_screencast_stream.ccat the pinned revisionwebrtc-a545de4:CaptureFramereturnslatest_available_frame_->Share()with no copy, l. 698;ProcessBufferrunsqueue_.MoveToNextFrame()and then logs "Overwriting frame that is still shared" but keeps writing, l. 915–917; the copy goes into that frame at l. 1040 (MemFd) and l. 1076 (DMA-BUF)).kQueueLengthis 2 (screen_capture_frame_queue.h).Our callback (
crates/fluxer-capture/src/desktop/rtc.rs,RtcGrabber::start) convertedframe.data()straight to I420 with libyuv. A 4K frame takes long enough to convert that sometimes the PipeWire thread starts rewriting it partway through. The writer starts at the top and is faster than the conversion, so it overtakes the read position. The result is a frame made of two pictures two buffers apart, with the boundary somewhere down the frame. That frame then gets encoded, so every viewer sees it, Electron included (Chromium uses the same capturer). It only lasts one frame, which is why a screenshot almost never catches it.Mixed frames seen in the test, bands top to bottom, digit = palette colour of the frame (7 colours, frame n = colour n mod 7):
6666611111,333?555555,4444444446. The two colours are always two frames apart, which is the signature of the 2-frame queue.Fix
rtc.rsnow copies the BGRA pixels into a buffer that is reused from frame to frame, and compares the copy with the frame (desktop::stable_copy). If anything changed during the copy it pulls the frame again (pull_stable, at most 3 retries, then the tick is skipped), and it converts only the copy. On X11 and Windows (WGC/DXGI fill the frame insidecapture_frame()on our thread,wgc_capture_session.ccGetFrame→ProcessFrame), the check never fires. It only costs the copy there.Cost: one memcpy plus one memcmp of the BGRA frame per captured frame. At 4K→4K@60 on this loaded machine (load average about 50 on 32 threads) the session delivered 749 frames in 20 s, against 973 before.
Measurements
Capture side, headless:
tools/capture-consistency/run.shThis uses a private D-Bus, a private PipeWire daemon, a stand-in ScreenCast portal and a
pw_streamsource (examples/solid_source.rs) whose frame n is one solid colour. The real Wayland path runs end to end (libwebrtc portal capturer, PipeWire stream, our conversion), and every frame is checked bytests/portal_consistency.rs. Nothing touches the desktop. Each run is 20 s with the source at 60 fps (libwebrtc asks for at most 60).Total: 75 mixed frames in 6,829 before, and 0 in 6,304 after.
The first version of the source used GStreamer's
pipewiresink. An independentpw_streamconsumer showed thatpipewiresinkhands over buffers whose pixels keep changing for a few milliseconds: 33 of 467 buffers were mixed if read at once, and 0 of 294 if read 30 ms later. That source was dropped, and the table uses thepw_streamsource, for which the same consumer saw 0 of 579 mixed at 4K.Receive side:
livekit_interop::received_screen_share_frames_are_whole_picturesThe test publishes a
SolidSequencethrough LiveKit and checks every frame the viewer's session delivers (what goes onto theFrameBus). It ran withFLUXER_MEASURE=1, 10 s per case, for AV1, H.264, VP9 and VP8, at 1080p30, 1080p60 and 1080p30 switching to 720p mid-stream. The four configurations were NVDEC on andLK_DISABLE_NVDEC=1, each with and without 6 busy cores. Result: 56 runs, about 18,900 frames, 0 inconsistent, 0 black top. The receive path is clean.Ownership map (audit)
DesktopFrame(the 2-slot queue). It is written on the PipeWire thread and read in our callback: the race. Our I420 frame is a newVecper frame, which becomesArc<[u8]>.apps/fluxer/src/capture_rt.rshand_on): immutableArcplanes are shared.fluxer-voiceto_rtc_buffercopies them into a newI420Bufferper frame.rtc.rsread_video):rtc_to_i420produces an I420 buffer that we hold by reference. Pooled decoder buffers, including NVDEC'sbuffer_pool_, are reused only once nobody references them.from_rtc_i420copies the buffer into newArcplanes, which go onto theFrameBus(two slots, never mutated, replaced byArcswap).fluxer-uiwidgets/video.rs):prepareconverts to I420 and writes all three planes withqueue.write_texture, which copies the data at call time, before the render pass of the same submission. Textures are recreated only on a size change and are written in that sameprepare, so no uninitialised texture is ever drawn.Debug switch
FLUXER_FRAME_CHECK=1logs atwarneach frame whose top fifth is black while the rest is not, and totals every 600 frames after the first one (fluxer_video::check::FrameCheck). It covers both the captured frames (side = "captured", before encoding) and the decoded frames (side = "received", per track sid), so a real share shows which side produces it.Not covered here
glReadPixelswith no explicit sync: it negotiates noSPA_META_SyncTimeline(egl_dmabuf.ccImageFromDmaBuf). On a driver without implicit sync this could read a buffer the compositor has not finished, which would be a separate, unfixed cause. Running the debug switch on the real machines should tell.Gate
FLUXER_SCREENSHOT_TESTS=1 FLUXER_LIVEKIT_TESTS=1 cargo xtask ci: the first run failed only on the known flakewatching_a_real_share_puts_its_frames_on_the_bus(#573), and the rerun passed every step ("all steps passed"). The newreceived_screen_share_frames_are_whole_picturespassed in both runs.tools/capture-consistency/run.sh 3840 2160 60 --target 3840x2160@60 --expect-cleanpasses on this branch (897 frames, 0 mixed); without the fix the same run fails (20 of 973 mixed).🤖 Generated with Claude Code