Screen share 4: Linux screen capture backend (#151) #201

Manually merged
charles merged 0 commits from screenshare-4-capture-linux into screenshare 2026-11-12 05:17:21 +00:00
Owner

Stacked on: screenshare-3-capture-crate

What

fluxer_capture::DesktopScreenCapturer (new src/desktop.rs, Linux): screen and window capture through libwebrtc's DesktopCapturer (spec §7.1).

  • Session detection as libwebrtc does it (DisplaySession::detect): Wayland when XDG_SESSION_TYPE=wayland and WAYLAND_DISPLAY are set, else X11 when DISPLAY is set, else CaptureError::Unsupported. CI containers get the error, not a panic.
  • Wayland: in_app_picker == false, empty source list, SourceChoice::SystemPicker → Generic capturer → ScreenCast portal picker. start returns at once; Started comes with the first frame after the person picks. Picker closed → Ended(Cancelled), session stopped later → Ended(SessionEnded). The GLib loop comes from libwebrtc's glib-main-loop feature.
  • X11: screens (screen:<n>) and windows (window:<xid>) with 480×270 thumbnails. A closed window → Ended(SourceClosed).
  • Common loop: one thread per session owns the libwebrtc capturer, pulls capture_frame() at the target rate (Pacer), converts BGRA → I420 with libyuv (yuv_helper::argb_to_i420), scales to fit (I420Buffer::scale, even size, no upscaling), Stalled/Resumed (StallDetector, armed only after the first frame so an open picker is not a stall), handles switch/update/stop through a command channel. Started is sent again when the frame size changes (README contract updated).
  • default_backends() returns it on Linux, or Unsupported(reason) without a session.
  • Workspace dep libwebrtc = "0.4.0" (the version livekit pins), Linux-only in fluxer-capture, with feature glib-main-loop.

Tests

  • Unit tests (CI): pacing_holds_target_fps, bgra_frames_become_i420_at_target_size (libyuv matches fluxer_video::convert within ±1), plus odd-size/malformed buffers, fit, update rate, failure → Ended once, Cancelled vs SessionEnded, no stall while the picker is open, Started again on resize, session detection, id round trip.
  • CI: tests/linux_real.rs::new_without_session_is_an_error_not_a_panic (also run here under env -u DISPLAY -u WAYLAND_DISPLAY).
  • Real (FLUXER_CAPTURE_TESTS=1), on KDE Plasma 6 Wayland (kwin_wayland + XWayland), Ryzen 9 9950X, two 4K outputs:
    • x11_lists_screens_and_windows_with_thumbnails: passed, run through XWayland (env -u WAYLAND_DISPLAY XDG_SESSION_TYPE=x11). 2 screens + 3 windows with thumbnails in 55-75 ms; 91 frames in 3.0 s at 30 fps; 4K→1080p30 costs 15-18 % of one core; window frames have real content; the switch keeps the session.
    • wayland_portal_capture_delivers_non_black_frames: not completed. The portal is reached (D-Bus monitor: CreateSession → SelectSources → Start, KDE dialog shown), but nobody was there to click, so it timed out at the picker after 120 s. Same for portal_close_reports_ended, which also needs a person. Follow-up #200 is to run both by hand.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green locally.

Findings are in spikes §2 (docs/superpowers/research/2026-11-11-screenshare-spikes.md, new file with only §2 on this branch; it will conflict with the §1/§4 branches only by section).

Deviations / notes for the reviewer

  • No fallback needed (the PipeWire path reaches the portal and X11 captures frames). Wayland frame delivery is still unverified (#200).
  • tests/fake.rs::unsupported_backends_return_errors now tests the Unsupported stub directly: on a Linux desktop default_backends() now returns the real capturer, and start(SystemPicker) would open the portal.
  • deny.toml: license exception for target-lexicon (Apache-2.0 WITH LLVM-exception, build-time only, via system-deps for glib-sys/gio-sys).
  • Packaging (Screen share 30): glib-main-loop makes binaries that link fluxer-capture need libgio-2.0, libgobject-2.0 and libglib-2.0 (readelf -d).
  • desktop is cfg(target_os = "linux") for now. Screen share 5 widens it to Windows; the loop, conversion and grabber are OS-independent.
  • app_name/pid are None on X11 because libwebrtc does not provide them. window_audio is false.

Closes #151

🤖 Generated with Claude Code

Stacked on: screenshare-3-capture-crate ## What `fluxer_capture::DesktopScreenCapturer` (new `src/desktop.rs`, Linux): screen and window capture through libwebrtc's `DesktopCapturer` (spec §7.1). - **Session detection** as libwebrtc does it (`DisplaySession::detect`): Wayland when `XDG_SESSION_TYPE=wayland` and `WAYLAND_DISPLAY` are set, else X11 when `DISPLAY` is set, else `CaptureError::Unsupported`. CI containers get the error, not a panic. - **Wayland**: `in_app_picker == false`, empty source list, `SourceChoice::SystemPicker` → `Generic` capturer → ScreenCast portal picker. `start` returns at once; `Started` comes with the first frame after the person picks. Picker closed → `Ended(Cancelled)`, session stopped later → `Ended(SessionEnded)`. The GLib loop comes from libwebrtc's `glib-main-loop` feature. - **X11**: screens (`screen:<n>`) and windows (`window:<xid>`) with 480×270 thumbnails. A closed window → `Ended(SourceClosed)`. - **Common loop**: one thread per session owns the libwebrtc capturer, pulls `capture_frame()` at the target rate (`Pacer`), converts BGRA → I420 with libyuv (`yuv_helper::argb_to_i420`), scales to fit (`I420Buffer::scale`, even size, no upscaling), `Stalled`/`Resumed` (`StallDetector`, armed only after the first frame so an open picker is not a stall), handles `switch`/`update`/`stop` through a command channel. `Started` is sent again when the frame size changes (README contract updated). - `default_backends()` returns it on Linux, or `Unsupported(reason)` without a session. - Workspace dep `libwebrtc = "0.4.0"` (the version livekit pins), Linux-only in `fluxer-capture`, with feature `glib-main-loop`. ## Tests - Unit tests (CI): `pacing_holds_target_fps`, `bgra_frames_become_i420_at_target_size` (libyuv matches `fluxer_video::convert` within ±1), plus odd-size/malformed buffers, fit, update rate, failure → `Ended` once, Cancelled vs SessionEnded, no stall while the picker is open, `Started` again on resize, session detection, id round trip. - CI: `tests/linux_real.rs::new_without_session_is_an_error_not_a_panic` (also run here under `env -u DISPLAY -u WAYLAND_DISPLAY`). - Real (`FLUXER_CAPTURE_TESTS=1`), on KDE Plasma 6 Wayland (kwin_wayland + XWayland), Ryzen 9 9950X, two 4K outputs: - `x11_lists_screens_and_windows_with_thumbnails`: **passed**, run through XWayland (`env -u WAYLAND_DISPLAY XDG_SESSION_TYPE=x11`). 2 screens + 3 windows with thumbnails in 55-75 ms; 91 frames in 3.0 s at 30 fps; 4K→1080p30 costs 15-18 % of one core; window frames have real content; the switch keeps the session. - `wayland_portal_capture_delivers_non_black_frames`: **not completed**. The portal is reached (D-Bus monitor: CreateSession → SelectSources → Start, KDE dialog shown), but nobody was there to click, so it timed out at the picker after 120 s. Same for `portal_close_reports_ended`, which also needs a person. Follow-up #200 is to run both by hand. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green locally. Findings are in *spikes §2* (`docs/superpowers/research/2026-11-11-screenshare-spikes.md`, new file with only §2 on this branch; it will conflict with the §1/§4 branches only by section). ## Deviations / notes for the reviewer - No fallback needed (the PipeWire path reaches the portal and X11 captures frames). Wayland frame delivery is still unverified (#200). - `tests/fake.rs::unsupported_backends_return_errors` now tests the `Unsupported` stub directly: on a Linux desktop `default_backends()` now returns the real capturer, and `start(SystemPicker)` would open the portal. - `deny.toml`: license exception for `target-lexicon` (`Apache-2.0 WITH LLVM-exception`, build-time only, via `system-deps` for `glib-sys`/`gio-sys`). - Packaging (Screen share 30): `glib-main-loop` makes binaries that link `fluxer-capture` need `libgio-2.0`, `libgobject-2.0` and `libglib-2.0` (`readelf -d`). - `desktop` is `cfg(target_os = "linux")` for now. Screen share 5 widens it to Windows; the loop, conversion and grabber are OS-independent. - `app_name`/`pid` are `None` on X11 because libwebrtc does not provide them. `window_audio` is `false`. Closes #151 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(video): frames, conversions, frame bus, test pattern and preview JPEG
All checks were successful
ci / Linux (x86_64) (push) Successful in 15m16s
ci / Windows (x64) (push) Successful in 16m21s
8b8a359082
Refs #148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(capture): capture traits and fake backends
All checks were successful
ci / Windows (x64) (push) Successful in 16m9s
ci / Linux (x86_64) (push) Successful in 35m35s
d821f19097
Refs #150

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DesktopScreenCapturer on libwebrtc's DesktopCapturer: Wayland through the
ScreenCast portal (SystemPicker, glib-main-loop), X11 with screen and window
lists and 480x270 thumbnails; one pull thread per session at the target
rate, BGRA to I420 at the target size with libyuv, Started/Stalled/Resumed/
Ended. default_backends() returns it on Linux desktops.

Refs #151

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs(capture): Linux screen capture spike findings (spikes §2)
All checks were successful
ci / Linux (x86_64) (push) Successful in 14m39s
ci / Windows (x64) (push) Successful in 29m27s
5f276b83e1
Refs #151

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 4 (Linux screen capture), diff d821f19..5f276b8

Verdict: APPROVE. No blocking findings. Local FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green in a clean review worktree (all steps pass; 12 desktop::tests unit tests pass, and linux_real runs its CI test with the real-device tests ignored).

What I checked

  1. Plan tests exist and test what the plan says. Unit tests pacing_holds_target_fps (50 fps, 40 to 52 frames/s, no bursts) and bgra_frames_become_i420_at_target_size (padded 1080p BGRA becomes I420 at 1280x720, colours within ±1 of fluxer-video's BT.601) are there. So are the CI test new_without_session_is_an_error_not_a_panic and the real tests wayland_portal_capture_delivers_non_black_frames, x11_lists_screens_and_windows_with_thumbnails (thumbnails at most 480x270 via THUMBNAIL_*) and portal_close_reports_ended, which are #[ignore] and gated on FLUXER_CAPTURE_TESTS=1. Extra tests: stall suppressed while the picker is open, Ended reported once, cancel before the first frame vs. session ended after it, size change, session detection.
  2. Interfaces. DesktopScreenCapturer: ScreenCapturer. On Wayland: in_app_picker == false, an empty list, and SystemPicker maps to Generic. On X11: screen and window lists with thumbnails. One thread per session pulls at target.fps (Pacer), converts BGRA to I420 with libyuv and fits it to the target. Stalled comes after 3 s (StallDetector). The portal session ending is Ended. default_backends() returns the capturer on Linux and falls back to Unsupported without a session.
  3. Invariants. libwebrtc types appear only in src/desktop.rs. The public API (new, with_env, session, DisplaySession) has no libwebrtc or LiveKit types. fluxer-capture does not depend on fluxer-core or livekit. Cargo.lock has a single libwebrtc 0.4.0, shared with livekit. The FFI callback cannot panic: it handles a poisoned mutex, checks sizes with try_from, and rejects short buffers or a bad stride without panicking (tested). No secrets.
  4. Review Focus. No item of the plan's Review Focus belongs to Task 4.
  5. deny.toml exception for target-lexicon (Apache-2.0 WITH LLVM-exception, used only at build time through system-deps) is justified.

Findings (all minor, none blocking)

  • minor: the spike's central Wayland question is still open. Portal frames and the portal ending (tray stop or cancel) were not verified by hand. The Cancelled/SessionEnded mapping is inferred from BaseCapturerPipeWire (a permanent error) and tested only with a scripted grabber. Follow-up #200 tracks it. It should be run before Screen share 21 relies on it, because if it fails, the spec's ashpd + pipewire fallback is needed.
  • minor: picker cancel is reported asynchronously (start returns Ok, then Ended(Cancelled)). The README contract line "a closed system picker is Cancelled" (in the start-error bullet) and the Wayland paragraph now describe two different behaviours. Screen share 10/21 must treat Ended(Cancelled) before Started as "picker cancelled", not as "stream ended". Suggest rewording that contract bullet so it says both forms are allowed.
  • minor: a switch(SystemPicker) on Wayland reuses the old session's state. size is still Some, so cancelling the re-opened picker reports Ended(SessionEnded) instead of Cancelled. The armed StallDetector will report Stalled if the person takes more than 3 s in the new picker. Resetting size/stall on a successful Switch would fix both.
  • minor: wall-clock timing tests (pacing_holds_target_fps 40 to 52 frames/s, update_changes_the_rate_on_the_fly 25 to 40) could flake on a heavily loaded runner. Watch for it, no change needed now.
  • minor: spikes §2 cites spikes §1.1, which this branch does not have yet (Task 2 writes it). The file holds only §2, so expect a mechanical merge with Task 2's §1.
  • note (packaging, Screen share 30): glib-main-loop adds libgio-2.0, libgobject-2.0 and libglib-2.0 as NEEDED libraries for every binary that links fluxer-capture. The README and spikes §2.4 record this correctly. Cargo feature unification also turns the feature on for fluxer-voice's libwebrtc. That is harmless, since the GLib loop starts only from DesktopCapturer::start under Wayland.

🤖 Generated with Claude Code

## Independent review: Screen share 4 (Linux screen capture), diff d821f19..5f276b8 **Verdict: APPROVE.** No blocking findings. Local `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green in a clean review worktree (all steps pass; 12 `desktop::tests` unit tests pass, and `linux_real` runs its CI test with the real-device tests ignored). ### What I checked 1. **Plan tests exist and test what the plan says.** Unit tests `pacing_holds_target_fps` (50 fps, 40 to 52 frames/s, no bursts) and `bgra_frames_become_i420_at_target_size` (padded 1080p BGRA becomes I420 at 1280x720, colours within ±1 of fluxer-video's BT.601) are there. So are the CI test `new_without_session_is_an_error_not_a_panic` and the real tests `wayland_portal_capture_delivers_non_black_frames`, `x11_lists_screens_and_windows_with_thumbnails` (thumbnails at most 480x270 via `THUMBNAIL_*`) and `portal_close_reports_ended`, which are `#[ignore]` and gated on `FLUXER_CAPTURE_TESTS=1`. Extra tests: stall suppressed while the picker is open, `Ended` reported once, cancel before the first frame vs. session ended after it, size change, session detection. 2. **Interfaces.** `DesktopScreenCapturer: ScreenCapturer`. On Wayland: `in_app_picker == false`, an empty list, and `SystemPicker` maps to `Generic`. On X11: screen and window lists with thumbnails. One thread per session pulls at `target.fps` (`Pacer`), converts BGRA to I420 with libyuv and fits it to the target. `Stalled` comes after 3 s (`StallDetector`). The portal session ending is `Ended`. `default_backends()` returns the capturer on Linux and falls back to `Unsupported` without a session. 3. **Invariants.** libwebrtc types appear only in `src/desktop.rs`. The public API (`new`, `with_env`, `session`, `DisplaySession`) has no libwebrtc or LiveKit types. fluxer-capture does not depend on fluxer-core or livekit. `Cargo.lock` has a single `libwebrtc` 0.4.0, shared with livekit. The FFI callback cannot panic: it handles a poisoned mutex, checks sizes with `try_from`, and rejects short buffers or a bad stride without panicking (tested). No secrets. 4. **Review Focus.** No item of the plan's Review Focus belongs to Task 4. 5. `deny.toml` exception for `target-lexicon` (Apache-2.0 WITH LLVM-exception, used only at build time through system-deps) is justified. ### Findings (all minor, none blocking) - **minor: the spike's central Wayland question is still open.** Portal frames and the portal ending (tray stop or cancel) were not verified by hand. The `Cancelled`/`SessionEnded` mapping is inferred from `BaseCapturerPipeWire` (a permanent error) and tested only with a scripted grabber. Follow-up #200 tracks it. It should be run before Screen share 21 relies on it, because if it fails, the spec's `ashpd` + `pipewire` fallback is needed. - **minor: picker cancel is reported asynchronously** (`start` returns `Ok`, then `Ended(Cancelled)`). The README contract line "a closed system picker is `Cancelled`" (in the start-error bullet) and the Wayland paragraph now describe two different behaviours. Screen share 10/21 must treat `Ended(Cancelled)` before `Started` as "picker cancelled", not as "stream ended". Suggest rewording that contract bullet so it says both forms are allowed. - **minor: a `switch(SystemPicker)` on Wayland reuses the old session's state.** `size` is still `Some`, so cancelling the re-opened picker reports `Ended(SessionEnded)` instead of `Cancelled`. The armed `StallDetector` will report `Stalled` if the person takes more than 3 s in the new picker. Resetting `size`/`stall` on a successful `Switch` would fix both. - **minor: wall-clock timing tests** (`pacing_holds_target_fps` 40 to 52 frames/s, `update_changes_the_rate_on_the_fly` 25 to 40) could flake on a heavily loaded runner. Watch for it, no change needed now. - **minor: spikes §2 cites *spikes §1.1*,** which this branch does not have yet (Task 2 writes it). The file holds only §2, so expect a mechanical merge with Task 2's §1. - **note (packaging, Screen share 30):** `glib-main-loop` adds `libgio-2.0`, `libgobject-2.0` and `libglib-2.0` as `NEEDED` libraries for every binary that links fluxer-capture. The README and spikes §2.4 record this correctly. Cargo feature unification also turns the feature on for fluxer-voice's libwebrtc. That is harmless, since the GLib loop starts only from `DesktopCapturer::start` under Wayland. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
charles manually merged commit 69edce3d34 into screenshare 2026-11-12 05:17:21 +00:00
Sign in to join this conversation.
No description provided.