Screen share 2: Spike — LiveKit video publish/receive and codec facts (#149) #187

Manually merged
charles merged 0 commits from screenshare-2-spike-livekit-video into screenshare 2026-11-12 05:17:13 +00:00
Owner

What

The Screen share 2 spike. It adds no production code (that is Screen share 16/17).

  • crates/fluxer-voice/tests/livekit_video_spike.rs drives livekit directly. It contains three tests, each #[ignore]d and doing nothing unless FLUXER_LIVEKIT_TESTS=1. cargo xtask ci never runs this file: its LiveKit step runs only livekit_interop.
    • encoder_backends: VideoEncoderBackend::list_available(), video_encoder_codecs(backend) for every backend, and the RTP sender capabilities.
    • decode_probe: the RTP receiver capabilities for video.
    • publish_receive_matrix: session A publishes a 1280×720/30 and a 1920×1080/30 Screenshare track (simulcast: false, stream: "screen_share") with every codec of the software backend and of every hardware backend present, plus Auto. Session B subscribes and counts decoded frames for 5 s. The output is a table of frames pushed, encoded, received and decoded, the encoder and decoder implementation names, encode and decode time, process CPU and bitrate. The frames are built inline (I420 moving bars over a noise texture), so the spike does not depend on Screen share 1.
  • docs/superpowers/research/2026-11-11-screenshare-spikes.md §1 has the findings.

Main findings (spikes §1)

  • Linux prebuilt webrtc-a545de4 defines: WEBRTC_USE_H264, RTC_USE_LIBAOM_AV1_ENCODER, RTC_DAV1D_IN_INTERNAL_DECODER_FACTORY, WEBRTC_USE_PIPEWIRE (plus RTC_ENABLE_H265, X11). Windows: the same minus PipeWire/X11, plus WGC. I read the Windows defines from the release zip's ninja files, not from the runner's log.
  • H264 decodes on Linux. The prebuilt has ffmpeg_branding = "Chrome" and the SDK's own decoder probe passes, so the receiver capabilities include H264 and frames arrive (decoder FFmpeg). This corrects spec §8.1/§15. For Screen share 12/17 the decodable set should be the receiver capabilities. No bundled-sample probe is needed.
  • Encoders: Software encodes VP8, H264 (OpenH264), AV1 and VP9. Nothing encodes H265 in software. VAAPI H264 is available only when libva headers are present at build time. It also depends on the runtime: the SDK only tries renderD128/renderD129, and LIBVA_DRIVER_NAME=nvidia hides it. The CI and release images have no libva-dev today. Windows builds no hardware encoder (from source).
  • The PeerConnectionFactory is reachable through livekit::rtc_engine::lk_runtime::LkRuntime::instance().pc_factory().
  • Implementation names: libvpx, libaom, OpenH264 and VAAPI H264 Encoder. powerEfficientEncoder is true only for VAAPI.
  • Software CPU on a Ryzen 9950X: roughly 0.55 core for VP9 at 720p30 and 1.4 cores for AV1 at 1080p30. OpenH264 drops more than half the frames of screen content. Auto picks VAAPI for H264.

How it was tested

  • I ran all three spike tests on this machine (Ryzen 9 9950X, RTX 5090, and a Radeon 610M iGPU with radeonsi VAAPI encode), against livekit-server v1.12.0 --dev on loopback. Software VP8 at 720p (the test's assertion) delivered frames, and so did every other codec and backend.
  • I ran encoder_backends three times: with LIBVA_DRIVER_NAME unset, with it set to nvidia, and in a separate build with libva hidden from pkg-config. The last build logged libva not found; building without hardware accelerated video codecs.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passes locally.
  • The tests were not run on Windows: the Windows runner never sets FLUXER_LIVEKIT_TESTS. The doc says which Windows facts come from source.

Deviations from the plan

  • The Windows defines come from the published archive and not from "the Windows runner's build log". The log does not print them, and this task does not wait for CI.
  • Windows list_available() was inferred from source. It was not run.
  • The spike did not hit a fallback, so there is no follow-up issue.

Closes #149

🤖 Generated with Claude Code

## What The Screen share 2 spike. It adds no production code (that is Screen share 16/17). - `crates/fluxer-voice/tests/livekit_video_spike.rs` drives `livekit` directly. It contains three tests, each `#[ignore]`d and doing nothing unless `FLUXER_LIVEKIT_TESTS=1`. `cargo xtask ci` never runs this file: its LiveKit step runs only `livekit_interop`. - `encoder_backends`: `VideoEncoderBackend::list_available()`, `video_encoder_codecs(backend)` for every backend, and the RTP sender capabilities. - `decode_probe`: the RTP receiver capabilities for video. - `publish_receive_matrix`: session A publishes a 1280×720/30 and a 1920×1080/30 `Screenshare` track (`simulcast: false`, `stream: "screen_share"`) with every codec of the software backend and of every hardware backend present, plus Auto. Session B subscribes and counts decoded frames for 5 s. The output is a table of frames pushed, encoded, received and decoded, the encoder and decoder implementation names, encode and decode time, process CPU and bitrate. The frames are built inline (I420 moving bars over a noise texture), so the spike does not depend on Screen share 1. - `docs/superpowers/research/2026-11-11-screenshare-spikes.md` §1 has the findings. ## Main findings (spikes §1) - **Linux prebuilt `webrtc-a545de4` defines:** `WEBRTC_USE_H264`, `RTC_USE_LIBAOM_AV1_ENCODER`, `RTC_DAV1D_IN_INTERNAL_DECODER_FACTORY`, `WEBRTC_USE_PIPEWIRE` (plus `RTC_ENABLE_H265`, X11). **Windows:** the same minus PipeWire/X11, plus WGC. I read the Windows defines from the release zip's ninja files, not from the runner's log. - **H264 decodes on Linux.** The prebuilt has `ffmpeg_branding = "Chrome"` and the SDK's own decoder probe passes, so the receiver capabilities include H264 and frames arrive (decoder `FFmpeg`). This corrects spec §8.1/§15. For Screen share 12/17 the decodable set should be the receiver capabilities. No bundled-sample probe is needed. - **Encoders:** Software encodes VP8, H264 (OpenH264), AV1 and VP9. Nothing encodes H265 in software. VAAPI H264 is available only when libva headers are present at build time. It also depends on the runtime: the SDK only tries `renderD128`/`renderD129`, and `LIBVA_DRIVER_NAME=nvidia` hides it. The CI and release images have no `libva-dev` today. Windows builds no hardware encoder (from source). - **The PeerConnectionFactory is reachable** through `livekit::rtc_engine::lk_runtime::LkRuntime::instance().pc_factory()`. - **Implementation names:** `libvpx`, `libaom`, `OpenH264` and `VAAPI H264 Encoder`. `powerEfficientEncoder` is true only for VAAPI. - **Software CPU on a Ryzen 9950X:** roughly 0.55 core for VP9 at 720p30 and 1.4 cores for AV1 at 1080p30. OpenH264 drops more than half the frames of screen content. Auto picks VAAPI for H264. ## How it was tested - I ran all three spike tests on this machine (Ryzen 9 9950X, RTX 5090, and a Radeon 610M iGPU with radeonsi VAAPI encode), against `livekit-server` v1.12.0 `--dev` on loopback. Software VP8 at 720p (the test's assertion) delivered frames, and so did every other codec and backend. - I ran `encoder_backends` three times: with `LIBVA_DRIVER_NAME` unset, with it set to `nvidia`, and in a separate build with libva hidden from pkg-config. The last build logged `libva not found; building without hardware accelerated video codecs`. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passes locally. - The tests were not run on Windows: the Windows runner never sets `FLUXER_LIVEKIT_TESTS`. The doc says which Windows facts come from source. ## Deviations from the plan - The Windows defines come from the published archive and not from "the Windows runner's build log". The log does not print them, and this task does not wait for CI. - Windows `list_available()` was inferred from source. It was not run. - The spike did not hit a fallback, so there is no follow-up issue. Closes #149 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test(voice): LiveKit video spike and codec facts
All checks were successful
ci / Linux (x86_64) (push) Successful in 15m48s
ci / Windows (x64) (push) Successful in 28m32s
6d989f427b
Refs #149

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 2 (spike, LiveKit video), commit 6d989f4

Scope: git diff 53e3d60..origin/screenshare-2-spike-livekit-video. Two files changed: crates/fluxer-voice/tests/livekit_video_spike.rs and spikes §1.

Plan steps. The test has encoder_backends, decode_probe and publish_receive_matrix. Each is #[ignore] and returns early unless FLUXER_LIVEKIT_TESTS=1. Session A publishes a NativeVideoSource (is_screencast) with TrackSource::Screenshare and simulcast: false at 1280×720/30, plus 1080p/30, once per encodable codec. Session B auto-subscribes and counts decoded frames and their size over a 5 s window. The I420 moving bars are built inline, so nothing depends on fluxer-video. xtask never runs this file: the screenshot step's --ignored run removes FLUXER_LIVEKIT_TESTS, and the interop step runs --test livekit_interop only.

Interfaces (facts). §1 answers every question the plan lists.

  • Defines: I checked webrtc.ninja and args.gn in a local Linux prebuilt. WEBRTC_USE_H264, RTC_USE_LIBAOM_AV1_ENCODER, RTC_DAV1D_IN_INTERNAL_DECODER_FACTORY, WEBRTC_USE_PIPEWIRE and ffmpeg_branding = "Chrome" are all present.
  • Backends and codecs: list_available() and video_encoder_codecs are recorded per backend.
  • Factory access: LkRuntime::instance().pc_factory() is reachable. I checked that pub mod lk_runtime, pub fn instance and pub fn pc_factory are public in livekit 0.10.1.
  • Decode matrix for VP8, VP9, AV1 and H264, CPU per second, the libva build with and without headers (warning line quoted), and the encoder implementation names.

Invariants. No crate code or Cargo.toml changed. Every dependency the test uses (livekit-api, futures-util, tokio) was already a dev-dependency of fluxer-voice. The only key is the public livekit-server --dev pair, devkey/secret, which livekit_interop.rs and tools/livekit-test-server.sh already use. There are no real tokens. The VideoEncoderBackend variants are not cfg-gated, so the file compiles on Windows. /proc/self/stat returns None there.

Review Focus 3 (codec advertisement). §1.3 corrects spec §8.1/§15: H264 does decode on the Linux prebuilt. I confirmed IsInternalH264DecoderAvailable() in webrtc-sys 0.3.53 src/video_decoder_factory.cpp. It does a real Configure() probe, so H264 is advertised only when FFmpeg can decode it. That supports the guidance for Screen share 12/17: decodable = receiver capabilities, and the encodable set never comes from sender capabilities, because they list H265.

Findings

  1. minor: The Windows defines come from the published webrtc-win-x64-release.zip, not from the Windows runner's build log as the plan asks. The archive is the same artefact, so this deviation is acceptable, and it is documented. Windows list_available() and H264 decode are inferred from source and were not run. §1.6 says so and hands the check to the Screen share 17 startup probe. That probe must report it.
  2. minor: The CPU figures are single runs with roughly ±0.2 core of noise. The process-CPU column includes decode and debug-build frame generation. The doc states both. Screen share 11 should treat the AV1 and VP9 cost numbers as rough guidance, not thresholds.
  3. minor: The commit attribution is Claude Opus 5.5, while CLAUDE.md says Fable 5.2. This matches the session-level attribution the workflow uses and the recent commits on voice. No action needed unless the integrator wants CLAUDE.md updated.
  4. minor (follow-up for Screen share 18/30, already named in §1.2/§1.6): The CI and release images do not install libva-dev, so builds today have no VAAPI.

Local gate: FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci in a fresh review worktree → ==> xtask ci: all steps passed.

Verdict: approve. No blocking findings, and local CI is green.

## Independent review: Screen share 2 (spike, LiveKit video), commit 6d989f4 Scope: `git diff 53e3d60..origin/screenshare-2-spike-livekit-video`. Two files changed: `crates/fluxer-voice/tests/livekit_video_spike.rs` and *spikes §1*. **Plan steps.** The test has `encoder_backends`, `decode_probe` and `publish_receive_matrix`. Each is `#[ignore]` and returns early unless `FLUXER_LIVEKIT_TESTS=1`. Session A publishes a `NativeVideoSource` (is_screencast) with `TrackSource::Screenshare` and `simulcast: false` at 1280×720/30, plus 1080p/30, once per encodable codec. Session B auto-subscribes and counts decoded frames and their size over a 5 s window. The I420 moving bars are built inline, so nothing depends on `fluxer-video`. `xtask` never runs this file: the screenshot step's `--ignored` run removes `FLUXER_LIVEKIT_TESTS`, and the interop step runs `--test livekit_interop` only. **Interfaces (facts).** §1 answers every question the plan lists. - Defines: I checked `webrtc.ninja` and `args.gn` in a local Linux prebuilt. `WEBRTC_USE_H264`, `RTC_USE_LIBAOM_AV1_ENCODER`, `RTC_DAV1D_IN_INTERNAL_DECODER_FACTORY`, `WEBRTC_USE_PIPEWIRE` and `ffmpeg_branding = "Chrome"` are all present. - Backends and codecs: `list_available()` and `video_encoder_codecs` are recorded per backend. - Factory access: `LkRuntime::instance().pc_factory()` is reachable. I checked that `pub mod lk_runtime`, `pub fn instance` and `pub fn pc_factory` are public in livekit 0.10.1. - Decode matrix for VP8, VP9, AV1 and H264, CPU per second, the libva build with and without headers (warning line quoted), and the encoder implementation names. **Invariants.** No crate code or `Cargo.toml` changed. Every dependency the test uses (`livekit-api`, `futures-util`, `tokio`) was already a dev-dependency of `fluxer-voice`. The only key is the public `livekit-server --dev` pair, `devkey`/`secret`, which `livekit_interop.rs` and `tools/livekit-test-server.sh` already use. There are no real tokens. The `VideoEncoderBackend` variants are not cfg-gated, so the file compiles on Windows. `/proc/self/stat` returns `None` there. **Review Focus 3** (codec advertisement). §1.3 corrects spec §8.1/§15: H264 does decode on the Linux prebuilt. I confirmed `IsInternalH264DecoderAvailable()` in `webrtc-sys` 0.3.53 `src/video_decoder_factory.cpp`. It does a real `Configure()` probe, so H264 is advertised only when FFmpeg can decode it. That supports the guidance for Screen share 12/17: decodable = receiver capabilities, and the encodable set never comes from sender capabilities, because they list H265. **Findings** 1. *minor*: The Windows defines come from the published `webrtc-win-x64-release.zip`, not from the Windows runner's build log as the plan asks. The archive is the same artefact, so this deviation is acceptable, and it is documented. Windows `list_available()` and H264 decode are inferred from source and were not run. §1.6 says so and hands the check to the Screen share 17 startup probe. That probe must report it. 2. *minor*: The CPU figures are single runs with roughly ±0.2 core of noise. The process-CPU column includes decode and debug-build frame generation. The doc states both. Screen share 11 should treat the AV1 and VP9 cost numbers as rough guidance, not thresholds. 3. *minor*: The commit attribution is `Claude Opus 5.5`, while `CLAUDE.md` says `Fable 5.2`. This matches the session-level attribution the workflow uses and the recent commits on `voice`. No action needed unless the integrator wants `CLAUDE.md` updated. 4. *minor (follow-up for Screen share 18/30, already named in §1.2/§1.6)*: The CI and release images do not install `libva-dev`, so builds today have no VAAPI. **Local gate**: `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` in a fresh review worktree → `==> xtask ci: all steps passed`. **Verdict: approve.** No blocking findings, and local CI is green.
charles manually merged commit 69edce3d34 into screenshare 2026-11-12 05:17:13 +00:00
Sign in to join this conversation.
No description provided.