Screen share 17: fluxer-voice LiveKit video, screen audio, codec probe, interop test (#164) #226

Manually merged
charles merged 5 commits from screenshare-17-voice-livekit-video into screenshare 2026-11-12 07:59:51 +00:00
Owner

What this builds

LiveKitRoom (crates/fluxer-voice/src/rtc.rs) implements Screen share 16's Room video methods per spec §7.3 and §8.1:

  • publish_screen: screencast NativeVideoSource, TrackSource::Screenshare, stream "screen_share", simulcast: false, video_codec, video_encoding { max_bitrate, max_framerate }, degradation_preference, scalability_mode (AV1/VP9 only), video_encoder from EncoderBackend (Auto → Auto, PreferHardware → Hardware, PreferSoftware → Software). With audio: a 48 kHz 2-channel ScreenshareAudio track, 128 kbit/s, RED on, DTX off, WebRTC processing off. If the audio publish fails, the video is taken down again.
  • Room joins with dynacast: true (adaptive_stream stays false).
  • push_video: any VideoFrame → I420 → WebRTC buffer, paced to the encoding's fps (FramePace, frames under 3/4 of an interval after the last one are dropped). push_screen_audio takes stereo 10 ms frames.
  • unpublish_screen / set_screen_audio_muted find the tracks by source, so a full reconnect that republished them under new sids is covered. The SDK's LocalTrackRepublished is reported as LocalTrackUnpublished + LocalTrackPublished, and a server-side removal as LocalTrackUnpublished. The room's own unpublishes are not reported.
  • Remote video: NativeVideoStream → I420 (whatever buffer the decoder produced) → RoomEvent::Video, try_send (dropped when the session is full).
  • codec_capabilities(): probed once per process, as spikes §1.6 says. Encodable = the software backend's codecs, hardware = the hardware backend's list. Decodable = the video codecs in get_rtp_receiver_capabilities(Video), which list H264 only once the SDK's FFmpeg decoder works, so H264 is never claimed without a decoder. warmed_up() runs the probe on the warm-up thread.
  • EncoderStats every 2 s from the outbound-rtp stats: codec (from the codec entry), encoderImplementation, powerEfficientEncoder as hardware, fps, bitrate over the interval.
  • #125: ParticipantJoined is reported once the SDK says the participant is active, or after a 5 s fallback (PARTICIPANT_ACTIVE_FALLBACK). Its tracks, mutes and data are held behind the arrival, so ParticipantJoined still comes before TrackPublished. A participant that leaves before it is reported is never reported.

How it was tested

  • Unit tests in rtc.rs: publish_options_from_params, probe_never_claims_h264_without_decoder, video_frames_cross_into_webrtc_and_back, encoder_stats_from_two_samples, frames_are_paced_to_the_encoding_fps, screen_share_without_a_room_refuses_politely, and the arrival tests (participant_joined_waits_for_active_and_keeps_order, fallback, leave-before-active).
  • Interop (FLUXER_LIVEKIT_TESTS=1, local livekit-server 1.12.0 --dev via tools/livekit-test-server.sh, this machine: Ryzen 9950X, RTX 5090 + Radeon 610M VAAPI, LIBVA_DRIVER_NAME unset). All 15 tests in livekit_interop.rs pass, including the new screen_share_pattern_arrives_for_each_encodable_codec (AV1, H264, VP9, VP8), screen_audio_tone_arrives_stereo, republish_with_other_codec_reaches_viewer, unsubscribe_stops_video_bytes and advert_reaches_newcomer_after_active. Before the implementation commit, four of the five new interop tests failed against the base rtc.rs.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passes locally.

Deviations from the plan

  • Stream audio is mono in practice (#223, question). livekit 0.10.1 has no stereo publish option and never sets stereo/TfStereo in AddTrackRequest, so Opus downmixes. The track is published as the spec says, and two channels arrive carrying the same mix. screen_audio_tone_arrives_stereo asserts what is true today and names #223. This builds #223's recommended option 1 (accept for now). It is not treated as a blocker.
  • update_screen_encoding applies a new frame rate at once. The SDK does not expose the RTP sender, so a new bitrate ceiling, degradation preference, scalability mode or backend is kept and only takes effect at the next publish.
  • #125 test is weak. advert_reaches_newcomer_after_active also passed against the base implementation (timing on loopback is too friendly to show the race). The ordering logic itself is covered by the unit tests.
  • TDD split is partial. The unit tests sit in rtc.rs beside the code, so they arrived in the same commit as the implementation. Only the interop tests have a separate failing-first commit. The work resumed from an earlier interrupted attempt's uncommitted tree (wip/screenshare-17), which I reviewed and re-verified.
  • No assets/codec-probe sample bitstreams: spikes §1.3 showed that the receiver capabilities are enough.

Closes #164

🤖 Generated with Claude Code

## What this builds `LiveKitRoom` (`crates/fluxer-voice/src/rtc.rs`) implements Screen share 16's `Room` video methods per spec §7.3 and §8.1: - `publish_screen`: screencast `NativeVideoSource`, `TrackSource::Screenshare`, stream `"screen_share"`, `simulcast: false`, `video_codec`, `video_encoding { max_bitrate, max_framerate }`, `degradation_preference`, `scalability_mode` (AV1/VP9 only), `video_encoder` from `EncoderBackend` (Auto → Auto, PreferHardware → Hardware, PreferSoftware → Software). With audio: a 48 kHz 2-channel `ScreenshareAudio` track, 128 kbit/s, RED on, DTX off, WebRTC processing off. If the audio publish fails, the video is taken down again. - Room joins with `dynacast: true` (`adaptive_stream` stays false). - `push_video`: any `VideoFrame` → I420 → WebRTC buffer, paced to the encoding's fps (`FramePace`, frames under 3/4 of an interval after the last one are dropped). `push_screen_audio` takes stereo 10 ms frames. - `unpublish_screen` / `set_screen_audio_muted` find the tracks by source, so a full reconnect that republished them under new sids is covered. The SDK's `LocalTrackRepublished` is reported as `LocalTrackUnpublished` + `LocalTrackPublished`, and a server-side removal as `LocalTrackUnpublished`. The room's own unpublishes are not reported. - Remote video: `NativeVideoStream` → I420 (whatever buffer the decoder produced) → `RoomEvent::Video`, `try_send` (dropped when the session is full). - `codec_capabilities()`: probed once per process, as spikes §1.6 says. Encodable = the software backend's codecs, hardware = the hardware backend's list. Decodable = the video codecs in `get_rtp_receiver_capabilities(Video)`, which list H264 only once the SDK's FFmpeg decoder works, so H264 is never claimed without a decoder. `warmed_up()` runs the probe on the warm-up thread. - `EncoderStats` every 2 s from the outbound-rtp stats: codec (from the codec entry), `encoderImplementation`, `powerEfficientEncoder` as `hardware`, fps, bitrate over the interval. - **#125**: `ParticipantJoined` is reported once the SDK says the participant is *active*, or after a 5 s fallback (`PARTICIPANT_ACTIVE_FALLBACK`). Its tracks, mutes and data are held behind the arrival, so `ParticipantJoined` still comes before `TrackPublished`. A participant that leaves before it is reported is never reported. ## How it was tested - Unit tests in `rtc.rs`: `publish_options_from_params`, `probe_never_claims_h264_without_decoder`, `video_frames_cross_into_webrtc_and_back`, `encoder_stats_from_two_samples`, `frames_are_paced_to_the_encoding_fps`, `screen_share_without_a_room_refuses_politely`, and the arrival tests (`participant_joined_waits_for_active_and_keeps_order`, fallback, leave-before-active). - Interop (`FLUXER_LIVEKIT_TESTS=1`, local `livekit-server` 1.12.0 `--dev` via `tools/livekit-test-server.sh`, this machine: Ryzen 9950X, RTX 5090 + Radeon 610M VAAPI, `LIBVA_DRIVER_NAME` unset). All 15 tests in `livekit_interop.rs` pass, including the new `screen_share_pattern_arrives_for_each_encodable_codec` (AV1, H264, VP9, VP8), `screen_audio_tone_arrives_stereo`, `republish_with_other_codec_reaches_viewer`, `unsubscribe_stops_video_bytes` and `advert_reaches_newcomer_after_active`. Before the implementation commit, four of the five new interop tests failed against the base `rtc.rs`. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passes locally. ## Deviations from the plan - **Stream audio is mono in practice (#223, `question`).** `livekit` 0.10.1 has no stereo publish option and never sets `stereo`/`TfStereo` in `AddTrackRequest`, so Opus downmixes. The track is published as the spec says, and two channels arrive carrying the same mix. `screen_audio_tone_arrives_stereo` asserts what is true today and names #223. This builds #223's recommended option 1 (accept for now). It is not treated as a blocker. - **`update_screen_encoding`** applies a new frame rate at once. The SDK does not expose the RTP sender, so a new bitrate ceiling, degradation preference, scalability mode or backend is kept and only takes effect at the next publish. - **#125 test is weak.** `advert_reaches_newcomer_after_active` also passed against the base implementation (timing on loopback is too friendly to show the race). The ordering logic itself is covered by the unit tests. - **TDD split is partial.** The unit tests sit in `rtc.rs` beside the code, so they arrived in the same commit as the implementation. Only the interop tests have a separate failing-first commit. The work resumed from an earlier interrupted attempt's uncommitted tree (`wip/screenshare-17`), which I reviewed and re-verified. - No `assets/codec-probe` sample bitstreams: spikes §1.3 showed that the receiver capabilities are enough. Closes #164 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Owner

Independent review: Screen share 17 (61a073c..9e7acfb)

Scope: crates/fluxer-voice/src/rtc.rs, tests/livekit_interop.rs and README.md only. I read the diff cold and checked it against plan Task 17, spec §3.3, §7.3 and §8.1, spikes §1.3 and §1.6, and the pinned livekit 0.10.1 source.

Plan tests: all present and asserting what the plan asks.

  • Interop: screen_share_pattern_arrives_for_each_encodable_codec (rising pattern counters on the published sid, then an encoder report for that codec), screen_audio_tone_arrives_stereo, republish_with_other_codec_reaches_viewer, unsubscribe_stops_video_bytes, advert_reaches_newcomer_after_active.
  • Unit: publish_options_from_params checks screen_share, simulcast: false, the encoding, degradation, scalability mode (AV1 and VP9 only), the backend mapping, and audio at 128 000 with RED on and DTX off. probe_never_claims_h264_without_decoder. There are extra tests for pacing, Arrivals and stats.

Interfaces: every method of Task 16's Room is implemented with the trait's signatures. dynacast: true, adaptive_stream stays false. Remote video goes NativeVideoStream → I420 VideoFrame. codec_capabilities = software ∪ hardware encoders, with hardware set from the Hardware backend's list. Decodable = the receiver RTP capability names (spikes §1.6), so H.264 is claimed only when the SDK's FFmpeg probe passed (Review Focus 3 holds). EncoderStats every 2 s from outbound-rtp. Arrival is now reported at "active" (#125), with held events released in order after ParticipantJoined. I checked the SDK's ParticipantConnected / ParticipantActive dispatch (room/mod.rs:1300-1343): both the race and the already-active case are handled, and a participant reported twice gets nothing duplicated.

Invariants: the new public items are four std-typed consts; no LiveKit or libwebrtc type appears in a public signature. Malformed or zero-size frames are refused (to_i420 validates, copy_plane bounds-checks). Stat casts saturate. There is no panic path from server data, and no secrets.

CI: FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green in a fresh review worktree (fmt, clippy, deny, core-isolation, parity, test, screenshot-tests; livekit-interop skipped without FLUXER_LIVEKIT_TESTS). I did not rerun the interop suite against a server.

Findings

  1. minor (process): stream audio is mono, against spec §7.3 "stereo". The SDK cannot mark a published track stereo; I confirmed this in local_participant.rs. #223 is open with label question, and CLAUDE.md says a question blocks the task until Charles answers. The PR should not get ready-to-merge until #223 is answered. The interop test is still called ..._arrives_stereo but accepts the mixed result. That matches the documented deviation, but the name overstates what it proves.
  2. minor (follow-up): update_screen_encoding silently defers bitrate, degradation, scalability and backend changes to the next publish, and returns Ok. I confirmed transceiver() is pub(crate) in 0.10.1, so set_parameters really is out of reach. A quality or bitrate change mid-share (Screen share 11/25) therefore has no effect until a republish, and the session cannot tell. Please open a follow-up issue: either have the session republish on such a change, or wait for an SDK sender API.
  3. minor (test strength): advert_reaches_newcomer_after_active passes on the base rtc.rs too, as the implementer notes, because loopback hides the race. The #125 ordering is really covered only by the Arrivals unit tests, which is acceptable.
  4. minor: Arrivals::hold silently drops events beyond 256 per pending participant. A dropped TrackPublished would hide a track for good. This is unlikely within the 5 s fallback window, but a debug! when it happens would help.

Verdict: approve, no blocking findings, local CI green. Merge waits on Charles's answer to #223 (finding 1); finding 2 should become a follow-up issue before merge.

## Independent review: Screen share 17 (`61a073c..9e7acfb`) Scope: `crates/fluxer-voice/src/rtc.rs`, `tests/livekit_interop.rs` and `README.md` only. I read the diff cold and checked it against plan Task 17, spec §3.3, §7.3 and §8.1, spikes §1.3 and §1.6, and the pinned `livekit` 0.10.1 source. **Plan tests:** all present and asserting what the plan asks. - Interop: `screen_share_pattern_arrives_for_each_encodable_codec` (rising pattern counters on the published sid, then an encoder report for that codec), `screen_audio_tone_arrives_stereo`, `republish_with_other_codec_reaches_viewer`, `unsubscribe_stops_video_bytes`, `advert_reaches_newcomer_after_active`. - Unit: `publish_options_from_params` checks `screen_share`, `simulcast: false`, the encoding, degradation, scalability mode (AV1 and VP9 only), the backend mapping, and audio at 128 000 with RED on and DTX off. `probe_never_claims_h264_without_decoder`. There are extra tests for pacing, `Arrivals` and stats. **Interfaces:** every method of Task 16's `Room` is implemented with the trait's signatures. `dynacast: true`, `adaptive_stream` stays false. Remote video goes `NativeVideoStream` → I420 `VideoFrame`. `codec_capabilities` = software ∪ hardware encoders, with `hardware` set from the Hardware backend's list. Decodable = the receiver RTP capability names (spikes §1.6), so H.264 is claimed only when the SDK's FFmpeg probe passed (Review Focus 3 holds). `EncoderStats` every 2 s from outbound-rtp. Arrival is now reported at "active" (#125), with held events released in order after `ParticipantJoined`. I checked the SDK's `ParticipantConnected` / `ParticipantActive` dispatch (`room/mod.rs:1300-1343`): both the race and the already-active case are handled, and a participant reported twice gets nothing duplicated. **Invariants:** the new public items are four std-typed consts; no LiveKit or libwebrtc type appears in a public signature. Malformed or zero-size frames are refused (`to_i420` validates, `copy_plane` bounds-checks). Stat casts saturate. There is no panic path from server data, and no secrets. **CI:** `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green in a fresh review worktree (fmt, clippy, deny, core-isolation, parity, test, screenshot-tests; livekit-interop skipped without `FLUXER_LIVEKIT_TESTS`). I did not rerun the interop suite against a server. ### Findings 1. **minor (process): stream audio is mono, against spec §7.3 "stereo".** The SDK cannot mark a published track stereo; I confirmed this in `local_participant.rs`. #223 is open with label `question`, and `CLAUDE.md` says a `question` blocks the task until Charles answers. The PR should not get `ready-to-merge` until #223 is answered. The interop test is still called `..._arrives_stereo` but accepts the mixed result. That matches the documented deviation, but the name overstates what it proves. 2. **minor (follow-up): `update_screen_encoding` silently defers bitrate, degradation, scalability and backend changes to the next publish, and returns `Ok`.** I confirmed `transceiver()` is `pub(crate)` in 0.10.1, so `set_parameters` really is out of reach. A quality or bitrate change mid-share (Screen share 11/25) therefore has no effect until a republish, and the session cannot tell. Please open a `follow-up` issue: either have the session republish on such a change, or wait for an SDK sender API. 3. **minor (test strength): `advert_reaches_newcomer_after_active` passes on the base `rtc.rs` too,** as the implementer notes, because loopback hides the race. The #125 ordering is really covered only by the `Arrivals` unit tests, which is acceptable. 4. **minor: `Arrivals::hold` silently drops events beyond 256 per pending participant.** A dropped `TrackPublished` would hide a track for good. This is unlikely within the 5 s fallback window, but a `debug!` when it happens would help. **Verdict: approve, no blocking findings, local CI green.** Merge waits on Charles's answer to #223 (finding 1); finding 2 should become a `follow-up` issue before merge.
charles manually merged commit 5096dd1ee4 into screenshare 2026-11-12 07:59:51 +00:00
Sign in to join this conversation.
No description provided.