Screen share 16: voice-video-session (#163) #205

Manually merged
charles merged 0 commits from screenshare-16-voice-video-session into screenshare 2026-11-12 05:57:04 +00:00
Owner

Stacked on: origin/screenshare-8-core-stream-ports, origin/screenshare-1-video-frames

What this builds

fluxer-voice gains screen share over the Room trait (plan Screen share 16; spec §6.4, §7.3, §8.1):

  • Room: publish_screen(ScreenPublish { codec, params, with_audio }) -> Result<ScreenSids { video, audio }, RoomError>, update_screen_encoding(EncodeParams), unpublish_screen(), set_screen_audio_muted(bool), push_video(VideoFrame), push_screen_audio(Frame10ms), codec_capabilities() -> CodecCapabilities { encode, decode }.
  • RoomEvent::{Video { identity, sid, frame }, LocalTrackPublished, LocalTrackUnpublished, EncoderStats}.
  • VoiceHandle::{push_video(VideoFrame), push_screen_audio(&[i16]), remote_video() -> mpsc::Receiver<RemoteVideo { identity, sid, frame }>}.
  • The session carries out PublishScreenShare, UpdateScreenShareEncoding, RepublishScreenShare (unpublish + publish with the same params and the new codec; audio mute carries over), UnpublishScreenShare, SetScreenShareAudioMuted, and reports LocalTrackPublished/LocalTrackUnpublished, CodecCapabilities, VideoFrameInfo (first frame after subscribing and on resize only), EncoderStats.
  • Captured video: a watch channel, so only the newest frame reaches the room and only while a video track is published. Stream audio: 10 ms stereo frames, only while an audio track is published and unmuted.
  • Remote video: only for subscribed tracks, latest wins: a channel of 1 plus the newest held frame per sid, so at most 2 frames of a track wait.
  • FakeRoom: publishes and records the share (screen(), video_frames(), screen_audio_frames()), remote streamers with fluxer_video's test pattern (play_video, push_remote_video), stereo screen audio (play_screen_audio, stereo_tone_frame), report_encoder_stats, set_codec_capabilities (default: software AV1/VP9/VP8 both ways, no H264), fail_next_publish_screen, remove_local_screen.

Testing

  • tests/session_video.rs, 13 tests: the 7 the plan lists (publish_reports_local_track_published, video_frames_dropped_when_not_publishing, republish_swaps_codec_and_reports_new_sid, remote_video_only_while_subscribed, video_frame_info_on_first_frame_and_resize_only, screen_audio_muted_stops_frames, disconnect_unpublishes_everything) plus codec_capabilities_reported_once_after_first_connect, failed_publish_reports_nothing_and_takes_no_frames, update_encoding_reaches_the_room_only_while_published, remote_video_latest_wins_two_per_sid, encoder_stats_are_reported, share_removed_by_the_server_is_reported_and_takes_no_frames.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally. The LiveKit interop test was not run (rtc.rs is unchanged).

Deviations and decisions for the reviewer

  • The new Room methods have default implementations: publishing fails, no codecs. That way LiveKitRoom (Screen share 17 implements them) and apps/fluxer's UnavailableRoom compile unchanged, and this PR stays inside fluxer-voice.
  • codec_capabilities() is async (Task 17's probe can take time). The session reports CodecCapabilities once per session, right after the first Connected. Reporting it at spawn would have broken the app's UnavailableRoom test. Because of this, tests/session.rs's Rig helpers skip or consume that event.
  • LocalTrackUnpublished is reported for each share track on UnpublishScreenShare, Disconnect, room loss (before Disconnected) and a move to another room. RoomEvent::LocalTrack* are only for changes the session did not ask for, such as the server removing the share.
  • A failed publish_screen is logged and reports nothing: the port has no failure event. The core's go-live timeout has to cover it.
  • A PublishScreenShare while a share is up replaces it (unpublish first) and does not reject it.

Closes #163

🤖 Generated with Claude Code

Stacked on: origin/screenshare-8-core-stream-ports, origin/screenshare-1-video-frames ## What this builds `fluxer-voice` gains screen share over the `Room` trait (plan Screen share 16; spec §6.4, §7.3, §8.1): - `Room`: `publish_screen(ScreenPublish { codec, params, with_audio }) -> Result<ScreenSids { video, audio }, RoomError>`, `update_screen_encoding(EncodeParams)`, `unpublish_screen()`, `set_screen_audio_muted(bool)`, `push_video(VideoFrame)`, `push_screen_audio(Frame10ms)`, `codec_capabilities() -> CodecCapabilities { encode, decode }`. - `RoomEvent::{Video { identity, sid, frame }, LocalTrackPublished, LocalTrackUnpublished, EncoderStats}`. - `VoiceHandle::{push_video(VideoFrame), push_screen_audio(&[i16]), remote_video() -> mpsc::Receiver<RemoteVideo { identity, sid, frame }>}`. - The session carries out `PublishScreenShare`, `UpdateScreenShareEncoding`, `RepublishScreenShare` (unpublish + publish with the same params and the new codec; audio mute carries over), `UnpublishScreenShare`, `SetScreenShareAudioMuted`, and reports `LocalTrackPublished`/`LocalTrackUnpublished`, `CodecCapabilities`, `VideoFrameInfo` (first frame after subscribing and on resize only), `EncoderStats`. - Captured video: a `watch` channel, so only the newest frame reaches the room and only while a video track is published. Stream audio: 10 ms stereo frames, only while an audio track is published and unmuted. - Remote video: only for subscribed tracks, latest wins: a channel of 1 plus the newest held frame per sid, so at most 2 frames of a track wait. - `FakeRoom`: publishes and records the share (`screen()`, `video_frames()`, `screen_audio_frames()`), remote streamers with `fluxer_video`'s test pattern (`play_video`, `push_remote_video`), stereo screen audio (`play_screen_audio`, `stereo_tone_frame`), `report_encoder_stats`, `set_codec_capabilities` (default: software AV1/VP9/VP8 both ways, no H264), `fail_next_publish_screen`, `remove_local_screen`. ## Testing - `tests/session_video.rs`, 13 tests: the 7 the plan lists (`publish_reports_local_track_published`, `video_frames_dropped_when_not_publishing`, `republish_swaps_codec_and_reports_new_sid`, `remote_video_only_while_subscribed`, `video_frame_info_on_first_frame_and_resize_only`, `screen_audio_muted_stops_frames`, `disconnect_unpublishes_everything`) plus `codec_capabilities_reported_once_after_first_connect`, `failed_publish_reports_nothing_and_takes_no_frames`, `update_encoding_reaches_the_room_only_while_published`, `remote_video_latest_wins_two_per_sid`, `encoder_stats_are_reported`, `share_removed_by_the_server_is_reported_and_takes_no_frames`. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. The LiveKit interop test was not run (`rtc.rs` is unchanged). ## Deviations and decisions for the reviewer - The new `Room` methods have **default implementations**: publishing fails, no codecs. That way `LiveKitRoom` (Screen share 17 implements them) and `apps/fluxer`'s `UnavailableRoom` compile unchanged, and this PR stays inside `fluxer-voice`. - `codec_capabilities()` is async (Task 17's probe can take time). The session reports `CodecCapabilities` **once per session, right after the first `Connected`**. Reporting it at spawn would have broken the app's `UnavailableRoom` test. Because of this, `tests/session.rs`'s `Rig` helpers skip or consume that event. - `LocalTrackUnpublished` is reported for each share track on `UnpublishScreenShare`, `Disconnect`, room loss (before `Disconnected`) and a move to another room. `RoomEvent::LocalTrack*` are only for changes the session did not ask for, such as the server removing the share. - A failed `publish_screen` is logged and reports nothing: the port has no failure event. The core's go-live timeout has to cover it. - A `PublishScreenShare` while a share is up replaces it (unpublish first) and does not reject it. Closes #163 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(proto,api): stream keys, instance limits, capability adverts and preview endpoints
All checks were successful
ci / Windows (x64) (push) Successful in 28m59s
ci / Linux (x86_64) (push) Successful in 37m2s
6ac431d9db
Refs #154

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(video): frames, conversions, frame bus, test pattern and preview JPEG
All checks were successful
ci / Linux (x86_64) (push) Successful in 15m16s
ci / Windows (x64) (push) Successful in 16m21s
8b8a359082
Refs #148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(core): stream state, ports and selectors
All checks were successful
ci / Linux (x86_64) (push) Successful in 13m48s
ci / Windows (x64) (push) Successful in 14m25s
0d406ebd96
Refs #155

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(voice): screen share over the room trait
All checks were successful
ci / Linux (x86_64) (push) Successful in 36m38s
ci / Windows (x64) (push) Successful in 30m16s
8ac5e7571b
Refs #163

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 16 (voice-video-session)

Reviewed 10060a8..8ac5e75 (fluxer-voice only: room.rs, session.rs, fake.rs, lib.rs, README, Cargo.toml, tests).

Plan tests: all 7 named tests exist in tests/session_video.rs and assert what the plan says (publish_reports_local_track_published, video_frames_dropped_when_not_publishing incl. latest-wins [2,5], republish_swaps_codec_and_reports_new_sid incl. break-before-make, same params, new sids, mute carried over, remote_video_only_while_subscribed, video_frame_info_on_first_frame_and_resize_only, screen_audio_muted_stops_frames, disconnect_unpublishes_everything incl. room loss and room move). The 6 extra tests are sound.

Interfaces: Room::{publish_screen(ScreenPublish{codec,params,with_audio}) -> Result<ScreenSids,RoomError>, update_screen_encoding, unpublish_screen, set_screen_audio_muted, push_video(VideoFrame), push_screen_audio(Frame10ms), codec_capabilities() -> CodecCapabilities} (async like the rest of the trait), RoomEvent::{Video{identity,sid,frame}, LocalTrackPublished, LocalTrackUnpublished, EncoderStats}, VoiceHandle::{push_video, push_screen_audio(&[i16]), remote_video() -> mpsc::Receiver<RemoteVideo{identity,sid,frame}>} are all present as written. The trait defaults leave LiveKitRoom and UnavailableRoom compiling, and Task 17 overrides them.

Invariants: the only new dependency is fluxer-voice -> fluxer-video (named by the plan). There are no LiveKit or libwebrtc types in public signatures, no new panics on room or server data, and no secrets in the code or fixtures. fluxer-core and fluxer-video are untouched.

Review Focus: none of the items belongs to Task 16 directly. Item 6 (latest wins, nothing queued) is respected on the voice side: a watch channel carries captured frames, and remote video is capped at 1 frame in the channel plus 1 held back per sid.

Findings

  • minor: a failed publish_screen only logs a warning and emits no event, so the go-live machine (Task 10) must time out on a missing LocalTrackPublished. Task 10 or 21 should confirm that timeout exists, or a later port change should add a failure event.
  • minor: SetScreenShareAudioMuted sent before a share exists is dropped, and PublishScreenShare always starts unmuted. The core must re-send the mute after LocalTrackPublished(ScreenShareAudio) if the user muted stream audio before going live.
  • minor: CodecCapabilities is emitted once per session, after the first Connected, not "once per process" at startup. This works with Task 12 (voice_only until it arrives). Task 21's --debug-info will have to call the room's codec_capabilities() directly.
  • minor: the FakeRoom remote streamer is driven by bursts (play_video, play_screen_audio) rather than a timed script. Task 20 has to pace the frames itself.
  • minor: the LiveKit interop test was not run. rtc.rs is unchanged, so this task has nothing in it to test.

CI

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci in a fresh review worktree, with its own target dir: all steps passed (livekit-interop skipped, as expected).

Verdict: APPROVE. No blocking findings and CI is green.

## Independent review: Screen share 16 (voice-video-session) Reviewed `10060a8..8ac5e75` (fluxer-voice only: room.rs, session.rs, fake.rs, lib.rs, README, Cargo.toml, tests). **Plan tests:** all 7 named tests exist in `tests/session_video.rs` and assert what the plan says (`publish_reports_local_track_published`, `video_frames_dropped_when_not_publishing` incl. latest-wins [2,5], `republish_swaps_codec_and_reports_new_sid` incl. break-before-make, same params, new sids, mute carried over, `remote_video_only_while_subscribed`, `video_frame_info_on_first_frame_and_resize_only`, `screen_audio_muted_stops_frames`, `disconnect_unpublishes_everything` incl. room loss and room move). The 6 extra tests are sound. **Interfaces:** `Room::{publish_screen(ScreenPublish{codec,params,with_audio}) -> Result<ScreenSids,RoomError>, update_screen_encoding, unpublish_screen, set_screen_audio_muted, push_video(VideoFrame), push_screen_audio(Frame10ms), codec_capabilities() -> CodecCapabilities}` (async like the rest of the trait), `RoomEvent::{Video{identity,sid,frame}, LocalTrackPublished, LocalTrackUnpublished, EncoderStats}`, `VoiceHandle::{push_video, push_screen_audio(&[i16]), remote_video() -> mpsc::Receiver<RemoteVideo{identity,sid,frame}>}` are all present as written. The trait defaults leave `LiveKitRoom` and `UnavailableRoom` compiling, and Task 17 overrides them. **Invariants:** the only new dependency is `fluxer-voice -> fluxer-video` (named by the plan). There are no LiveKit or libwebrtc types in public signatures, no new panics on room or server data, and no secrets in the code or fixtures. fluxer-core and fluxer-video are untouched. **Review Focus:** none of the items belongs to Task 16 directly. Item 6 (latest wins, nothing queued) is respected on the voice side: a watch channel carries captured frames, and remote video is capped at 1 frame in the channel plus 1 held back per sid. ### Findings - **minor**: a failed `publish_screen` only logs a warning and emits no event, so the go-live machine (Task 10) must time out on a missing `LocalTrackPublished`. Task 10 or 21 should confirm that timeout exists, or a later port change should add a failure event. - **minor**: `SetScreenShareAudioMuted` sent before a share exists is dropped, and `PublishScreenShare` always starts unmuted. The core must re-send the mute after `LocalTrackPublished(ScreenShareAudio)` if the user muted stream audio before going live. - **minor**: `CodecCapabilities` is emitted once per session, after the first `Connected`, not "once per process" at startup. This works with Task 12 (`voice_only` until it arrives). Task 21's `--debug-info` will have to call the room's `codec_capabilities()` directly. - **minor**: the FakeRoom remote streamer is driven by bursts (`play_video`, `play_screen_audio`) rather than a timed script. Task 20 has to pace the frames itself. - **minor**: the LiveKit interop test was not run. `rtc.rs` is unchanged, so this task has nothing in it to test. ### CI `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` in a fresh review worktree, with its own target dir: **all steps passed** (livekit-interop skipped, as expected). **Verdict: APPROVE.** No blocking findings and CI is green.
charles manually merged commit 61a073cefa into screenshare 2026-11-12 05:57:04 +00:00
Sign in to join this conversation.
No description provided.