Screen share 17: fluxer-voice LiveKit video, screen audio, codec probe, interop test (#164) #226
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "screenshare-17-voice-livekit-video"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this builds
LiveKitRoom(crates/fluxer-voice/src/rtc.rs) implements Screen share 16'sRoomvideo methods per spec §7.3 and §8.1:publish_screen: screencastNativeVideoSource,TrackSource::Screenshare, stream"screen_share",simulcast: false,video_codec,video_encoding { max_bitrate, max_framerate },degradation_preference,scalability_mode(AV1/VP9 only),video_encoderfromEncoderBackend(Auto → Auto, PreferHardware → Hardware, PreferSoftware → Software). With audio: a 48 kHz 2-channelScreenshareAudiotrack, 128 kbit/s, RED on, DTX off, WebRTC processing off. If the audio publish fails, the video is taken down again.dynacast: true(adaptive_streamstays false).push_video: anyVideoFrame→ I420 → WebRTC buffer, paced to the encoding's fps (FramePace, frames under 3/4 of an interval after the last one are dropped).push_screen_audiotakes stereo 10 ms frames.unpublish_screen/set_screen_audio_mutedfind the tracks by source, so a full reconnect that republished them under new sids is covered. The SDK'sLocalTrackRepublishedis reported asLocalTrackUnpublished+LocalTrackPublished, and a server-side removal asLocalTrackUnpublished. The room's own unpublishes are not reported.NativeVideoStream→ I420 (whatever buffer the decoder produced) →RoomEvent::Video,try_send(dropped when the session is full).codec_capabilities(): probed once per process, as spikes §1.6 says. Encodable = the software backend's codecs, hardware = the hardware backend's list. Decodable = the video codecs inget_rtp_receiver_capabilities(Video), which list H264 only once the SDK's FFmpeg decoder works, so H264 is never claimed without a decoder.warmed_up()runs the probe on the warm-up thread.EncoderStatsevery 2 s from the outbound-rtp stats: codec (from the codec entry),encoderImplementation,powerEfficientEncoderashardware, fps, bitrate over the interval.ParticipantJoinedis reported once the SDK says the participant is active, or after a 5 s fallback (PARTICIPANT_ACTIVE_FALLBACK). Its tracks, mutes and data are held behind the arrival, soParticipantJoinedstill comes beforeTrackPublished. A participant that leaves before it is reported is never reported.How it was tested
rtc.rs:publish_options_from_params,probe_never_claims_h264_without_decoder,video_frames_cross_into_webrtc_and_back,encoder_stats_from_two_samples,frames_are_paced_to_the_encoding_fps,screen_share_without_a_room_refuses_politely, and the arrival tests (participant_joined_waits_for_active_and_keeps_order, fallback, leave-before-active).FLUXER_LIVEKIT_TESTS=1, locallivekit-server1.12.0--devviatools/livekit-test-server.sh, this machine: Ryzen 9950X, RTX 5090 + Radeon 610M VAAPI,LIBVA_DRIVER_NAMEunset). All 15 tests inlivekit_interop.rspass, including the newscreen_share_pattern_arrives_for_each_encodable_codec(AV1, H264, VP9, VP8),screen_audio_tone_arrives_stereo,republish_with_other_codec_reaches_viewer,unsubscribe_stops_video_bytesandadvert_reaches_newcomer_after_active. Before the implementation commit, four of the five new interop tests failed against the basertc.rs.FLUXER_SCREENSHOT_TESTS=1 cargo xtask cipasses locally.Deviations from the plan
question).livekit0.10.1 has no stereo publish option and never setsstereo/TfStereoinAddTrackRequest, so Opus downmixes. The track is published as the spec says, and two channels arrive carrying the same mix.screen_audio_tone_arrives_stereoasserts what is true today and names #223. This builds #223's recommended option 1 (accept for now). It is not treated as a blocker.update_screen_encodingapplies a new frame rate at once. The SDK does not expose the RTP sender, so a new bitrate ceiling, degradation preference, scalability mode or backend is kept and only takes effect at the next publish.advert_reaches_newcomer_after_activealso passed against the base implementation (timing on loopback is too friendly to show the race). The ordering logic itself is covered by the unit tests.rtc.rsbeside the code, so they arrived in the same commit as the implementation. Only the interop tests have a separate failing-first commit. The work resumed from an earlier interrupted attempt's uncommitted tree (wip/screenshare-17), which I reviewed and re-verified.assets/codec-probesample bitstreams: spikes §1.3 showed that the receiver capabilities are enough.Closes #164
🤖 Generated with Claude Code
Independent review: Screen share 17 (
61a073c..9e7acfb)Scope:
crates/fluxer-voice/src/rtc.rs,tests/livekit_interop.rsandREADME.mdonly. I read the diff cold and checked it against plan Task 17, spec §3.3, §7.3 and §8.1, spikes §1.3 and §1.6, and the pinnedlivekit0.10.1 source.Plan tests: all present and asserting what the plan asks.
screen_share_pattern_arrives_for_each_encodable_codec(rising pattern counters on the published sid, then an encoder report for that codec),screen_audio_tone_arrives_stereo,republish_with_other_codec_reaches_viewer,unsubscribe_stops_video_bytes,advert_reaches_newcomer_after_active.publish_options_from_paramschecksscreen_share,simulcast: false, the encoding, degradation, scalability mode (AV1 and VP9 only), the backend mapping, and audio at 128 000 with RED on and DTX off.probe_never_claims_h264_without_decoder. There are extra tests for pacing,Arrivalsand stats.Interfaces: every method of Task 16's
Roomis implemented with the trait's signatures.dynacast: true,adaptive_streamstays false. Remote video goesNativeVideoStream→ I420VideoFrame.codec_capabilities= software ∪ hardware encoders, withhardwareset from the Hardware backend's list. Decodable = the receiver RTP capability names (spikes §1.6), so H.264 is claimed only when the SDK's FFmpeg probe passed (Review Focus 3 holds).EncoderStatsevery 2 s from outbound-rtp. Arrival is now reported at "active" (#125), with held events released in order afterParticipantJoined. I checked the SDK'sParticipantConnected/ParticipantActivedispatch (room/mod.rs:1300-1343): both the race and the already-active case are handled, and a participant reported twice gets nothing duplicated.Invariants: the new public items are four std-typed consts; no LiveKit or libwebrtc type appears in a public signature. Malformed or zero-size frames are refused (
to_i420validates,copy_planebounds-checks). Stat casts saturate. There is no panic path from server data, and no secrets.CI:
FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciis green in a fresh review worktree (fmt, clippy, deny, core-isolation, parity, test, screenshot-tests; livekit-interop skipped withoutFLUXER_LIVEKIT_TESTS). I did not rerun the interop suite against a server.Findings
local_participant.rs. #223 is open with labelquestion, andCLAUDE.mdsays aquestionblocks the task until Charles answers. The PR should not getready-to-mergeuntil #223 is answered. The interop test is still called..._arrives_stereobut accepts the mixed result. That matches the documented deviation, but the name overstates what it proves.update_screen_encodingsilently defers bitrate, degradation, scalability and backend changes to the next publish, and returnsOk. I confirmedtransceiver()ispub(crate)in 0.10.1, soset_parametersreally is out of reach. A quality or bitrate change mid-share (Screen share 11/25) therefore has no effect until a republish, and the session cannot tell. Please open afollow-upissue: either have the session republish on such a change, or wait for an SDK sender API.advert_reaches_newcomer_after_activepasses on the basertc.rstoo, as the implementer notes, because loopback hides the race. The #125 ordering is really covered only by theArrivalsunit tests, which is acceptable.Arrivals::holdsilently drops events beyond 256 per pending participant. A droppedTrackPublishedwould hide a track for good. This is unlikely within the 5 s fallback window, but adebug!when it happens would help.Verdict: approve, no blocking findings, local CI green. Merge waits on Charles's answer to #223 (finding 1); finding 2 should become a
follow-upissue before merge.