Text chat 54: stereo screen-share audio through a patched livekit 0.10.1 (#354) #358
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "textchat-54-livekit-stereo-audio"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was built (Text chat 54, #354; #223)
Stream audio (
screen_share_audio) is published in stereo, as the Electron app does withforceStereo.livekit0.10.1 cannot ask for it, so the workspace builds the SDK fromthird_party/livekit: the published 0.10.1 package, vendored unchanged in its own commit (ddda212), then patched (35369a9), and used through[patch.crates-io].livekit::options::TrackPublishOptions { stereo: bool, .. }, defaultfalse, so every existing caller is unchanged.stereo: trueon an audio track, theAddTrackRequestcarriesstereoandTfStereo. When the publisher applies the server's answer, the Opusa=fmtpline of that track'sm=audiosection (found through its transceiver's mid) getsstereo=1;sprop-stereo=1where they are missing. No other section or line changes. Mono audio and video are untouched.fluxer-voicepublishesscreen_share_audiowithstereo: true. The microphone stays mono.fluxer-voice,docs/parity/screenshare.md,third_party/README.mdandthird_party/livekit/PATCHES.mdare updated.The full list of changes against upstream, with file and line, is in
third_party/livekit/PATCHES.md. This patch is a candidate for upstream (livekit/rust-sdks).Build: the vendored crate is not a workspace member
third_party/livekitis listed underexcludein the workspaceCargo.tomland is used only through[patch.crates-io]. Cargo locks every optional feature of a workspace member, and livekit's unused TLS features would add openssl, native-tls and webpki-roots (CDLA-Permissive-2.0) to the lock. Against the base,Cargo.lockchanges in one way only: thelivekitentry loses its registrysourceandchecksum. No crate is added.deny.tomlis not changed.The crate's unit tests run as their own
cargo xtask cistep,vendored-tests, on both runners. That step runscargo test --lib --manifest-path third_party/livekit/Cargo.tomlwith the workspace'sCargo.lockcopied in for the run, so the tests build with the versions the client ships. The published lock is put back afterwards.How it was tested
third_party/livekit:add_track_request_marks_stereoandanswer_munging_sets_stereo_only_for_stereo_mid.screen_share_audio_arrives_in_stereoincrates/fluxer-voice/tests/livekit_interop.rs, run againstlivekit-server1.12.0. It sends 440 Hz on the left and 660 Hz on the right, and each channel must carry at least ten times the other's energy over 2 s. It fails on the unpatched SDK, where both channels carry the same mix, and passes with the patch. It replacesscreen_audio_tone_arrives_stereo, which accepted the mix-down.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciis green locally.For the reviewer
Closes #354
🤖 Generated with Claude Code
Review round 1, changes in
0167c95:third_party/livekitis now under workspaceexcludeand is used only through[patch.crates-io]. Against the base79af1e1, the only change toCargo.lockis that thelivekitentry loses its registrysourceandchecksum. foreign-types, native-tls, openssl*, hyper-tls, tokio-native-tls and webpki-roots are no longer in the lock.deny.tomlis unchanged, andcargo deny checkpasses (advisories, bans, licenses and sources all ok). The crate's unit tests run in a newcargo xtask cistep,vendored-tests. It runscargo test --lib --manifest-path third_party/livekit/Cargo.tomlwith the workspace lock copied in for the run, and the published lock is restored afterwards.add_track_request_marks_stereoandanswer_munging_sets_stereo_only_for_stereo_midpass there.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciis green locally.Cargo.lockchange (source and checksum only, no crate added), explains why the crate is excluded, and describes how its tests run. PATCHES.md ("Build" section), the vendored Cargo.toml comments and xtask/README.md (new step 8) are updated to match.remove_trackis synchronous while the set is behind an async mutex, and pruning while an answer is applied could race withcreate_sender.textchat.Independent review of Text chat 54 (diff 79af1e1..0167c95).
Checked
ddda212is byte-identical to the publishedlivekit-0.10.1.crate(sha256 9ae9e2c7…, matches the old lock checksum). The patch commits hold only the change.add_track_request_marks_stereocovers stereo vs mono, video, and TfStereo next to the preconnect buffer.answer_munging_sets_stereo_only_for_stereo_miduses two Opus sections and checks: only that mid changes, CRLF and LF are kept, the trailing EOL is kept, an existingstereo=is left alone, the munge is idempotent, and a missing fmtp is added.screen_share_audio_arrives_in_stereouses 440 Hz left and 660 Hz right, a 2 s span, a single-bin (Goertzel-style) energy ratio of at least 10, and checks left != right.TrackPublishOptions { stereo: bool }exists with defaultfalse.AddTrackRequest.stereoandTfStereoare set for stereo audio only. The answer munge is matched by the transceiver's mid, and other sections are untouched.rtc.rssetsstereo: trueonly forscreen_share_audio, and a unit test asserts the microphone stays mono.Cargo.lockchange is the livekitsource/checksumlines only, anddeny.tomlis unchanged. Excluding the crate from the workspace plus[patch.crates-io]keeps the TLS crates out.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: green (deny: advisories, bans, licenses, sources ok).vendored-testsran both patch tests and they passed. The worktree was clean afterwards, so the published lock was restored.Findings
screen_share_audio_arrives_in_stereowas not run withFLUXER_LIVEKIT_TESTS=1after0167c95, by the implementer or by me. The build change keeps the same patched path crate, so this is low risk, but the integrator should run the Linux interop job before merge.run_vendored_testsoverwrites the trackedthird_party/livekit/Cargo.lockand restores it in-process. If the run is killed (Ctrl-C, runner timeout), the tracked file is left modified. A temp copy of the package, orgit checkoutof the file in CI, would avoid this. No correctness impact.stereo_track_idsonly grows (#360). Dual-PC stereo receive is not covered (#359). Both are documented in PATCHES.md.Verdict: approve. No blocking findings, and local CI is green.