Text chat 54: stereo screen-share audio through a patched livekit 0.10.1 #354

Closed
opened 2026-11-12 20:46:12 +00:00 by charles · 0 comments
Owner

Plan: docs/superpowers/plans/2026-11-12-textchat.md, section "Task 54" (Stereo stream audio, added 2026-11-12 at Charles's request). Fixes #223.

Task 54: Stereo screen-share audio through a patched livekit 0.10.1

Issue: #ISSUE54

Why: livekit 0.10.1 never marks a published audio track as stereo: its AddTrackRequest (src/room/participant/local_participant.rs:414) leaves stereo false and never adds AudioTrackFeature::TfStereo, and TrackPublishOptions has no option for it. The server therefore answers without stereo=1 for the track and libwebrtc's Opus encoder downmixes the 48 kHz two-channel screen_share_audio source (crates/fluxer-voice/src/rtc.rs:715-725). The Electron app (LiveKit JS client) sends stereo: true + TF_STEREO in the request and munges the answer's Opus a=fmtp for that track's mid to stereo=1; the Rust SDK already has the munging code for single-PC offers (src/rtc_engine/peer_transport.rs:335 munge_stereo_for_audio).

Files:

  • Create: third_party/livekit/ — the published livekit 0.10.1 sources (copy of the crates.io package, its Cargo.toml normalised so it builds as a path dependency), plus third_party/livekit/PATCHES.md listing every change against upstream with the upstream file/line, and third_party/README.md (what is vendored, why, how to re-apply on an upgrade, licence: Apache-2.0, keep its LICENSE file).
  • Modify: workspace Cargo.toml ([patch.crates-io] livekit = { path = "third_party/livekit" }), deny.toml only if needed (a path crate is not checked against the registry), crates/fluxer-voice/src/rtc.rs (publish screen_share_audio with stereo: true; the microphone stays mono), crates/fluxer-voice/README.md (remove the "Stream audio is mono end to end today" paragraph; describe the patch), docs/parity/screenshare.md (stream audio row), xtask core-isolation/README checks if they walk crate directories (exclude third_party).
  • Test: third_party/livekit unit tests for the patch; crates/fluxer-voice/tests/livekit_interop.rs new test.

Interfaces:

  • livekit::options::TrackPublishOptions { stereo: bool, .. } (default false, so every existing caller is unchanged).

  • With stereo: true on an audio track: AddTrackRequest.stereo = true and audio_features contains TfStereo; when the publisher applies the server's answer, the Opus a=fmtp line of that track's m=audio section (matched by its mid / transceiver) gets stereo=1;sprop-stereo=1 if absent, and no other section changes. Mono tracks and video are untouched.

  • Step 1: Write failing tests: in third_party/livekit: add_track_request_marks_stereo (request built for a stereo audio track has stereo and TfStereo; mono has neither), answer_munging_sets_stereo_only_for_stereo_mid (an SDP with two Opus audio sections: only the stereo track's fmtp gains stereo=1;sprop-stereo=1; CRLF/LF preserved; existing stereo= left alone). In fluxer-voice (behind FLUXER_LIVEKIT_TESTS=1, like the other interop tests, run by cargo xtask ci on the Linux runner): screen_share_audio_arrives_in_stereo — publish a screen share whose audio carries a 440 Hz tone on the left channel and 660 Hz on the right, subscribe from a second session, and assert the received left and right channels differ and each is dominated by its own frequency (Goertzel or FFT bin energy ratio ≥ 10:1 after 2 s of audio).

  • Step 2: run them, see them fail (the interop test fails against unpatched 0.10.1 because both channels carry the same mix).

  • Step 3: vendor 0.10.1 verbatim first and commit that alone (chore: vendor livekit 0.10.1 unchanged), so the patch commit's diff is only the change; then implement the patch, set `st

🤖 Generated with Claude Code

Plan: `docs/superpowers/plans/2026-11-12-textchat.md`, section "Task 54" (Stereo stream audio, added 2026-11-12 at Charles's request). Fixes #223. ### Task 54: Stereo screen-share audio through a patched `livekit` 0.10.1 **Issue:** #ISSUE54 **Why:** `livekit` 0.10.1 never marks a published audio track as stereo: its `AddTrackRequest` (`src/room/participant/local_participant.rs:414`) leaves `stereo` false and never adds `AudioTrackFeature::TfStereo`, and `TrackPublishOptions` has no option for it. The server therefore answers without `stereo=1` for the track and libwebrtc's Opus encoder downmixes the 48 kHz two-channel `screen_share_audio` source (`crates/fluxer-voice/src/rtc.rs:715-725`). The Electron app (LiveKit JS client) sends `stereo: true` + `TF_STEREO` in the request and munges the answer's Opus `a=fmtp` for that track's mid to `stereo=1`; the Rust SDK already has the munging code for single-PC offers (`src/rtc_engine/peer_transport.rs:335` `munge_stereo_for_audio`). **Files:** - Create: `third_party/livekit/` — the published `livekit` 0.10.1 sources (copy of the crates.io package, its `Cargo.toml` normalised so it builds as a path dependency), plus `third_party/livekit/PATCHES.md` listing every change against upstream with the upstream file/line, and `third_party/README.md` (what is vendored, why, how to re-apply on an upgrade, licence: Apache-2.0, keep its LICENSE file). - Modify: workspace `Cargo.toml` (`[patch.crates-io] livekit = { path = "third_party/livekit" }`), `deny.toml` only if needed (a path crate is not checked against the registry), `crates/fluxer-voice/src/rtc.rs` (publish `screen_share_audio` with `stereo: true`; the microphone stays mono), `crates/fluxer-voice/README.md` (remove the "Stream audio is mono end to end today" paragraph; describe the patch), `docs/parity/screenshare.md` (stream audio row), `xtask` core-isolation/README checks if they walk crate directories (exclude `third_party`). - Test: `third_party/livekit` unit tests for the patch; `crates/fluxer-voice/tests/livekit_interop.rs` new test. **Interfaces:** - `livekit::options::TrackPublishOptions { stereo: bool, .. }` (default `false`, so every existing caller is unchanged). - With `stereo: true` on an audio track: `AddTrackRequest.stereo = true` and `audio_features` contains `TfStereo`; when the publisher applies the server's answer, the Opus `a=fmtp` line of that track's `m=audio` section (matched by its mid / transceiver) gets `stereo=1;sprop-stereo=1` if absent, and no other section changes. Mono tracks and video are untouched. - [ ] **Step 1: Write failing tests**: in `third_party/livekit`: `add_track_request_marks_stereo` (request built for a stereo audio track has `stereo` and `TfStereo`; mono has neither), `answer_munging_sets_stereo_only_for_stereo_mid` (an SDP with two Opus audio sections: only the stereo track's fmtp gains `stereo=1;sprop-stereo=1`; CRLF/LF preserved; existing `stereo=` left alone). In `fluxer-voice` (behind `FLUXER_LIVEKIT_TESTS=1`, like the other interop tests, run by `cargo xtask ci` on the Linux runner): `screen_share_audio_arrives_in_stereo` — publish a screen share whose audio carries a 440 Hz tone on the left channel and 660 Hz on the right, subscribe from a second session, and assert the received left and right channels differ and each is dominated by its own frequency (Goertzel or FFT bin energy ratio ≥ 10:1 after 2 s of audio). - [ ] **Step 2:** run them, see them fail (the interop test fails against unpatched 0.10.1 because both channels carry the same mix). - [ ] **Step 3:** vendor 0.10.1 verbatim first and commit that alone (`chore: vendor livekit 0.10.1 unchanged`), so the patch commit's diff is only the change; then implement the patch, set `st 🤖 Generated with [Claude Code](https://claude.com/claude-code)
charles added this to the Text chat milestone 2026-11-12 20:46:12 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#354
No description provided.