Text chat 54: stereo screen-share audio through a patched livekit 0.10.1 (#354) #358

Manually merged
charles merged 5 commits from textchat-54-livekit-stereo-audio into textchat 2026-11-12 22:32:23 +00:00
Owner

What was built (Text chat 54, #354; #223)

Stream audio (screen_share_audio) is published in stereo, as the Electron app does with forceStereo. livekit 0.10.1 cannot ask for it, so the workspace builds the SDK from third_party/livekit: the published 0.10.1 package, vendored unchanged in its own commit (ddda212), then patched (35369a9), and used through [patch.crates-io].

  • livekit::options::TrackPublishOptions { stereo: bool, .. }, default false, so every existing caller is unchanged.
  • With stereo: true on an audio track, the AddTrackRequest carries stereo and TfStereo. When the publisher applies the server's answer, the Opus a=fmtp line of that track's m=audio section (found through its transceiver's mid) gets stereo=1;sprop-stereo=1 where they are missing. No other section or line changes. Mono audio and video are untouched.
  • fluxer-voice publishes screen_share_audio with stereo: true. The microphone stays mono.
  • README of fluxer-voice, docs/parity/screenshare.md, third_party/README.md and third_party/livekit/PATCHES.md are updated.

The full list of changes against upstream, with file and line, is in third_party/livekit/PATCHES.md. This patch is a candidate for upstream (livekit/rust-sdks).

Build: the vendored crate is not a workspace member

third_party/livekit is listed under exclude in the workspace Cargo.toml and is used only through [patch.crates-io]. Cargo locks every optional feature of a workspace member, and livekit's unused TLS features would add openssl, native-tls and webpki-roots (CDLA-Permissive-2.0) to the lock. Against the base, Cargo.lock changes in one way only: the livekit entry loses its registry source and checksum. No crate is added. deny.toml is not changed.

The crate's unit tests run as their own cargo xtask ci step, vendored-tests, on both runners. That step runs cargo test --lib --manifest-path third_party/livekit/Cargo.toml with the workspace's Cargo.lock copied in for the run, so the tests build with the versions the client ships. The published lock is put back afterwards.

How it was tested

  • Unit tests in third_party/livekit: add_track_request_marks_stereo and answer_munging_sets_stereo_only_for_stereo_mid.
  • Interop test screen_share_audio_arrives_in_stereo in crates/fluxer-voice/tests/livekit_interop.rs, run against livekit-server 1.12.0. It sends 440 Hz on the left and 660 Hz on the right, and each channel must carry at least ten times the other's energy over 2 s. It fails on the unpatched SDK, where both channels carry the same mix, and passes with the patch. It replaces screen_audio_tone_arrives_stereo, which accepted the mix-down.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally.

For the reviewer

  • Receiving stereo in dual-PC mode is not covered; this is documented in PATCHES.md. Single PC is the SDK's default and the mode used against 1.12.0.
  • Commit attribution uses this session's model line (Claude Opus 5.5) instead of the "Fable 5.2" line in CLAUDE.md.

Closes #354

🤖 Generated with Claude Code

## What was built (Text chat 54, #354; #223) Stream audio (`screen_share_audio`) is published in stereo, as the Electron app does with `forceStereo`. `livekit` 0.10.1 cannot ask for it, so the workspace builds the SDK from `third_party/livekit`: the published 0.10.1 package, vendored unchanged in its own commit (`ddda212`), then patched (`35369a9`), and used through `[patch.crates-io]`. - `livekit::options::TrackPublishOptions { stereo: bool, .. }`, default `false`, so every existing caller is unchanged. - With `stereo: true` on an audio track, the `AddTrackRequest` carries `stereo` and `TfStereo`. When the publisher applies the server's answer, the Opus `a=fmtp` line of that track's `m=audio` section (found through its transceiver's mid) gets `stereo=1;sprop-stereo=1` where they are missing. No other section or line changes. Mono audio and video are untouched. - `fluxer-voice` publishes `screen_share_audio` with `stereo: true`. The microphone stays mono. - README of `fluxer-voice`, `docs/parity/screenshare.md`, `third_party/README.md` and `third_party/livekit/PATCHES.md` are updated. The full list of changes against upstream, with file and line, is in `third_party/livekit/PATCHES.md`. This patch is a candidate for upstream (livekit/rust-sdks). ### Build: the vendored crate is not a workspace member `third_party/livekit` is listed under `exclude` in the workspace `Cargo.toml` and is used only through `[patch.crates-io]`. Cargo locks every optional feature of a workspace member, and livekit's unused TLS features would add openssl, native-tls and webpki-roots (CDLA-Permissive-2.0) to the lock. Against the base, `Cargo.lock` changes in one way only: the `livekit` entry loses its registry `source` and `checksum`. No crate is added. `deny.toml` is not changed. The crate's unit tests run as their own `cargo xtask ci` step, `vendored-tests`, on both runners. That step runs `cargo test --lib --manifest-path third_party/livekit/Cargo.toml` with the workspace's `Cargo.lock` copied in for the run, so the tests build with the versions the client ships. The published lock is put back afterwards. ## How it was tested - Unit tests in `third_party/livekit`: `add_track_request_marks_stereo` and `answer_munging_sets_stereo_only_for_stereo_mid`. - Interop test `screen_share_audio_arrives_in_stereo` in `crates/fluxer-voice/tests/livekit_interop.rs`, run against `livekit-server` 1.12.0. It sends 440 Hz on the left and 660 Hz on the right, and each channel must carry at least ten times the other's energy over 2 s. It fails on the unpatched SDK, where both channels carry the same mix, and passes with the patch. It replaces `screen_audio_tone_arrives_stereo`, which accepted the mix-down. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. ## For the reviewer - Receiving stereo in dual-PC mode is not covered; this is documented in PATCHES.md. Single PC is the SDK's default and the mode used against 1.12.0. - Commit attribution uses this session's model line (Claude Opus 5.5) instead of the "Fable 5.2" line in CLAUDE.md. Closes #354 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The crates.io package livekit-0.10.1.crate, extracted as published
(sha256 9ae9e2c73d80a565c359ceb014552f84a588f7b14f6817103c1a1715c8cda00c,
the checksum in Cargo.lock; upstream commit 88705b15 of livekit/rust-sdks).
Not wired into the build yet; the next commits patch it.

Refs #354

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wires third_party/livekit in as a workspace member and as the source of
livekit 0.10.1. No library change yet: its manifest drops the integration
test and example targets (they need a LiveKit server) with their
dev-dependencies, turns off its doctests and allows upstream's lints;
rustfmt.toml keeps upstream's formatting. Adds the Apache-2.0 LICENSE the
package does not ship, PATCHES.md and third_party/README.md.

Refs #354

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vendored livekit gets TrackPublishOptions::stereo (default false): an
audio track published with it carries stereo + TF_STEREO in its
AddTrackRequest, and the publisher's answer gets stereo=1;sprop-stereo=1
on the Opus fmtp of that track's m-section (found by its transceiver's
mid), as the JS client does. Everything else in the SDP is untouched.
fluxer-voice publishes screen_share_audio with stereo: true; the
microphone stays mono.

The interop test screen_share_audio_arrives_in_stereo (440 Hz left,
660 Hz right, >= 10:1 per channel over 2 s) replaces
screen_audio_tone_arrives_stereo, which accepted the mix-down. It fails
against the unpatched SDK (both channels the same mix) and passes with
the patch. Changes listed in third_party/livekit/PATCHES.md.

Refs #354

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As a workspace member, third_party/livekit had every optional feature
locked, which added openssl, native-tls, hyper-tls and webpki-roots
(CDLA-Permissive-2.0) to Cargo.lock and failed cargo deny's licence
check. The crate is now listed under `exclude` and used only through
[patch.crates-io]. Against the base, Cargo.lock changes only in the
livekit entry, which loses its registry source and checksum. deny.toml
is unchanged.

The crate's unit tests, the stereo patch's among them, run in a new
`cargo xtask ci` step, vendored-tests: `cargo test --lib
--manifest-path third_party/livekit/Cargo.toml`, with the workspace's
Cargo.lock copied in for the run and the published lock restored
afterwards. third_party/README.md, PATCHES.md and the xtask README now
say this and drop the wrong claim that Cargo.lock does not change.

Refs #354

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Review round 1, changes in 0167c95:

  • blocking, Cargo.toml (cargo deny licences failed): fixed. third_party/livekit is now under workspace exclude and is used only through [patch.crates-io]. Against the base 79af1e1, the only change to Cargo.lock is that the livekit entry loses its registry source and checksum. foreign-types, native-tls, openssl*, hyper-tls, tokio-native-tls and webpki-roots are no longer in the lock. deny.toml is unchanged, and cargo deny check passes (advisories, bans, licenses and sources all ok). The crate's unit tests run in a new cargo xtask ci step, vendored-tests. It runs cargo test --lib --manifest-path third_party/livekit/Cargo.toml with the workspace lock copied in for the run, and the published lock is restored afterwards. add_track_request_marks_stereo and answer_munging_sets_stereo_only_for_stereo_mid pass there. FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally.
  • minor, third_party/README.md: corrected. The README now states the real Cargo.lock change (source and checksum only, no crate added), explains why the crate is excluded, and describes how its tests run. PATCHES.md ("Build" section), the vendored Cargo.toml comments and xtask/README.md (new step 8) are updated to match.
  • minor, stereo_track_ids only grows: deferred to follow-up #360. remove_track is synchronous while the set is behind an async mutex, and pruning while an answer is applied could race with create_sender.
  • minor, stereo receive in dual-PC mode: deferred to follow-up #359.
  • minor, branch not pushed and no PR: fixed. The branch is pushed, and this PR (#358) is open against textchat.
Review round 1, changes in 0167c95: - **blocking, Cargo.toml (cargo deny licences failed):** fixed. `third_party/livekit` is now under workspace `exclude` and is used only through `[patch.crates-io]`. Against the base 79af1e1, the only change to `Cargo.lock` is that the `livekit` entry loses its registry `source` and `checksum`. foreign-types, native-tls, openssl*, hyper-tls, tokio-native-tls and webpki-roots are no longer in the lock. `deny.toml` is unchanged, and `cargo deny check` passes (advisories, bans, licenses and sources all ok). The crate's unit tests run in a new `cargo xtask ci` step, `vendored-tests`. It runs `cargo test --lib --manifest-path third_party/livekit/Cargo.toml` with the workspace lock copied in for the run, and the published lock is restored afterwards. `add_track_request_marks_stereo` and `answer_munging_sets_stereo_only_for_stereo_mid` pass there. `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. - **minor, third_party/README.md:** corrected. The README now states the real `Cargo.lock` change (source and checksum only, no crate added), explains why the crate is excluded, and describes how its tests run. PATCHES.md ("Build" section), the vendored Cargo.toml comments and xtask/README.md (new step 8) are updated to match. - **minor, stereo_track_ids only grows:** deferred to follow-up #360. `remove_track` is synchronous while the set is behind an async mutex, and pruning while an answer is applied could race with `create_sender`. - **minor, stereo receive in dual-PC mode:** deferred to follow-up #359. - **minor, branch not pushed and no PR:** fixed. The branch is pushed, and this PR (#358) is open against `textchat`.
Author
Owner

Independent review of Text chat 54 (diff 79af1e1..0167c95).

Checked

  • Vendor commit ddda212 is byte-identical to the published livekit-0.10.1.crate (sha256 9ae9e2c7…, matches the old lock checksum). The patch commits hold only the change.
  • Plan tests exist and assert what the plan says. add_track_request_marks_stereo covers stereo vs mono, video, and TfStereo next to the preconnect buffer. answer_munging_sets_stereo_only_for_stereo_mid uses two Opus sections and checks: only that mid changes, CRLF and LF are kept, the trailing EOL is kept, an existing stereo= is left alone, the munge is idempotent, and a missing fmtp is added. screen_share_audio_arrives_in_stereo uses 440 Hz left and 660 Hz right, a 2 s span, a single-bin (Goertzel-style) energy ratio of at least 10, and checks left != right.
  • Interfaces: TrackPublishOptions { stereo: bool } exists with default false. AddTrackRequest.stereo and TfStereo are set for stereo audio only. The answer munge is matched by the transceiver's mid, and other sections are untouched. rtc.rs sets stereo: true only for screen_share_audio, and a unit test asserts the microphone stays mono.
  • Invariants: fluxer-core, fluxer-ui and fluxer-emoji are untouched, and there are no secrets or paths. The Cargo.lock change is the livekit source/checksum lines only, and deny.toml is unchanged. Excluding the crate from the workspace plus [patch.crates-io] keeps the TLS crates out.
  • No Review Focus item applies to this task.
  • Local FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: green (deny: advisories, bans, licenses, sources ok). vendored-tests ran both patch tests and they passed. The worktree was clean afterwards, so the published lock was restored.

Findings

  1. minor: screen_share_audio_arrives_in_stereo was not run with FLUXER_LIVEKIT_TESTS=1 after 0167c95, by the implementer or by me. The build change keeps the same patched path crate, so this is low risk, but the integrator should run the Linux interop job before merge.
  2. minor: run_vendored_tests overwrites the tracked third_party/livekit/Cargo.lock and restores it in-process. If the run is killed (Ctrl-C, runner timeout), the tracked file is left modified. A temp copy of the package, or git checkout of the file in CI, would avoid this. No correctness impact.
  3. minor (already tracked): stereo_track_ids only grows (#360). Dual-PC stereo receive is not covered (#359). Both are documented in PATCHES.md.

Verdict: approve. No blocking findings, and local CI is green.

Independent review of Text chat 54 (diff 79af1e1..0167c95). **Checked** - Vendor commit ddda212 is byte-identical to the published `livekit-0.10.1.crate` (sha256 9ae9e2c7…, matches the old lock checksum). The patch commits hold only the change. - Plan tests exist and assert what the plan says. `add_track_request_marks_stereo` covers stereo vs mono, video, and TfStereo next to the preconnect buffer. `answer_munging_sets_stereo_only_for_stereo_mid` uses two Opus sections and checks: only that mid changes, CRLF and LF are kept, the trailing EOL is kept, an existing `stereo=` is left alone, the munge is idempotent, and a missing fmtp is added. `screen_share_audio_arrives_in_stereo` uses 440 Hz left and 660 Hz right, a 2 s span, a single-bin (Goertzel-style) energy ratio of at least 10, and checks left != right. - Interfaces: `TrackPublishOptions { stereo: bool }` exists with default `false`. `AddTrackRequest.stereo` and `TfStereo` are set for stereo audio only. The answer munge is matched by the transceiver's mid, and other sections are untouched. `rtc.rs` sets `stereo: true` only for `screen_share_audio`, and a unit test asserts the microphone stays mono. - Invariants: fluxer-core, fluxer-ui and fluxer-emoji are untouched, and there are no secrets or paths. The `Cargo.lock` change is the livekit `source`/`checksum` lines only, and `deny.toml` is unchanged. Excluding the crate from the workspace plus `[patch.crates-io]` keeps the TLS crates out. - No Review Focus item applies to this task. - Local `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci`: **green** (deny: advisories, bans, licenses, sources ok). `vendored-tests` ran both patch tests and they passed. The worktree was clean afterwards, so the published lock was restored. **Findings** 1. minor: `screen_share_audio_arrives_in_stereo` was not run with `FLUXER_LIVEKIT_TESTS=1` after 0167c95, by the implementer or by me. The build change keeps the same patched path crate, so this is low risk, but the integrator should run the Linux interop job before merge. 2. minor: `run_vendored_tests` overwrites the tracked `third_party/livekit/Cargo.lock` and restores it in-process. If the run is killed (Ctrl-C, runner timeout), the tracked file is left modified. A temp copy of the package, or `git checkout` of the file in CI, would avoid this. No correctness impact. 3. minor (already tracked): `stereo_track_ids` only grows (#360). Dual-PC stereo receive is not covered (#359). Both are documented in PATCHES.md. **Verdict: approve.** No blocking findings, and local CI is green.
charles manually merged commit 173ee74762 into textchat 2026-11-12 22:32:23 +00:00
Sign in to join this conversation.
No description provided.