Name a stalled WebRTC audio device, and remake the runtime after it (#931) #1002

Closed
charles wants to merge 0 commits from fix-931-silent-voices into textchat
Owner

Refs #931. Status: no reproduction. This PR adds a mitigation and sharper diagnostics. The cause is still unknown, so it uses Refs, not Closes. Change it to Closes #931 if you want merging to close the issue.

What was tried (local livekit-server 1.12.0, the instance's version, no audio device opened)

Kept as regression tests in crates/fluxer-voice/tests/livekit_interop.rs (env-gated like the others; cargo xtask ci runs them with FLUXER_LIVEKIT_TESTS=1):

  • a_voice_is_heard_after_being_alone_and_after_rejoining: the reported sequence. Alice joins alone and keeps feeding her muted microphone, as the app does. Bob joins later, unmutes and plays a tone, and Alice subscribes and hears it. Then Alice leaves and joins again on the same session (same process, same kept WebRTC runtime) and still hears it. It uses 3 s alone by default. 960 s alone (longer than the report's 15 min) also passed, run by hand with FLUXER_931_ALONE_SECS=960.
  • voices_are_heard_across_many_calls_of_one_session: one session, a voice joins and leaves 6 times, with unsubscribe/resubscribe and rejoins. 20 rounds were also run by hand and passed.
  • a_subscribed_track_delivers_while_its_sender_sends_nothing: a subscribed, unmuted track whose sender never sends anything still delivers about 100 frames/s, because libwebrtc's audio device pull runs NetEq whether packets arrive or not. So the #932 watcher's premise holds. The report's "no frames at all" means one of two things. Either the SDK never handed the track over (a subscription that did not complete), or the process-wide WebRTC audio device (webrtc-sys SyntheticAudioDevice, the 10 ms task that calls NeedMorePlayData) stopped pulling.

Read and ruled out or judged unlikely:

  • third_party/webrtc-sys/src/synthetic_audio_device.cpp: playout is a flag (playing_) checked by a 10 ms repeating task that lives from Init() to Terminate(). Its only Terminate() caller is the factory's destruction, and the client keeps the factory for the whole run. libwebrtc's AudioState::AddReceivingStream calls StartPlayout whenever Playing() is false, so a playout left stopped is restarted by the next receive stream, and a rejoin would fix it. The report says a rejoin does not, so a stopped flag does not fit. A dead or hung pump task, or a broken mixer, would fit, but I could not cause either from outside libwebrtc.
  • Our side: everything between the SDK and the #932 watcher is per room (Pump, read_track, the event channel), and the session's loop demonstrably ran (the watcher logged). The kept runtime is the only per-process state in fluxer-voice. LiveKitRoom does not leak the runtime across joins (checked: closing a room releases its hold).
  • A caveat on the evidence: the user-leave chime at 05:04:18Z, before the restart, says the remote participant left. The join after the restart had a participant again (participants=1). So "restart fixes it" also coincides with the remote reconnecting. A broken publication on the remote's side cannot be excluded. The new log lines below tell these cases apart.

Changes

  1. Diagnostics (crates/fluxer-voice/src/rtc.rs):
    • voice: subscribed to an audio track is now logged at info (it was debug, so the reported log could not say whether the SDK ever handed the track over).
    • TrackSubscriptionFailed, which was silently ignored, is logged at warn.
    • New: a handed-over audio track that delivers no frame at all within 3 s is logged at warn: WebRTC hands a subscribed audio track no audio at all: its audio device does not pull audio …. This is the line that names the process-wide audio device.
  2. Mitigation (labelled as such in the code): after that warning, the next join makes the SDK's WebRTC runtime again: a new peer connection factory, audio device module and audio state, which is what a restart did. It logs voice: WebRTC runtime made again. The SDK hands out its current runtime for as long as anything holds it, so the vendored livekit gets a small patch, LkRuntime::forget_instance(), recorded in third_party/livekit/PATCHES.md. Test: a_stalled_audio_device_makes_the_runtime_again_at_the_next_join (the stall itself cannot be caused from outside libwebrtc, so the test declares it). It checks that the join after the stall uses a new factory although the old one is still held, and that a voice is heard with it. The whole interop suite also passed afterwards in the same process (22/22), running on the new runtime.
  3. crates/fluxer-voice/README.md describes both.

What the next log should say

  • No subscribed to an audio track for the silent track (or could not subscribe): the subscription never completed. That is the SDK or server side, or the remote's publication. A rebuilt runtime does not help, and the mitigation does not fire.
  • hands a subscribed audio track no audio at all: the process's WebRTC audio device stopped. Leave and rejoin should now bring voices back (look for WebRTC runtime made again). If it does, that confirms the cause's location, and the next step is instrumenting SyntheticAudioDevice itself.

Not verified on Windows. The changed code is shared and has no platform branches. Only Linux ran here.

Verification

  • FLUXER_LIVEKIT_TESTS=1 cargo test -p fluxer-voice --test livekit_interop -- --ignored --test-threads=1: 22 passed.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: ==> xtask ci: all steps passed (livekit-interop skipped there; run separately above)

🤖 Generated with Claude Code

Refs #931. **Status: no reproduction. This PR adds a mitigation and sharper diagnostics. The cause is still unknown**, so it uses `Refs`, not `Closes`. Change it to `Closes #931` if you want merging to close the issue. ## What was tried (local `livekit-server` 1.12.0, the instance's version, no audio device opened) Kept as regression tests in `crates/fluxer-voice/tests/livekit_interop.rs` (env-gated like the others; `cargo xtask ci` runs them with `FLUXER_LIVEKIT_TESTS=1`): - `a_voice_is_heard_after_being_alone_and_after_rejoining`: the reported sequence. Alice joins alone and keeps feeding her muted microphone, as the app does. Bob joins later, unmutes and plays a tone, and Alice subscribes and hears it. Then Alice leaves and joins again **on the same session** (same process, same kept WebRTC runtime) and still hears it. It uses 3 s alone by default. **960 s alone (longer than the report's 15 min) also passed**, run by hand with `FLUXER_931_ALONE_SECS=960`. - `voices_are_heard_across_many_calls_of_one_session`: one session, a voice joins and leaves 6 times, with unsubscribe/resubscribe and rejoins. 20 rounds were also run by hand and passed. - `a_subscribed_track_delivers_while_its_sender_sends_nothing`: a subscribed, unmuted track whose sender **never sends anything** still delivers about 100 frames/s, because libwebrtc's audio device pull runs NetEq whether packets arrive or not. So the #932 watcher's premise holds. The report's "no frames at all" means one of two things. Either the SDK never handed the track over (a subscription that did not complete), or the process-wide WebRTC audio device (`webrtc-sys` `SyntheticAudioDevice`, the 10 ms task that calls `NeedMorePlayData`) stopped pulling. Read and ruled out or judged unlikely: - `third_party/webrtc-sys/src/synthetic_audio_device.cpp`: playout is a flag (`playing_`) checked by a 10 ms repeating task that lives from `Init()` to `Terminate()`. Its only `Terminate()` caller is the factory's destruction, and the client keeps the factory for the whole run. libwebrtc's `AudioState::AddReceivingStream` calls `StartPlayout` whenever `Playing()` is false, so a playout left stopped is restarted by the next receive stream, and a rejoin would fix it. The report says a rejoin does not, so a stopped flag does not fit. A dead or hung pump task, or a broken mixer, would fit, but I could not cause either from outside libwebrtc. - Our side: everything between the SDK and the #932 watcher is per room (`Pump`, `read_track`, the event channel), and the session's loop demonstrably ran (the watcher logged). The kept runtime is the only per-process state in `fluxer-voice`. `LiveKitRoom` does not leak the runtime across joins (checked: closing a room releases its hold). - **A caveat on the evidence:** the `user-leave` chime at 05:04:18Z, before the restart, says the remote participant left. The join after the restart had a participant again (participants=1). So "restart fixes it" also coincides with the remote reconnecting. A broken publication on the remote's side cannot be excluded. The new log lines below tell these cases apart. ## Changes 1. **Diagnostics** (`crates/fluxer-voice/src/rtc.rs`): - `voice: subscribed to an audio track` is now logged at `info` (it was `debug`, so the reported log could not say whether the SDK ever handed the track over). - `TrackSubscriptionFailed`, which was silently ignored, is logged at `warn`. - New: a handed-over audio track that delivers no frame at all within 3 s is logged at `warn`: `WebRTC hands a subscribed audio track no audio at all: its audio device does not pull audio …`. This is the line that names the process-wide audio device. 2. **Mitigation** (labelled as such in the code): after that warning, the **next join makes the SDK's WebRTC runtime again**: a new peer connection factory, audio device module and audio state, which is what a restart did. It logs `voice: WebRTC runtime made again`. The SDK hands out its current runtime for as long as anything holds it, so the vendored `livekit` gets a small patch, `LkRuntime::forget_instance()`, recorded in `third_party/livekit/PATCHES.md`. Test: `a_stalled_audio_device_makes_the_runtime_again_at_the_next_join` (the stall itself cannot be caused from outside libwebrtc, so the test declares it). It checks that the join after the stall uses a new factory although the old one is still held, and that a voice is heard with it. The whole interop suite also passed afterwards in the same process (22/22), running on the new runtime. 3. `crates/fluxer-voice/README.md` describes both. ## What the next log should say - No `subscribed to an audio track` for the silent track (or `could not subscribe`): the subscription never completed. That is the SDK or server side, or the remote's publication. A rebuilt runtime does not help, and the mitigation does not fire. - `hands a subscribed audio track no audio at all`: the process's WebRTC audio device stopped. Leave and rejoin should now bring voices back (look for `WebRTC runtime made again`). If it does, that confirms the cause's location, and the next step is instrumenting `SyntheticAudioDevice` itself. Not verified on Windows. The changed code is shared and has no platform branches. Only Linux ran here. ## Verification - `FLUXER_LIVEKIT_TESTS=1 cargo test -p fluxer-voice --test livekit_interop -- --ignored --test-threads=1`: 22 passed. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci`: `==> xtask ci: all steps passed` (livekit-interop skipped there; run separately above) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Not a fix: #931 is not reproduced locally (alone for 960 s, then a voice joins;
rejoins on the same session; many calls in one session: all heard). The tests
for that stay, env-gated, with one showing that a subscribed track delivers
frames even when its sender sends nothing, which the diagnostics rely on.

- LiveKitRoom logs at info each audio track the SDK hands over, and at warn a
  subscription the SDK gives up on (it was ignored).
- A handed-over track that delivers no frame within 3 s means the process's
  WebRTC audio device stopped pulling remote audio: logged at warn, and the
  next join makes the SDK's WebRTC runtime again (mitigation), through a new
  vendored-SDK patch, LkRuntime::forget_instance (third_party/livekit/PATCHES.md).

Refs #931

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged by hand into textchat (8f270714) with the other open PRs; the combined tree passed the full gate locally, LiveKit interop included.

Merged by hand into textchat (8f270714) with the other open PRs; the combined tree passed the full gate locally, LiveKit interop included.
charles closed this pull request 2026-11-20 07:25:19 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.