Voices that stop are logged where they stop; playback restarts itself; departed participants leave the mix (#931) #932

Closed
charles wants to merge 0 commits from fix-931-voices-silent-after-long-uptime into textchat
Owner

Refs #931. This uses Refs, not Closes, because the cause is not demonstrated (see below), and merging should not close the issue. Change it to Closes if you want it closed.

Status: diagnostics and two hardening fixes. The cause of #931 is not demonstrated yet. No log of the failure exists, and every path in our code that I could drive in tests keeps voices playing. This PR makes the next occurrence say where the audio stopped, removes two ways it could stay silent, and fixes a leak found on the way.

What the investigation found

Remote voices and every other sound (chimes, soundboard, UI sounds) go through one AudioEngine, one Mixer and one output stream. So "voices silent, other sounds fine" cannot be the device, the PipeWire/WASAPI stream or the mixer output. It has to start upstream of the mixer:

LiveKit/WebRTC receive → NativeAudioStream (rtc.rs read_track) → room event channel → session (session.rs, gated by link.subscribed) → remote_audio() channel → the playback task in voice_rt.rs → AudioEngine::push_remote. Or the participant gain (SourceId::Remote) going to 0. That fits both Linux and Windows.

Ruled out by tests (new, in apps/fluxer/tests/voice_runtime.rs; they passed before this change too):

  • voices_are_heard_after_a_full_reconnect: the SDK drops all participants, announces them again, then Reconnected. The store subscribes again and the voice is heard.
  • voices_are_heard_after_resumed_connections: three resumes in a row.
  • voices_are_heard_across_calls_and_deafens: four calls, with a deafen/undeafen in each, then leave and rejoin.

Read and judged unlikely (with the reason):

  • Mixer prefill and queue (prefill <= MAX_QUEUE_MS, so a source can always start), playout target and ring size: all shared with the clips, which keep playing.
  • Per-participant gain: sent again whenever a participant is new to the store, and after a full reconnect.
  • NativeAudioStream's bounded queue drops the oldest frames but never stalls. capture_frame for the microphone and stream audio uses the synchronous queue-0 path, so it cannot block the session.
  • publish_data waits on the SDK's data-channel buffered-amount bookkeeping (2 MiB threshold). In theory it could hang the session if that count drifted, but nothing showed it.

Not reachable in tests and still possible: the session task stuck awaiting a room call; libwebrtc's process-wide synthetic audio device no longer pulling remote audio (webrtc-sys synthetic_audio_device.cpp / adm_proxy.cpp; the runtime is kept for the whole process, which would explain "only a restart fixes it"); or end-to-end decryption failing (the experimental setting). The diagnostics below tell these apart.

Changes

  1. Silent tracks are logged (fluxer-voice/src/audio_watch.rs, used by the session). A received WebRTC audio track hands out 10 ms every 10 ms, silence included. So a subscribed, unmuted track that delivers nothing for 5 s is logged at warn once per stall, with the track id, how long, and how many subscribed tracks are silent. When every track is silent, the line says that no remote voice can be heard. A track that delivers again is logged at info. Nothing is checked while the room reconnects. Unit tests in the module.
  2. Session logs: remote audio arriving with no reader; the reader closing its channel (before, the sink was dropped silently and voices stayed silent until a restart); 50 frames in a row dropped because the reader is full, and when it catches up. A room call that has not returned after 5 s is logged with its name, because no remote audio is passed on while the session waits for it (slow(), tested so it waits and never cancels). LiveKitRoom now logs a track stream that ends by itself, or that delivers frames of another format (both were dropped silently).
  3. The playback task is supervised (voice_rt::supervise_playback). If its channel closes or play panics, that is logged and a new channel is read: after 1 s, doubling up to 60 s while it keeps failing. Before, every voice stayed silent for the rest of the run. Tests: the_remote_audio_playback_reads_again_after_its_channel_closed, ..._starts_again_after_a_panic, ..._backs_off_while_its_channel_keeps_closing.
  4. Leak fixed: participants never left the mixer. Nothing called AudioEngine::remove_remote, so every identity heard during a run stayed a mixer source for good, and the loudspeaker never closed again. Now a participant who leaves leaves the mix (microphone and stream sound). Leaving or replacing the room takes them all out (remove_all_remote). The engine also drops a remote source that was given nothing for 5 s, which covers frames that arrive after the room was left (two channels, as in #908). Test leaving_takes_the_voices_out_of_the_mix failed before this change and passes now. Also removing_every_remote_source_releases_the_loudspeaker (engine) and idle_sources_are_removed_and_keep_their_gain (mixer). I could not show that this leak silences voices.

Verification

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passes locally (Linux).

What to ask the reporter next time it happens (before restarting)

Logs are in the log directory (flaxative.<date>.log, default filter info). Please send the lines from the last join onward that contain voice:. The lines that tell the causes apart are:

  • voice: a subscribed audio track delivers no audio, and neither does any other → the audio stopped inside LiveKit/WebRTC (the receive side or its audio device), not in our playback.
  • voice: the room has not answered a call for 5 s (with call = …) → the session is stuck in that room call.
  • voice: the remote audio playback stopped reading / remote audio arrives but nothing reads it / the remote audio channel closed / playback panicked → our playback path, which now restarts itself.
  • voice: remote audio is dropped: the playback does not keep up → the runtime is overloaded.
  • No such line at all, while others speak → the frames reach the mixer. Then check the user volume of those participants and the output volume.

Questions: Does leaving and rejoining the call bring voices back, or only a restart? Do the speaking rings still light up when people talk? Is screen-share sound silent too? Was the experimental end-to-end encryption on? Did the machine sleep or change audio devices before it happened? Roughly how many hours had the app been open?

🤖 Generated with Claude Code

https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K

Refs #931. This uses `Refs`, not `Closes`, because the cause is not demonstrated (see below), and merging should not close the issue. Change it to `Closes` if you want it closed. **Status: diagnostics and two hardening fixes. The cause of #931 is not demonstrated yet.** No log of the failure exists, and every path in our code that I could drive in tests keeps voices playing. This PR makes the next occurrence say where the audio stopped, removes two ways it could stay silent, and fixes a leak found on the way. ## What the investigation found Remote voices and every other sound (chimes, soundboard, UI sounds) go through **one** `AudioEngine`, one `Mixer` and one output stream. So "voices silent, other sounds fine" cannot be the device, the PipeWire/WASAPI stream or the mixer output. It has to start **upstream of the mixer**: LiveKit/WebRTC receive → `NativeAudioStream` (`rtc.rs` `read_track`) → room event channel → session (`session.rs`, gated by `link.subscribed`) → `remote_audio()` channel → the playback task in `voice_rt.rs` → `AudioEngine::push_remote`. Or the participant gain (`SourceId::Remote`) going to 0. That fits both Linux and Windows. Ruled out by tests (new, in `apps/fluxer/tests/voice_runtime.rs`; they passed before this change too): - `voices_are_heard_after_a_full_reconnect`: the SDK drops all participants, announces them again, then `Reconnected`. The store subscribes again and the voice is heard. - `voices_are_heard_after_resumed_connections`: three resumes in a row. - `voices_are_heard_across_calls_and_deafens`: four calls, with a deafen/undeafen in each, then leave and rejoin. Read and judged unlikely (with the reason): - Mixer prefill and queue (`prefill <= MAX_QUEUE_MS`, so a source can always start), playout target and ring size: all shared with the clips, which keep playing. - Per-participant gain: sent again whenever a participant is new to the store, and after a full reconnect. - `NativeAudioStream`'s bounded queue drops the oldest frames but never stalls. `capture_frame` for the microphone and stream audio uses the synchronous queue-0 path, so it cannot block the session. - `publish_data` waits on the SDK's data-channel buffered-amount bookkeeping (2 MiB threshold). In theory it could hang the session if that count drifted, but nothing showed it. Not reachable in tests and still possible: the session task stuck awaiting a room call; libwebrtc's process-wide synthetic audio device no longer pulling remote audio (`webrtc-sys` `synthetic_audio_device.cpp` / `adm_proxy.cpp`; the runtime is kept for the whole process, which would explain "only a restart fixes it"); or end-to-end decryption failing (the experimental setting). The diagnostics below tell these apart. ## Changes 1. **Silent tracks are logged** (`fluxer-voice/src/audio_watch.rs`, used by the session). A received WebRTC audio track hands out 10 ms every 10 ms, silence included. So a subscribed, unmuted track that delivers nothing for 5 s is logged at `warn` once per stall, with the track id, how long, and how many subscribed tracks are silent. When every track is silent, the line says that no remote voice can be heard. A track that delivers again is logged at `info`. Nothing is checked while the room reconnects. Unit tests in the module. 2. **Session logs**: remote audio arriving with no reader; the reader closing its channel (before, the sink was dropped silently and voices stayed silent until a restart); 50 frames in a row dropped because the reader is full, and when it catches up. **A room call that has not returned after 5 s** is logged with its name, because no remote audio is passed on while the session waits for it (`slow()`, tested so it waits and never cancels). `LiveKitRoom` now logs a track stream that ends by itself, or that delivers frames of another format (both were dropped silently). 3. **The playback task is supervised** (`voice_rt::supervise_playback`). If its channel closes or `play` panics, that is logged and a new channel is read: after 1 s, doubling up to 60 s while it keeps failing. Before, every voice stayed silent for the rest of the run. Tests: `the_remote_audio_playback_reads_again_after_its_channel_closed`, `..._starts_again_after_a_panic`, `..._backs_off_while_its_channel_keeps_closing`. 4. **Leak fixed: participants never left the mixer.** Nothing called `AudioEngine::remove_remote`, so every identity heard during a run stayed a mixer source for good, and the loudspeaker never closed again. Now a participant who leaves leaves the mix (microphone and stream sound). Leaving or replacing the room takes them all out (`remove_all_remote`). The engine also drops a remote source that was given nothing for 5 s, which covers frames that arrive after the room was left (two channels, as in #908). Test `leaving_takes_the_voices_out_of_the_mix` failed before this change and passes now. Also `removing_every_remote_source_releases_the_loudspeaker` (engine) and `idle_sources_are_removed_and_keep_their_gain` (mixer). I could not show that this leak silences voices. ## Verification `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passes locally (Linux). ## What to ask the reporter next time it happens (before restarting) Logs are in the log directory (`flaxative.<date>.log`, default filter `info`). Please send the lines from the last join onward that contain `voice:`. The lines that tell the causes apart are: - `voice: a subscribed audio track delivers no audio, and neither does any other` → the audio stopped inside LiveKit/WebRTC (the receive side or its audio device), not in our playback. - `voice: the room has not answered a call for 5 s` (with `call = …`) → the session is stuck in that room call. - `voice: the remote audio playback stopped reading` / `remote audio arrives but nothing reads it` / `the remote audio channel closed` / `playback panicked` → our playback path, which now restarts itself. - `voice: remote audio is dropped: the playback does not keep up` → the runtime is overloaded. - No such line at all, while others speak → the frames reach the mixer. Then check the user volume of those participants and the output volume. Questions: Does leaving and rejoining the call bring voices back, or only a restart? Do the speaking rings still light up when people talk? Is screen-share sound silent too? Was the experimental end-to-end encryption on? Did the machine sleep or change audio devices before it happened? Roughly how many hours had the app been open? 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Remote voices and every other sound share one audio engine, mixer and output
stream, so a silence of voices alone has to start upstream of the mixer: in
the LiveKit room, the session, or the task that hands the session's frames to
the engine. Nothing on that path said anything when it stopped. Now:

- fluxer-voice: `AudioWatch` logs, once per stall, a subscribed unmuted audio
  track that delivers nothing for 5 s (WebRTC hands out 10 ms every 10 ms,
  silence included), says when all of them are silent at once, and logs when
  one delivers again. The session logs remote audio arriving with no reader,
  the reader closing its channel, and sustained drops. A call into the room
  that has not returned after 5 s is logged with its name (the session does
  one thing at a time, so no remote audio is passed on meanwhile).
  LiveKitRoom logs a track stream that ends by itself or delivers another
  format, both silently dropped before.
- fluxer (voice_rt): the remote audio playback task is supervised: if its
  channel closes or it panics it is logged and a new channel is read (1 s,
  doubling to 60 s while it keeps failing), instead of staying silent until a
  restart.
- fluxer-audio / voice_rt: participants' mixer sources were never removed
  (nothing called remove_remote), so every identity ever heard stayed in the
  mix for the rest of the run and kept the loudspeaker open. A participant
  who leaves leaves the mix (microphone and stream sound), leaving or
  replacing the room takes them all out (remove_all_remote), and a remote
  source given nothing for 5 s is dropped by the engine.

Regression tests: voices are heard after a full reconnect, after resumed
connections, and across calls with deafen cycles (all passed before this
change too, ruling those paths out); leaving releases the loudspeaker (failed
before).

Refs #931

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Author
Owner

Merged into textchat by hand (the merge API returns 405), in 8a5a29193; the combined tree passed the full local gate.

Merged into textchat by hand (the merge API returns 405), in 8a5a29193; the combined tree passed the full local gate.
charles closed this pull request 2026-11-19 10:45:34 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.