Name a stalled WebRTC audio device, and remake the runtime after it (#931) #1002
Closed
charles
wants to merge 0 commits from
fix-931-silent-voices into textchat
pull from: fix-931-silent-voices
merge into: charles:textchat
charles:main
charles:camera
charles:dailyuse
charles:textchat
charles:fix-1008-idle-in-voice
charles:camera-21-acceptance
charles:feat-1000-app-sounds-output
charles:fix-1001-local-time
charles:camera-20-parity-list-for-the
charles:camera-15-ui-settings-voice-video
charles:camera-14-ui-camera-buttons-camera
charles:camera-16-ui-layout-with-cameras
charles:camera-17-core-toggle-camera-shortcut
charles:camera-13-ui-mirrored-video-and
charles:camera-12-appsfluxer-camera-runtime-and
charles:camera-10-capture-camera-on-linux
charles:camera-11-capture-camera-on-windows
charles:camera-9-voice-the-camera-track
charles:camera-7-core-the-own-camera
charles:camera-6-capture-camera-trait-mode
charles:camera-8-core-remote-cameras-tile
charles:camera-3-spike-a-camera-track
charles:camera-4-video-camera-formats-and
charles:camera-18-devserver-camera-scenario
charles:camera-1-spike-capture-linux
charles:camera-5-core-camera-ports-commands
charles:camera-2-spike-capture-windows
charles:camera-plan
charles:shell
charles:fix-925-keybinds-persist
charles:feat-934-935-stream-view
charles:feat-933-stream-output-device
charles:shell-44-acceptance
charles:feat-946-select-message-text
charles:shell-43-parity-list-for-the
charles:fix-589-black-band-linux-sender
charles:shell-37-ui-system-wide-shortcuts
charles:shell-ship
charles:fix-931-voices-silent-after-long-uptime
charles:shell-39-ui-settings-notifications
charles:shell-36-ui-settings-keybinds
charles:shell-16-core-autostart
charles:fix-923-late-publication-next-share
charles:fix-921-share-teardown-publish-race
charles:shell-1-spike-iced-daemon-hide
charles:shell-10-core-instance-messages-tasks
charles:shell-11-core-keybinds-actions-combos
charles:shell-13-core-badge-attention-sound
charles:shell-14-core-notification-settings-inactive
charles:shell-18-platform-desktop-entry-and
charles:shell-2-spike-tray-on-linux
charles:shell-3-spike-wayland-global-shortcuts
charles:shell-41-devserver-shell-scenario
charles:shell-5-spike-overlay-technique-for
charles:shell-6-core-shell-ports-settings
charles:shell-7-platform-settings-shell-keybinds
charles:shell-8-core-tray-model-and
charles:shell-9-core-window-rules-close
charles:fix-913-keep-awake-retry
charles:shell-35-ui-in-window-voice
charles:shell-sync-textchat
charles:shell-34-app-shortcuts-runtime
charles:fix-913-keep-screen-awake
charles:fix-910-deepfilter-windows-timing
charles:shell-33-app-shell-runtime
charles:shell-28-platform-shortcuts-on-wayland
charles:fix-910-deepfilter-late-frame
charles:shell-29-platform-shortcuts-by-direct
charles:shell-27-platform-shortcuts-on-x11
charles:shell-17-platform-single-instance
charles:shell-24-platform-badge-displays-and-24
charles:shell-26-platform-shortcuts-on-windows
charles:shell-21-platform-tray-on-linux
charles:shell-22-platform-tray-on-windows
charles:fix-908-flaky-stream-end-test
charles:shell-30-platform-wayland-activation
charles:shell-38-ui-settings-desktop
charles:shell-19-platform-autostart
charles:shell-25-platform-shortcut-back-end
charles:shell-23-platform-badge-displays-and
charles:shell-40-ui-companion-apps-control
charles:shell-31-statusapi-default-avatars
charles:shell-12-core-system-wide-shortcut
charles:shell-20-platform-tray-trait-icons
charles:shell-32-ui-daemon-window-lifecycle
charles:shell-4-spike-raising-a-wayland
charles:shell-15-core-status-payload-gaps
charles:fix-876-share-resolution-and-bars
charles:shell-42-packaging-windows-shortcuts-and
charles:fix-874-tests-must-not-notify
charles:fix-868-deepfilternet-start
charles:gamenight-plan
charles:fix-867-869-stream-end-soundboard-limits
charles:feat-229-windows-hardware-encoding
charles:fix-857-vaapi-hardware-encoding
charles:fix-849-inert-cards
charles:fix-848-pinned-messages-clickable
charles:shell-plan
charles:fix-764-nvenc-above-720p30
charles:serveradmin
charles:feat-780-forum-archived-filter
charles:serveradmin-35-acceptance
charles:feat-781-no-join-chime-with-entrance-sound
charles:feat-784-rejoin-voice-after-update
charles:fix-779-782-search-dropdown-paste-preview
charles:fix-783-781-dm-rail-entrance-sound
charles:serveradmin-34-parity-screenshots
charles:serveradmin-31-ui-channel-permissions
charles:fix-773-cargo-deny-advisories
charles:serveradmin-33-ui-sudo-delete-transfer
charles:serveradmin-28-ui-members-page
charles:serveradmin-26-ui-role-permissions
charles:serveradmin-30-ui-voice-overview-channel-invites
charles:serveradmin-fix-27-29
charles:serveradmin-25-ui-roles-tab
charles:serveradmin-27-ui-menus-moderation-modals
charles:serveradmin-29-ui-channel-settings
charles:fix-589-stream-top-strip-encode
charles:serveradmin-24-ui-moderation-invites-bans
charles:feat-760-version-in-title
charles:fix-751-stream-audio-pipewire
charles:serveradmin-20-core-menu-rows
charles:fix-752-linux-notification-panic
charles:serveradmin-17-core-overrides
charles:fix-740-updater-always-latest
charles:serveradmin-23-ui-community-settings-overview
charles:serveradmin-12-core-roles
charles:serveradmin-19-core-sudo-delete-transfer
charles:serveradmin-32-ui-sidebar-drag
charles:serveradmin-15-core-members-page
charles:serveradmin-16-core-channel-settings
charles:serveradmin-14-core-bans-invites
charles:fix-615-restore-on-stored-session-start
charles:serveradmin-22-runtime
charles:serveradmin-13-core-member-moderation
charles:serveradmin-18-core-channel-reorder
charles:fix-732-status-api-speaking
charles:serveradmin-10-core-permission-lists
charles:serveradmin-11-core-forms-community
charles:serveradmin-21b-devserver-channels-members-sudo
charles:serveradmin-4-api-community-roles-sudo
charles:serveradmin-9b-core-hierarchy-bulk-events
charles:serveradmin-21a-devserver-community-roles
charles:serveradmin-9a-core-ports
charles:serveradmin-8-form-widgets
charles:serveradmin-5-api-channels-members-bans
charles:serveradmin-7-settings-shell
charles:serveradmin-1-proto-community-roles
charles:serveradmin-3-proto-members-bans-invites
charles:serveradmin-6-reorder-widget
charles:serveradmin-2-proto-channel-admin
charles:fix-706-presence-after-reconnect
charles:fix-615-start-on-last-view
charles:feat-663-create-channel
charles:serveradmin-plan
charles:fix-662-long-text-wrap
charles:fix-660-661-menu-toolbar-paste-focus
charles:fix-620-forum-list-reactions
charles:fix-657-windows-console
charles:fix-622-emoji-in-names
charles:fix-smoke-645-648
charles:fix-614-install-location
charles:dailyuse-41-acceptance
charles:fix/597-copy-id
charles:dailyuse-40-parity-screenshots
charles:dailyuse-35-ui-quick-switcher-keybinds
charles:dailyuse-38-ui-profile-modal
charles:fix/voice-view
charles:fix/side-panel-composer
charles:dailyuse-37b-ui-composer-barriers-blocked
charles:fix/image-viewer
charles:fix/619-picker-flicker
charles:fix/editor-forum-edit
charles:dailyuse-32-ui-status-picker
charles:dailyuse-34b-ui-add-guild-invite-modal
charles:dailyuse-28-app-dailyuse-runtime
charles:dailyuse-36-ui-voice-badge-refusals
charles:fix/594-menu-placement-textchat
charles:fix/589-stream-top-strip
charles:fix/594-menu-placement
charles:dailyuse-39-ui-report-modal
charles:dailyuse-37a-ui-attachment-audio-group-icon
charles:dailyuse-24-core-notification-icon-passive
charles:fix/587-quiet-voice
charles:dailyuse-30-ui-notification-settings-modal
charles:dailyuse-31-ui-user-dm-menus
charles:dailyuse-22b-core-composer-barriers-blocked
charles:dailyuse-22a-core-mark-unread-attachments
charles:dailyuse-29-ui-channel-guild-menus
charles:dailyuse-18-core-user-dm-menus
charles:dailyuse-33-ui-confirm-friends-page
charles:dailyuse-22c-core-trusted-domains-hide-muted
charles:dailyuse-20-core-keyboard-navigation
charles:dailyuse-15-core-friends-blocking
charles:dailyuse-23-core-profile-report-flow
charles:dailyuse-19-core-quick-switcher
charles:dailyuse-14-core-status-idle-presence
charles:dailyuse-17-core-channel-guild-menus
charles:ci/27-linux-cache
charles:dailyuse-37c-ui-trust-links-hide-muted
charles:dailyuse-13-core-honour-mutes
charles:dailyuse-34a-ui-invite-cards-accept
charles:dailyuse-16b-core-create-invites-guilds
charles:dailyuse-12-core-notification-settings-mutes
charles:dailyuse-16a-core-invite-links
charles:dailyuse-25-audio-attachments
charles:dailyuse-21-core-voice-refusals
charles:dailyuse-11b-core-state-skeleton
charles:dailyuse-11a-core-dailyuse-ports
charles:dailyuse-26b-devserver-dailyuse-b
charles:dailyuse-26a-devserver-dailyuse-a
charles:dailyuse-4-api-settings-acks-reports
charles:dailyuse-5-api-relationships-invites-guilds
charles:dailyuse-27-platform-notification-idle
charles:dailyuse-10-highlight-languages
charles:fix-search-focus-hover-emoji
charles:dailyuse-8-context-menu-widget
charles:dailyuse-9-image-load-retry
charles:dailyuse-7-chat-navigation-persistence
charles:dailyuse-2-proto-relationships-invites-reports
charles:dailyuse-1-proto-notification-presence
charles:dailyuse-6-gateway-op12-presence
charles:textchat-next
charles:fix-stream-black
charles:fix-reactions-hover
charles:textchat-68-app-video-playback-runtime
charles:textchat-35-acceptance
charles:textchat-34-parity-screenshots
charles:textchat-74-app-updater-runtime
charles:textchat-67-ui-video-player
charles:textchat-stage-46-47
charles:textchat-47-ui-media-channel
charles:textchat-75-promote-stable-workflow
charles:textchat-46-ui-forum-composer
charles:textchat-73-ui-update-button
charles:textchat-64-media-playback-crate
charles:textchat-stage-43-48
charles:textchat-72-platform-updater
charles:textchat-44-ui-thread-view
charles:textchat-70-release-update-feed
charles:textchat-48-ui-announcements
charles:textchat-45-ui-forum-view
charles:textchat-71-core-updater
charles:textchat-43-ui-sidebar-links-notes
charles:textchat-stage-40
charles:textchat-42-app-channel-types-runtime
charles:textchat-39-core-forums
charles:textchat-40-core-announcements-thread-admin
charles:textchat-60-ui-entrance-sounds
charles:textchat-63-ffmpeg-lgpl-packaging
charles:textchat-stage-batch6
charles:textchat-69-proto-update-manifest
charles:textchat-61-app-soundboard-entrance-runtime
charles:textchat-52-app-audio-devices-runtime
charles:textchat-38-core-channel-routing
charles:soundboard-effect-emoji
charles:status-api-rebind
charles:textchat-28-ui-gif-picker
charles:textchat-41-devserver-channel-types
charles:voice-panel-soundboard
charles:textchat-29-ui-message-actions
charles:textchat-37-api-channel-types
charles:textchat-66-core-video-players
charles:autoupdate-plan
charles:textchat-33-ui-message-list-settings
charles:textchat-32-ui-dms
charles:textchat-62-rename-flaxative
charles:fix-chatcolumn-tree
charles:textchat-65-audio-media-source
charles:textchat-31-ui-threads
charles:textchat-stage-27
charles:textchat-30-ui-search-pins-inbox
charles:textchat-53-ui-audio-sources-menu
charles:textchat-27-ui-expression-picker
charles:textchat-57-core-entrance-library
charles:textchat-36-proto-channel-types
charles:textchat-26-ui-upload-tray
charles:textchat-59-ui-soundboard-popout-parity
charles:textchat-58-audio-sound-edit
charles:textchat-55-proto-api-entrance-library
charles:textchat-25-ui-composer
charles:textchat-56-core-soundboard-parity
charles:rename-video-plan
charles:textchat-stage-25
charles:textchat-stage-23-24
charles:soundboard-plan
charles:textchat-24-ui-message-row
charles:textchat-22-ui-attachments-media
charles:textchat-23-ui-embeds
charles:textchat-14-core-expressions
charles:textchat-21-app-textchat-runtime
charles:textchat-16-core-navigation-read
charles:textchat-17-core-search
charles:textchat-19-core-dms-settings
charles:textchat-stage-13-15-18
charles:textchat-54-livekit-stereo-audio
charles:textchat-50-core-stream-audio-sources
charles:textchat-12-core-uploads
charles:textchat-15-core-message-actions
charles:textchat-11-core-composer
charles:textchat-18-core-threads
charles:textchat-13-core-reactions
charles:textchat-51-audio-pipewire-devices
charles:textchat-49-capture-audio-targets
charles:audio-plan
charles:textchat-plan2
charles:textchat-20-devserver-textchat
charles:textchat-6-api-chat-endpoints
charles:textchat-10-core-chat-ports
charles:textchat-9-ui-code-highlighting
charles:textchat-3-proto-expressions-threads-prefs
charles:textchat-5-api-messages-uploads
charles:textchat-2-proto-chat-requests-events
charles:textchat-8-ui-inline-flow-emoji
charles:screenshare
charles:textchat-7-platform-files-clipboard
charles:textchat-1-proto-message-content
charles:textchat-4-emoji-crate
charles:screenshare-31-acceptance
charles:screenshare-29-parity-screenshots
charles:textchat-plan
charles:screenshare-24-ui-focus-layout
charles:screenshare-27-ui-stream-menu
charles:screenshare-30-packaging-ci
charles:screenshare-21-app-screenshare-runtime
charles:screenshare-22-ui-sidebar-live
charles:screenshare-23-ui-stream-tile
charles:screenshare-26-ui-source-picker
charles:screenshare-28-ui-screenshare-settings
charles:screenshare-13-core-previews
charles:screenshare-12-core-codec-negotiation
charles:screenshare-25-ui-share-controls
charles:screenshare-18-hw-encode-linux
charles:screenshare-19-spike-hw-encode-windows
charles:screenshare-11-core-stream-params
charles:screenshare-20-devserver-screenshare
charles:screenshare-10-core-go-live
charles:screenshare-17-voice-livekit-video
charles:screenshare-14-audio-capture-linux
charles:wip/screenshare-17
charles:screenshare-integration
charles:feat-user-profile
charles:fix-voice-join
charles:feat-entity-images
charles:screenshare-16-voice-video-session
charles:screenshare-9-core-watching
charles:screenshare-5-capture-windows
charles:fix-wire-livekit
charles:screenshare-4-capture-linux
charles:screenshare-15-audio-capture-windows
charles:fix-member-list
charles:screenshare-8-core-stream-ports
charles:fix-live-protocol
charles:screenshare-6-ui-video-widget
charles:screenshare-3-capture-crate
charles:fix-altcha-captcha
charles:screenshare-1-video-frames
charles:screenshare-7-proto-api-streams
charles:screenshare-2-spike-livekit-video
charles:voice
charles:voice-26-acceptance
charles:voice-23-voice-e2ee
charles:voice-25-packaging-ci
charles:voice-13-voice-livekit-interop
charles:voice-24-parity-screenshots
charles:voice-19-ui-soundboard
charles:voice-20-ui-participant-menu
charles:voice-22-ui-dm-calls
charles:voice-21-ui-voice-settings
charles:voice-18-ui-call-view
charles:voice-14-app-voice-runtime
charles:voice-10-core-soundboard
charles:voice-17-ui-voice-panel
charles:voice-9-core-voice-settings-advert
charles:voice-5-audio-clips-chimes
charles:voice-15-devserver-voice
charles:voice-8-core-live-room-volumes
charles:voice-11-core-calls-status
charles:voice-12-voice-session
charles:voice-4-audio-devices-mixer
charles:voice-16-ui-voice-sidebar
charles:voice-7-core-voice-connection
charles:foundation
charles:voice-2-proto-api-soundboard-rtc
charles:voice-6-core-voice-states
charles:voice-3-audio-dsp
charles:voice-1-proto-voice-gateway
charles:task/26-acceptance
charles:task/25-packaging-release
charles:task/24-screenshot-tests
charles:task/20-ui-message-list-composer
charles:task/22-ui-settings-parity
charles:task/18-ui-main-layout
charles:task/23-notifications
charles:task/17-ui-shell-login
charles:task/14-app-runtime
charles:ci/51-windows-disk-space
charles:task/12-platform-persistence
charles:task/13-devserver
charles:task/21-ui-avatar-image-cache
charles:task/10-core-messages-readstate
charles:task/9-core-guilds-channels
charles:task/7-gateway-machine-transport
charles:task/11-core-presence-ui
charles:task/19-ui-rich-text
charles:task/8-core-store-session
charles:task/16-ui-theme-fonts-icons
charles:task/15-status-api
charles:task/4-proto-gateway-events
charles:task/5-openapi-vendor-contract
charles:task/6-api-client
charles:task/3-proto-rest-models
charles:task/2-proto-primitives
charles:task/1-workspace-scaffold
No reviewers
Labels
Clear labels
bug
A bug found while working on something else; steps to reproduce and the commit seen on
ci
Build, CI runners and workflows
follow-up
Deferred reviewer finding, linked from the PR it came from
good-first-task
Small, well-scoped, no cross-crate knowledge needed
parity
Settings/behaviour parity with the Electron client
question
Design question that needs Charles; blocks its task until answered
ready-to-merge
Reviewer approved, foundation merged in, CI green on both runners; waiting for the orchestrator to merge
task
A plan task (one per Foundation plan task)
No labels
bug
ci
follow-up
good-first-task
parity
question
ready-to-merge
task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
charles/fluxer-native!1002
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix-931-silent-voices"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Refs #931. Status: no reproduction. This PR adds a mitigation and sharper diagnostics. The cause is still unknown, so it uses
Refs, notCloses. Change it toCloses #931if you want merging to close the issue.What was tried (local
livekit-server1.12.0, the instance's version, no audio device opened)Kept as regression tests in
crates/fluxer-voice/tests/livekit_interop.rs(env-gated like the others;cargo xtask ciruns them withFLUXER_LIVEKIT_TESTS=1):a_voice_is_heard_after_being_alone_and_after_rejoining: the reported sequence. Alice joins alone and keeps feeding her muted microphone, as the app does. Bob joins later, unmutes and plays a tone, and Alice subscribes and hears it. Then Alice leaves and joins again on the same session (same process, same kept WebRTC runtime) and still hears it. It uses 3 s alone by default. 960 s alone (longer than the report's 15 min) also passed, run by hand withFLUXER_931_ALONE_SECS=960.voices_are_heard_across_many_calls_of_one_session: one session, a voice joins and leaves 6 times, with unsubscribe/resubscribe and rejoins. 20 rounds were also run by hand and passed.a_subscribed_track_delivers_while_its_sender_sends_nothing: a subscribed, unmuted track whose sender never sends anything still delivers about 100 frames/s, because libwebrtc's audio device pull runs NetEq whether packets arrive or not. So the #932 watcher's premise holds. The report's "no frames at all" means one of two things. Either the SDK never handed the track over (a subscription that did not complete), or the process-wide WebRTC audio device (webrtc-sysSyntheticAudioDevice, the 10 ms task that callsNeedMorePlayData) stopped pulling.Read and ruled out or judged unlikely:
third_party/webrtc-sys/src/synthetic_audio_device.cpp: playout is a flag (playing_) checked by a 10 ms repeating task that lives fromInit()toTerminate(). Its onlyTerminate()caller is the factory's destruction, and the client keeps the factory for the whole run. libwebrtc'sAudioState::AddReceivingStreamcallsStartPlayoutwheneverPlaying()is false, so a playout left stopped is restarted by the next receive stream, and a rejoin would fix it. The report says a rejoin does not, so a stopped flag does not fit. A dead or hung pump task, or a broken mixer, would fit, but I could not cause either from outside libwebrtc.Pump,read_track, the event channel), and the session's loop demonstrably ran (the watcher logged). The kept runtime is the only per-process state influxer-voice.LiveKitRoomdoes not leak the runtime across joins (checked: closing a room releases its hold).user-leavechime at 05:04:18Z, before the restart, says the remote participant left. The join after the restart had a participant again (participants=1). So "restart fixes it" also coincides with the remote reconnecting. A broken publication on the remote's side cannot be excluded. The new log lines below tell these cases apart.Changes
crates/fluxer-voice/src/rtc.rs):voice: subscribed to an audio trackis now logged atinfo(it wasdebug, so the reported log could not say whether the SDK ever handed the track over).TrackSubscriptionFailed, which was silently ignored, is logged atwarn.warn:WebRTC hands a subscribed audio track no audio at all: its audio device does not pull audio …. This is the line that names the process-wide audio device.voice: WebRTC runtime made again. The SDK hands out its current runtime for as long as anything holds it, so the vendoredlivekitgets a small patch,LkRuntime::forget_instance(), recorded inthird_party/livekit/PATCHES.md. Test:a_stalled_audio_device_makes_the_runtime_again_at_the_next_join(the stall itself cannot be caused from outside libwebrtc, so the test declares it). It checks that the join after the stall uses a new factory although the old one is still held, and that a voice is heard with it. The whole interop suite also passed afterwards in the same process (22/22), running on the new runtime.crates/fluxer-voice/README.mddescribes both.What the next log should say
subscribed to an audio trackfor the silent track (orcould not subscribe): the subscription never completed. That is the SDK or server side, or the remote's publication. A rebuilt runtime does not help, and the mitigation does not fire.hands a subscribed audio track no audio at all: the process's WebRTC audio device stopped. Leave and rejoin should now bring voices back (look forWebRTC runtime made again). If it does, that confirms the cause's location, and the next step is instrumentingSyntheticAudioDeviceitself.Not verified on Windows. The changed code is shared and has no platform branches. Only Linux ran here.
Verification
FLUXER_LIVEKIT_TESTS=1 cargo test -p fluxer-voice --test livekit_interop -- --ignored --test-threads=1: 22 passed.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci:==> xtask ci: all steps passed(livekit-interop skipped there; run separately above)🤖 Generated with Claude Code
Merged by hand into textchat (
8f270714) with the other open PRs; the combined tree passed the full gate locally, LiveKit interop included.Pull request closed