Fix slow voice joins and the missing connect chime; add a voice log #215

Manually merged
charles merged 0 commits from fix-voice-join into main 2026-11-12 05:16:39 +00:00
Owner

Stacked on: #203 (fix-wire-livekit). Merge #203 first; this branch contains it.

Two voice bugs reported from the first real joins (slow connect, no connect sound), plus a voice log, which did not exist before. Refs #126.

Slow connect

The cause is proven by the new end-to-end test on loopback. The LiveKit SDK builds its WebRTC runtime inside the first Room::connect: the peer connection factory, APM/AEC3, the codec factories and their VAAPI probes. It then drops that runtime with the last room. So every join rebuilt it before the first signalling byte: 0.55 to 1.2 s here, and it blocked a tokio worker while doing so (livekit-0.10.1/src/rtc_engine/lk_runtime.rs:76-95 keeps it behind a Weak).

  • LiveKitRoom now keeps the runtime for its whole life.
  • LiveKitRoom::warmed_up(), which the app uses, builds it on its own thread at start.
  • The chimes are also decoded in the background at start. The first decode used to delay the "connected" snapshot by about 60 ms.

Join timeline on loopback (since_join_ms, from the log):

step before after
join requested → op 4 → VOICE_SERVER_UPDATE → room connecting 0–3 0–3
WebRTC runtime built (LkRuntime::new) 3 → ~790 done at app start
signalling + ICE/DTLS ("connection setup took") ~190 ~120–370
room joined → mic published → UI Connected 980 / 982 / 982 211 / 224 / 224
click → "Voice connected" (test measure) 0.71–1.21 s 0.12–0.41 s

No connect sound

The cause is proven by code: fluxer-core never emitted a chime for this client's own join. live.rs chime_for explicitly skips the client's own connection, and connection.rs on_room(Connected) played nothing. The original plays user-join on LiveKit RoomEvent.Connected (VoiceRoomEventBinder.ts:329, VoiceSelfJoinChime.ts).

What the client does now:

  • A fresh join plays user-join when it becomes Connected, also while deafened.
  • A move plays user-move at the click (not while deafened), with no join sound after it. The user-move.mp3 asset is copied from the fork.
  • A replaced room (token refresh) is silent.
  • voice-disconnect also plays when a join is abandoned while still connecting, as the original's runDisconnectFromVoiceChannel does.

Checked and fine: the assets are include_bytes!, so they are present in release builds. No setting gates them: the sound settings are not built yet (sub-project 6), and the original leaves the voice sounds on by default. Output is opened on demand for CpalHost and NullHost alike.

Voice log

  • info: one voice: line per step, with since_join_ms. Steps: join requested, op 4 sent, server update (host + connection id), room connect requested / connecting / joined (with duration), mic published, UI state Connected, capture started/stopped, audio input/output opened (device name, rate, channels), chime played (name), disconnected/failed (with why).
  • debug: room connection state, subscriptions, participants, tracks.
  • LiveKit SDK records (#126): the SDK's log records are bridged to tracing as livekit_sdk (at most debug). URL queries, JWTs and token/Bearer values are struck out first (logging::redact, unit-tested). The e2e test logs at debug and asserts the token and its signature are absent from the log file.

End-to-end test

apps/fluxer/tests/voice_livekit.rs runs in cargo xtask ci under FLUXER_LIVEKIT_TESTS=1, in the livekit-interop step. Setup:

  • the real runtime with LiveKitRoom::warmed_up() and NullHost
  • the fixture gateway, using the new VoiceScript::set_media_server. It sends VOICE_SERVER_UPDATE before VOICE_STATE_UPDATE, as the real server does.
  • a /livekit prefix proxy in front of the local livekit-server

It joins and leaves twice. Each time it checks: Connected in under 1 s; the user-join chime found in the virtual output (by correlation) within 1.5 s; the voice-disconnect chime on leave. It also checks that the runtime was built only once (LkRuntime::new() appears once in the log).

On the old code it failed on the chime. Without keeping the runtime it measures 0.71–1.21 s per join.

CI

FLUXER_SCREENSHOT_TESTS=1 FLUXER_LIVEKIT_TESTS=1 cargo xtask ci: all steps pass locally.

For the real server

What the loopback test cannot show is WAN signalling/ICE time and the gateway's op 4 → VOICE_SERVER_UPDATE time. Join once with the default info log: the voice: lines give each step's since_join_ms, so any remaining delay shows which step it is in.

🤖 Generated with Claude Code

Stacked on: #203 (`fix-wire-livekit`). Merge #203 first; this branch contains it. Two voice bugs reported from the first real joins (slow connect, no connect sound), plus a voice log, which did not exist before. Refs #126. ## Slow connect The cause is **proven by the new end-to-end test on loopback**. The LiveKit SDK builds its WebRTC runtime inside the first `Room::connect`: the peer connection factory, APM/AEC3, the codec factories and their VAAPI probes. It then drops that runtime with the last room. So **every** join rebuilt it before the first signalling byte: 0.55 to 1.2 s here, and it blocked a tokio worker while doing so (`livekit-0.10.1/src/rtc_engine/lk_runtime.rs:76-95` keeps it behind a `Weak`). - `LiveKitRoom` now keeps the runtime for its whole life. - `LiveKitRoom::warmed_up()`, which the app uses, builds it on its own thread at start. - The chimes are also decoded in the background at start. The first decode used to delay the "connected" snapshot by about 60 ms. Join timeline on loopback (`since_join_ms`, from the log): | step | before | after | |---|---|---| | join requested → op 4 → VOICE_SERVER_UPDATE → room connecting | 0–3 | 0–3 | | WebRTC runtime built (`LkRuntime::new`) | 3 → ~790 | done at app start | | signalling + ICE/DTLS ("connection setup took") | ~190 | ~120–370 | | room joined → mic published → UI Connected | 980 / 982 / 982 | 211 / 224 / 224 | | click → "Voice connected" (test measure) | 0.71–1.21 s | 0.12–0.41 s | ## No connect sound The cause is **proven by code**: `fluxer-core` never emitted a chime for this client's own join. `live.rs` `chime_for` explicitly skips the client's own connection, and `connection.rs` `on_room(Connected)` played nothing. The original plays `user-join` on LiveKit `RoomEvent.Connected` (`VoiceRoomEventBinder.ts:329`, `VoiceSelfJoinChime.ts`). What the client does now: - A fresh join plays `user-join` when it becomes Connected, also while deafened. - A move plays `user-move` at the click (not while deafened), with no join sound after it. The `user-move.mp3` asset is copied from the fork. - A replaced room (token refresh) is silent. - `voice-disconnect` also plays when a join is abandoned while still connecting, as the original's `runDisconnectFromVoiceChannel` does. Checked and fine: the assets are `include_bytes!`, so they are present in release builds. No setting gates them: the sound settings are not built yet (sub-project 6), and the original leaves the voice sounds on by default. Output is opened on demand for `CpalHost` and `NullHost` alike. ## Voice log - **`info`:** one `voice:` line per step, with `since_join_ms`. Steps: join requested, op 4 sent, server update (host + connection id), room connect requested / connecting / joined (with duration), mic published, UI state Connected, capture started/stopped, audio input/output opened (device name, rate, channels), chime played (name), disconnected/failed (with why). - **`debug`:** room connection state, subscriptions, participants, tracks. - **LiveKit SDK records (#126):** the SDK's `log` records are bridged to `tracing` as `livekit_sdk` (at most `debug`). URL queries, JWTs and token/Bearer values are struck out first (`logging::redact`, unit-tested). The e2e test logs at `debug` and asserts the token and its signature are absent from the log file. ## End-to-end test `apps/fluxer/tests/voice_livekit.rs` runs in `cargo xtask ci` under `FLUXER_LIVEKIT_TESTS=1`, in the `livekit-interop` step. Setup: - the real runtime with `LiveKitRoom::warmed_up()` and `NullHost` - the fixture gateway, using the new `VoiceScript::set_media_server`. It sends VOICE_SERVER_UPDATE before VOICE_STATE_UPDATE, as the real server does. - a `/livekit` prefix proxy in front of the local `livekit-server` It joins and leaves twice. Each time it checks: Connected in under 1 s; the `user-join` chime found in the virtual output (by correlation) within 1.5 s; the `voice-disconnect` chime on leave. It also checks that the runtime was built only once (`LkRuntime::new()` appears once in the log). On the old code it failed on the chime. Without keeping the runtime it measures 0.71–1.21 s per join. ## CI `FLUXER_SCREENSHOT_TESTS=1 FLUXER_LIVEKIT_TESTS=1 cargo xtask ci`: all steps pass locally. ## For the real server What the loopback test cannot show is WAN signalling/ICE time and the gateway's op 4 → VOICE_SERVER_UPDATE time. Join once with the default `info` log: the `voice:` lines give each step's `since_join_ms`, so any remaining delay shows which step it is in. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Solve the fork's ALTCHA captcha in fluxer-api
All checks were successful
ci / Linux (x86_64) (push) Successful in 16m3s
ci / Windows (x64) (push) Successful in 16m12s
d281e073da
A 400 CAPTCHA_REQUIRED / INVALID_CAPTCHA carrying an ALTCHA v2 challenge
is now solved by the client (PBKDF2-HMAC-SHA-256 over nonce || counter_be32,
counters from 0 until the key starts with keyPrefix) and the same request is
sent again with X-Captcha-Token, at most twice per request, as the web
app's CaptchaInterceptor does for every request. The search runs in
spawn_blocking on up to 8 threads, bounded by 60 s and the challenge's
expiry, and stops when the request is dropped. The token is byte for byte
the web client's encoding.

ApiError::CaptchaRequired is left for captchas the client cannot solve
(unknown provider, malformed challenge, timeout, both solutions refused);
INVALID_CAPTCHA maps to it instead of a plain 400 that the core read as
wrong credentials. The login screen's message says so.

Test vector fixtures/captcha/altcha_login.json was produced by the fork's
own AltchaProvider; the API tests run against a mock that verifies tokens
the way the fork does. Adds ring (already in the graph via rustls) as a
direct dependency of fluxer-api.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`threads` was logged before the cap of 8 (the web app's MAX_SOLVER_WORKERS)
that `solve` applies, so a 32-core machine logged threads=32 while 8 ran.
Clamp before logging; behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read READY as a real Fluxer server sends it
All checks were successful
ci / Linux (x86_64) (push) Successful in 14m30s
ci / Windows (x64) (push) Successful in 15m57s
3bf4ee7931
The server deduplicates users in READY: guilds[].members[].user is
{"id"} only and the full users are in the top-level `users` list
(session_ready_collect.erl). fluxer-proto required a full user on every
member, so a real READY failed to parse as a whole and the client showed
no guilds, no DMs and "Unknown User".

- fluxer-proto: `UserRef` (full user or `{"id"}` reference) for
  GuildMember::user, Presence::user (the offline PRESENCE_UPDATE sent when
  a friend/group-DM subscription is dropped) and GUILD_MEMBER_REMOVE's
  user (always a reference). `ReadyPayload::resolve_users()` fills the
  references from `users` and `user`.
- Every list of ReadyPayload / GuildReadyData is read leniently: a
  malformed entry is dropped with a warning instead of losing the
  session; a malformed guild object leaves `properties: None`. Adds
  `tracing` to fluxer-proto for that warning.
- fluxer-core: members and presences that are references add nothing to
  the user cache; names come from READY.users, unknown users show as
  Unknown User.
- fixtures/gateway/ready.json now has the real structure (member user
  references, `users` with every member, relationships without `user`,
  empty guild presences, sessions/version/...), with the same fake
  identities. The devserver resolves it before indexing message authors.
- tests/replay.rs in fluxer-core: with FLUXER_REPLAY_DIR set, replays a
  --record recording through the parsers and the store and reports
  mismatches by key path and redacted error only; skipped otherwise.
- Regression tests: references resolved via `users`, an unknown member
  user id kept, malformed members/channels/users/guilds dropped without
  failing READY, references in GUILD_MEMBER_REMOVE and PRESENCE_UPDATE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Subscribe to guild member lists as the web app does
All checks were successful
ci / Linux (x86_64) (push) Successful in 37m5s
ci / Windows (x64) (push) Successful in 29m8s
0b6c986ffd
The real server sends only the user's own member in READY and no guild
presences; the other members arrive in GUILD_MEMBER_LIST_UPDATE after an
op 14 (LAZY_REQUEST) subscription, which the client never sent.

- fluxer-proto: LazyRequest/GuildSubscription/MemberListRange, the
  GUILD_MEMBER_LIST_UPDATE and GUILD_SYNC dispatches, with a fixture.
- fluxer-core: MemberLists sends {active, sync} on guild select and the
  shown channel's member list ranges, again after every reconnect; keeps
  the server list and feeds member_list, users and presences from it;
  falls back to READY's members when no answer comes. GUILD_CREATE and
  GUILD_SYNC add their partial members instead of replacing them.
- fluxer-ui: group headers show the server's count.
- fluxer-devserver: READY and GUILD_CREATE carry only the signed-in
  member; op 14 is answered with a server-shaped member list, and
  presence changes resend it. New end-to-end test.

Follow-ups: #195 (scroll ranges past 1,000 rows), #196 (PASSIVE_UPDATES).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(app): join voice rooms with the LiveKit room on real instances
All checks were successful
ci / Linux (x86_64) (push) Successful in 14m42s
ci / Windows (x64) (push) Successful in 15m45s
247fa06f45
The runtime of Voice task 14 used a placeholder room that refused every join
("this build cannot join voice rooms yet") until the LiveKit room of task 13
existed. Task 13 merged later and nothing replaced the placeholder, so voice
only worked with --devserver.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
fix(voice): fast joins with the connect chime, and a voice log
All checks were successful
ci / Linux (x86_64) (push) Successful in 40m57s
ci / Windows (x64) (push) Successful in 31m11s
6ef9c5fd09
On a real Fluxer + LiveKit server the native client was slow to reach
"Voice connected" and never played the connect sound; nothing about
voice reached the log. An end-to-end test of the application's real
voice path against a local livekit-server (with the fixture gateway and
virtual audio) now measures and guards all three.

Slow connect. The LiveKit SDK makes its WebRTC runtime (peer
connection factory, audio processing, codec factories with their VAAPI
probes) inside the first Room::connect and drops it with the last room,
so every join paid for it before the first signalling byte: 0.7-1.2 s
of the measured 0.74-1.21 s click-to-connected on loopback, blocking a
tokio worker meanwhile. LiveKitRoom now keeps the runtime for its life,
and LiveKitRoom::warmed_up() (what the application uses) makes it on a
thread of its own at start. Click to "Voice connected" on loopback is
now 120-410 ms. The bundled chimes are decoded in the background at
start too: the first one used to hold the "Voice connected" snapshot
back by ~60 ms on the runtime task.

Connect sound. fluxer-core never played a chime for this client's own
join (only for other people's). As the original does on LiveKit's
RoomEvent.Connected, the room of a fresh join reaching Connected now
plays user-join (also while deafened); a move plays user-move at the
click (not while deafened, bundled from the fork's assets) and no join
sound after it; a replaced room is silent. The disconnect sound now
also plays when a join is given up while still connecting.

Logging. At info, one `voice:` line per lifecycle step with
since_join_ms (fluxer_voice::trace): join requested, op 4 sent, server
update (host only), room connect requested/connecting/joined (with how
long), microphone published, UI state Connected, capture, audio
devices opened (EngineEvent::DeviceOpened with the device's name),
chime played, disconnected/failed with why. At debug: room connection
state, subscriptions, participants, tracks. The SDK's `log` records are
bridged to tracing (target livekit_sdk, at most debug) with URL
queries, JWTs and token values struck out (issue #126); the end-to-end
test logs at debug and checks the token is not in the log file.

The devserver can point VOICE_SERVER_UPDATE at a real media server
(VoiceScript::set_media_server), sent before the connection's
VOICE_STATE_UPDATE as the real server does. `cargo xtask ci` runs the
new test in its livekit-interop step.

Refs #126
Stacked on #203

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
charles manually merged commit b493918451 into main 2026-11-12 05:16:39 +00:00
Sign in to join this conversation.
No description provided.