Voice 4: fluxer-audio devices, capture and playback mixer (#65) #106

Manually merged
charles merged 0 commits from voice-4-audio-devices-mixer into voice 2026-11-11 19:59:23 +00:00
Owner

Stacked on: voice-3-audio-dsp

What this adds (crates/fluxer-audio)

  • host: AudioHost (list devices per direction, open input/output streams with callbacks, one listener for device and default-device changes), DeviceInfo { id, name, is_default }, HostError, StreamError.
  • host_cpal: CpalHost on cpal 0.18 (ALSA / WASAPI / CoreAudio). Sample formats are converted to i16 in a buffer allocated when the stream is built. cpal has no device notifications, so a small thread compares the device lists every 3 s.
  • host_null: NullHost::new() -> (NullHost, NullControl) with advance, feed_input, take_output, unplug_input, plus unplug_output, plug, set_default, fail_stream, set_tone, is_open, open_device, and NullHost::with_config for other rates and channel counts.
  • mixer: Mixer with set_gain(SourceId, f32), push(SourceId, Frame), remove(&SourceId), 120 ms cap per source, soft clip on the sum, per-source levels. SourceId::{Remote(String), Clip(u64), MicTest}.
  • capture: Capture, the device stream → 48 kHz mono/stereo 10 ms frames step in front of the chain.
  • engine: AudioEngine::start(host, events) with set_input_device, set_output_device, start_capture(ChainConfig) -> Receiver<Frame>, stop_capture, set_chain_config, push_remote, remove_remote, set_gain, devices(), start_mic_test / stop_mic_test; EngineEvent::{Devices, DeviceLost { input }, LocalLevel { rms, gate_open }, RemoteLevels, Error}, levels once per 50 ms of audio.
  • Workspace: cpal = "0.18", ringbuf = "0.6". No CI change (libasound2-dev is already installed on the Linux runner). cargo deny is green without new licences or ignores.

Design: the device callbacks only copy between the device buffer and a ringbuf ring. One worker thread (fluxer-audio) does the rest: channel layout, resampling, the chain, the mixer. It wakes every 5 ms while a stream is open and sleeps on its command queue otherwise. Devices are opened on demand (microphone while capturing or testing; loudspeaker while the mixer has a source, plus 2 s).

Review Focus 6 (device loss): a lost stream is reported as DeviceLost, the selection falls back to the default device, the stream is reopened in the same worker pass, and the capture receiver and the mixer's sources are kept, so the call goes on. Tests: unplugging_input_falls_back_to_default_and_reports_device_lost, unplugging_output_falls_back_to_default_and_keeps_playing, losing_every_input_device_reports_an_error_and_recovers_when_one_returns, a_new_default_device_takes_over_streams_that_follow_the_default, a_device_that_dies_on_every_open_is_not_reopened_forever.

How it was tested

  • tests/mixer.rs (5) and tests/engine.rs (14), all on NullHost, including the seven tests the plan names. callbacks_do_not_allocate installs a counting global allocator and asserts zero allocations on the thread running NullControl::advance over 2 s of capture + mic test + two remote participants on 44.1 kHz devices (resampling both ways), after warm-up.
  • The engine tests were run 40 times in parallel on a loaded machine without a failure.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally; cargo check --all-targets --target x86_64-pc-windows-msvc -p fluxer-audio type-checks.
  • CpalHost was exercised once on a real machine (FLUXER_AUDIO_DEVICE_TESTS=1 cargo test -p fluxer-audio --lib): default input and output opened at 48 kHz stereo and delivered audio. Without that variable the test returns at once, so CI needs no sound card.

Deviations from the plan and things to look at

  • Channels are std::sync::mpsc, not tokio: the crate has no async runtime and no task names one. Voice 14 bridges them.
  • AudioEngine::sync() is an addition: it blocks until the worker has caught up. Tests on the virtual clock need it to be deterministic; the application does not.
  • set_gain takes a SourceId (the mixer's key) rather than an identity string, so Voice 5's clips can use the same call.
  • Gains survive remove_remote. A participant whose track comes back after a deafen or reconnect keeps their volume without the caller re-sending it.
  • Mixer prefill. A source starts (and resumes after running dry) once it has one device period plus 10 ms queued, at least 20 ms. Without it a device that asks for 20 ms at a time would play gaps between 10 ms frames. The 120 ms cap is as planned: it drops the oldest audio.
  • devices() returns Result<Devices, HostError> with Devices { inputs, outputs }.
  • Branch name. The remote cannot hold voice/4-audio-devices-mixer next to the voice branch (ref name conflict), so it is pushed as voice-4-audio-devices-mixer, like Voice 1 to 3.
  • Linux device names. With cpal's default (ALSA) backend the lists show ALSA PCM names (default, pipewire, sysdefault:CARD=…), not PipeWire node names, and a default-device change is invisible because the id stays alsa:default (PipeWire reroutes it itself). cpal 0.18 has an optional native PipeWire backend; using it would be a new decision for the settings UI task (Voice 21), not made here.
  • Not verified here: the Windows runner (WASAPI code paths are compiled but were not run), and behaviour with real hot-plugging; only the NullHost simulates it.

Closes #65

🤖 Generated with Claude Code

Stacked on: voice-3-audio-dsp ## What this adds (`crates/fluxer-audio`) - `host`: `AudioHost` (list devices per direction, open input/output streams with callbacks, one listener for device and default-device changes), `DeviceInfo { id, name, is_default }`, `HostError`, `StreamError`. - `host_cpal`: `CpalHost` on `cpal` 0.18 (ALSA / WASAPI / CoreAudio). Sample formats are converted to `i16` in a buffer allocated when the stream is built. `cpal` has no device notifications, so a small thread compares the device lists every 3 s. - `host_null`: `NullHost::new() -> (NullHost, NullControl)` with `advance`, `feed_input`, `take_output`, `unplug_input`, plus `unplug_output`, `plug`, `set_default`, `fail_stream`, `set_tone`, `is_open`, `open_device`, and `NullHost::with_config` for other rates and channel counts. - `mixer`: `Mixer` with `set_gain(SourceId, f32)`, `push(SourceId, Frame)`, `remove(&SourceId)`, 120 ms cap per source, soft clip on the sum, per-source levels. `SourceId::{Remote(String), Clip(u64), MicTest}`. - `capture`: `Capture`, the device stream → 48 kHz mono/stereo 10 ms frames step in front of the chain. - `engine`: `AudioEngine::start(host, events)` with `set_input_device`, `set_output_device`, `start_capture(ChainConfig) -> Receiver<Frame>`, `stop_capture`, `set_chain_config`, `push_remote`, `remove_remote`, `set_gain`, `devices()`, `start_mic_test` / `stop_mic_test`; `EngineEvent::{Devices, DeviceLost { input }, LocalLevel { rms, gate_open }, RemoteLevels, Error}`, levels once per 50 ms of audio. - Workspace: `cpal = "0.18"`, `ringbuf = "0.6"`. No CI change (`libasound2-dev` is already installed on the Linux runner). `cargo deny` is green without new licences or ignores. Design: the device callbacks only copy between the device buffer and a `ringbuf` ring. One worker thread (`fluxer-audio`) does the rest: channel layout, resampling, the chain, the mixer. It wakes every 5 ms while a stream is open and sleeps on its command queue otherwise. Devices are opened on demand (microphone while capturing or testing; loudspeaker while the mixer has a source, plus 2 s). Review Focus 6 (device loss): a lost stream is reported as `DeviceLost`, the selection falls back to the default device, the stream is reopened in the same worker pass, and the capture receiver and the mixer's sources are kept, so the call goes on. Tests: `unplugging_input_falls_back_to_default_and_reports_device_lost`, `unplugging_output_falls_back_to_default_and_keeps_playing`, `losing_every_input_device_reports_an_error_and_recovers_when_one_returns`, `a_new_default_device_takes_over_streams_that_follow_the_default`, `a_device_that_dies_on_every_open_is_not_reopened_forever`. ## How it was tested - `tests/mixer.rs` (5) and `tests/engine.rs` (14), all on `NullHost`, including the seven tests the plan names. `callbacks_do_not_allocate` installs a counting global allocator and asserts zero allocations on the thread running `NullControl::advance` over 2 s of capture + mic test + two remote participants on 44.1 kHz devices (resampling both ways), after warm-up. - The engine tests were run 40 times in parallel on a loaded machine without a failure. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally; `cargo check --all-targets --target x86_64-pc-windows-msvc -p fluxer-audio` type-checks. - `CpalHost` was exercised once on a real machine (`FLUXER_AUDIO_DEVICE_TESTS=1 cargo test -p fluxer-audio --lib`): default input and output opened at 48 kHz stereo and delivered audio. Without that variable the test returns at once, so CI needs no sound card. ## Deviations from the plan and things to look at - **Channels are `std::sync::mpsc`**, not tokio: the crate has no async runtime and no task names one. Voice 14 bridges them. - **`AudioEngine::sync()`** is an addition: it blocks until the worker has caught up. Tests on the virtual clock need it to be deterministic; the application does not. - **`set_gain` takes a `SourceId`** (the mixer's key) rather than an identity string, so Voice 5's clips can use the same call. - **Gains survive `remove_remote`.** A participant whose track comes back after a deafen or reconnect keeps their volume without the caller re-sending it. - **Mixer prefill.** A source starts (and resumes after running dry) once it has one device period plus 10 ms queued, at least 20 ms. Without it a device that asks for 20 ms at a time would play gaps between 10 ms frames. The 120 ms cap is as planned: it drops the oldest audio. - **`devices()` returns `Result<Devices, HostError>`** with `Devices { inputs, outputs }`. - **Branch name.** The remote cannot hold `voice/4-audio-devices-mixer` next to the `voice` branch (ref name conflict), so it is pushed as `voice-4-audio-devices-mixer`, like Voice 1 to 3. - **Linux device names.** With `cpal`'s default (ALSA) backend the lists show ALSA PCM names (`default`, `pipewire`, `sysdefault:CARD=…`), not PipeWire node names, and a default-device change is invisible because the id stays `alsa:default` (PipeWire reroutes it itself). `cpal` 0.18 has an optional native PipeWire backend; using it would be a new decision for the settings UI task (Voice 21), not made here. - Not verified here: the Windows runner (WASAPI code paths are compiled but were not run), and behaviour with real hot-plugging; only the `NullHost` simulates it. Closes #65 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(audio): frames and microphone DSP chain
All checks were successful
ci / Linux (x86_64) (pull_request) Successful in 7m37s
ci / Windows (x64) (pull_request) Successful in 8m11s
ci / Linux (x86_64) (push) Successful in 7m13s
ci / Windows (x64) (push) Successful in 8m5s
fc38bd2e78
New crate fluxer-audio: Frame (48 kHz, 10 ms), streaming Resampler (rubato),
per-user volume curve, voice-activity gate (port of the fork's worklet),
speaking level detector, soft clip, RNNoise denoiser (nnnoiseless) and the
CaptureChain that runs them in the original's order.

Refs #64

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
feat(audio): device hosts, capture and playback mixer
Some checks failed
ci / Linux (x86_64) (push) Successful in 7m31s
ci / Windows (x64) (push) Successful in 8m39s
ci / Linux (x86_64) (pull_request) Successful in 7m25s
ci / Windows (x64) (pull_request) Has been cancelled
0743d38035
Refs #65

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Author
Owner

Independent review — Voice 4 (audio devices, capture, mixer) @ 0743d38

Scope: git diff fc38bd2..0743d38 (13 files, crate fluxer-audio + workspace Cargo.toml/Cargo.lock).

1. Named tests — all seven exist and assert what the plan says: mixer_sums_sources_with_gain_and_soft_clips, mixer_drops_backlog_beyond_120ms (200 ms pushed, exactly 120 ms kept, frames 9..=20 played in order), capture_delivers_10ms_frames_through_chain, unplugging_input_falls_back_to_default_and_reports_device_lost, remote_levels_reported_per_identity, mic_test_loops_input_to_output, callbacks_do_not_allocate (counting global allocator, thread-local to the thread running NullControl::advance, 2 s after a 0.5 s warm-up, with a self-check that the counter counts). All on NullHost.

2. Interfaces — AudioHost, CpalHost, NullHost::new() -> (NullHost, NullControl), NullControl::{advance, feed_input, take_output, unplug_input}, DeviceInfo { id, name, is_default }, Mixer, AudioEngine::start(Arc<dyn AudioHost>, mpsc::Sender<EngineEvent>) and every listed method, and all five EngineEvent variants are present. Deviations from the letter of the plan are listed under findings (minor).

3. Constraints — fluxer-audio depends on neither fluxer-core nor LiveKit; only cpal and ringbuf were added (both named by the task); no logging in non-test code; no secrets. Real-time rule: the engine's callbacks only touch a ringbuf half and atomics; the cpal wrappers convert through a scratch buffer allocated at stream build. I checked the implementer's claim that the 3 s device-list poll opens no device: true for cpal 0.18.2 ALSA (supports_input/output are answered from the hint direction, not by opening the PCM).

4. Review Focus 6 (device loss keeps the call) — holds. Loss sets a flag in the error callback; devices_changed runs check_streams(false) before listing, so the order is always DeviceLost then Devices, whichever of the flag or HostChanged the worker sees first (the other order only produces a duplicate Devices). The stream is reopened on the default in the same worker pass; the capture Receiver and the mixer sources survive. Covered for input, output, loss of every device + re-plug, a device that dies on every open, and a default change.

5. CI — cargo xtask ci in the review worktree (own target dir): all steps passed (screenshot step skipped; this PR has no UI). Remote runners not checked by me; Windows/WASAPI paths have never executed.

Findings

  • minor — crates/fluxer-audio/src/engine.rs: signatures differ from the plan's Interfaces block; Voice 5 and Voice 14 must build against the code, not the plan: channels are std::sync::mpsc (Voice 14 needs a bridging thread to its async runtime), AudioEngine::set_gain(SourceId, f32) rather than an identity string, devices() -> Result<Devices { inputs, outputs }, HostError>, and Mixer::remove(&SourceId) (plan: by value).
  • minor — engine.rs pump_playback/reconcile_output: the mixer only runs while an output stream is open. On a machine with no usable loudspeaker (output_blocked) remote audio is never mixed, so RemoteLevels is never emitted and level-based speaking indicators stay dark.
  • minor — engine.rs start_capture: the frame channel is unbounded; a consumer that stalls (LiveKit publish blocked) grows memory at 100 frames/s with no drop policy, unlike the 120 ms cap on the playback side.
  • minor — engine.rs devices_changed: a stream opened on an explicitly chosen device is only replaced when the backend reports a stream error; if a backend drops the device from the list without erroring the stream, the engine keeps a dead stream. Also, after any loss the selection is reset to default and not restored when the device returns (documented; the core has to re-send the choice).
  • minor — host_cpal.rs: on Linux the lists are ALSA PCM names and a default-device change is invisible (alsa:default never changes id); WASAPI code is type-checked only. Per CLAUDE.md these deferred items need a follow-up issue linked from this PR before merge.

Verdict: APPROVE — no blocking findings, local ci green.

## Independent review — Voice 4 (audio devices, capture, mixer) @ 0743d38 Scope: `git diff fc38bd2..0743d38` (13 files, crate `fluxer-audio` + workspace `Cargo.toml`/`Cargo.lock`). **1. Named tests** — all seven exist and assert what the plan says: `mixer_sums_sources_with_gain_and_soft_clips`, `mixer_drops_backlog_beyond_120ms` (200 ms pushed, exactly 120 ms kept, frames 9..=20 played in order), `capture_delivers_10ms_frames_through_chain`, `unplugging_input_falls_back_to_default_and_reports_device_lost`, `remote_levels_reported_per_identity`, `mic_test_loops_input_to_output`, `callbacks_do_not_allocate` (counting global allocator, thread-local to the thread running `NullControl::advance`, 2 s after a 0.5 s warm-up, with a self-check that the counter counts). All on `NullHost`. **2. Interfaces** — `AudioHost`, `CpalHost`, `NullHost::new() -> (NullHost, NullControl)`, `NullControl::{advance, feed_input, take_output, unplug_input}`, `DeviceInfo { id, name, is_default }`, `Mixer`, `AudioEngine::start(Arc<dyn AudioHost>, mpsc::Sender<EngineEvent>)` and every listed method, and all five `EngineEvent` variants are present. Deviations from the letter of the plan are listed under findings (minor). **3. Constraints** — `fluxer-audio` depends on neither `fluxer-core` nor LiveKit; only `cpal` and `ringbuf` were added (both named by the task); no logging in non-test code; no secrets. Real-time rule: the engine's callbacks only touch a `ringbuf` half and atomics; the `cpal` wrappers convert through a scratch buffer allocated at stream build. I checked the implementer's claim that the 3 s device-list poll opens no device: true for cpal 0.18.2 ALSA (`supports_input/output` are answered from the hint direction, not by opening the PCM). **4. Review Focus 6 (device loss keeps the call)** — holds. Loss sets a flag in the error callback; `devices_changed` runs `check_streams(false)` before listing, so the order is always `DeviceLost` then `Devices`, whichever of the flag or `HostChanged` the worker sees first (the other order only produces a duplicate `Devices`). The stream is reopened on the default in the same worker pass; the capture `Receiver` and the mixer sources survive. Covered for input, output, loss of every device + re-plug, a device that dies on every open, and a default change. **5. CI** — `cargo xtask ci` in the review worktree (own target dir): all steps passed (screenshot step skipped; this PR has no UI). Remote runners not checked by me; Windows/WASAPI paths have never executed. ### Findings - **minor** — `crates/fluxer-audio/src/engine.rs`: signatures differ from the plan's Interfaces block; Voice 5 and Voice 14 must build against the code, not the plan: channels are `std::sync::mpsc` (Voice 14 needs a bridging thread to its async runtime), `AudioEngine::set_gain(SourceId, f32)` rather than an identity string, `devices() -> Result<Devices { inputs, outputs }, HostError>`, and `Mixer::remove(&SourceId)` (plan: by value). - **minor** — `engine.rs` `pump_playback`/`reconcile_output`: the mixer only runs while an output stream is open. On a machine with no usable loudspeaker (`output_blocked`) remote audio is never mixed, so `RemoteLevels` is never emitted and level-based speaking indicators stay dark. - **minor** — `engine.rs` `start_capture`: the frame channel is unbounded; a consumer that stalls (LiveKit publish blocked) grows memory at 100 frames/s with no drop policy, unlike the 120 ms cap on the playback side. - **minor** — `engine.rs` `devices_changed`: a stream opened on an explicitly chosen device is only replaced when the backend reports a stream error; if a backend drops the device from the list without erroring the stream, the engine keeps a dead stream. Also, after any loss the selection is reset to default and not restored when the device returns (documented; the core has to re-send the choice). - **minor** — `host_cpal.rs`: on Linux the lists are ALSA PCM names and a default-device change is invisible (`alsa:default` never changes id); WASAPI code is type-checked only. Per CLAUDE.md these deferred items need a `follow-up` issue linked from this PR before merge. **Verdict: APPROVE** — no blocking findings, local ci green.
Merge remote-tracking branch 'origin/voice' into voice/4-audio-devices-mixer
All checks were successful
ci / Linux (x86_64) (push) Successful in 8m2s
ci / Linux (x86_64) (pull_request) Successful in 7m39s
ci / Windows (x64) (push) Successful in 8m29s
ci / Windows (x64) (pull_request) Successful in 8m45s
8843c4cd42
charles manually merged commit b6c6251df2 into voice 2026-11-11 19:59:23 +00:00
Sign in to join this conversation.
No description provided.