Voice 3: fluxer-audio crate — frames and DSP (#64) #103

Merged
charles merged 1 commit from voice-3-audio-dsp into voice 2026-11-11 18:30:57 +00:00
Owner

What this builds

New crate crates/fluxer-audio (no devices, no network, no dependency on fluxer-core or LiveKit), with the constants of the Electron client:

  • Frame { samples: Vec<i16>, channels: u8 } (48 kHz, 10 ms), SAMPLE_RATE, FRAME_MS, FRAME_LEN.
  • Resampler::new(from_hz, to_hz, channels) -> Result<_, ResampleError>: streaming over rubato's FFT resampler; process(&[i16], &mut Vec<i16>), flush, reset. The filter delay is trimmed and flush ends a stream with exactly the input's duration.
  • gain::user_volume_gain(percent) (per-user curve) and gain::percent(p) (linear 0-200 % microphone volume).
  • Gate::new(GateConfig { mode: Auto | Manual { threshold_rms }, attack_ms: 5, release_ms: 20, hold_ms: 180, lookahead_ms: 10 }): a port of voiceGateProcessor.worklet.js (floor = 10th percentile of 2 s x 1.5, auto threshold = 3 x floor capped at 0.04, 300 ms seeding, release at 70 % of the threshold). gate::threshold_rms_for_slider.
  • LevelDetector::new(DetectorConfig) (30 ms attack, 220 ms release), level::remote_threshold_rms_for_slider, level::rms.
  • soft_clip(&mut [f32]) (knee 0.8, tanh shoulder).
  • Denoiser::{Rnnoise, None} (a plain selector, so ChainConfig is Copy + PartialEq) and denoise::DenoiseProcessor holding the RNNoise state per channel.
  • CaptureChain::new(ChainConfig { denoise, gate, input_volume_percent, stereo }), process(&mut Frame) -> ChainOutput { rms, gate_open }, plus set_config (keeps the gate's noise floor when only the volume or threshold changes) for Voice 4's set_chain_config.

Workspace Cargo.toml gains nnnoiseless (no default features: the defaults build its CLI) and rubato. deny.toml is unchanged: every new licence was already allowed.

How it was tested

crates/fluxer-audio/tests/dsp.rs, synthetic signals only: the nine tests the plan names plus gate_attack_ramp_takes_5ms, rms_of_reference_signals and chain_never_gates_stereo_capture. Written first against a signatures-only stub (12 failed), then implemented (12 pass). FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally.

Deviations from the plan

  • user_volume_gain(0.0) is 0.0, not 10^-1.2. The plan's reference point came from the facts report, which left out the original's special case (VoiceVolumeUtils.ts:24: if (clamped === 0) return 0). Review focus 5 asks to match the original, so 0 % is silence; the test checks the 10^-1.2 limit just above zero (0.001 %) and exact zero separately. Voice 8/10 should not expect a non-zero gain at 0 %.
  • rnnoise_reduces_white_noise_energy asserts at least 3 dB, measured over seconds 3-4 of white noise at RMS 0.02. RNNoise's attenuation of pure white noise varies a lot with level and seed (measured: 3 % to 12 % of the energy left at this level, but 55-75 % left at RMS 0.05 or 0.2), so a stricter bound would be fragile.
  • Remote branch name is voice-3-audio-dsp. The remote has a branch voice, so git refuses voice/3-audio-dsp there (refname conflict); same as Voice 1 (#102). The local branch keeps the plan's name.
  • The gate analyses 2.5 ms blocks (120 samples) where the worklet uses the 128-sample render quantum; time constants are unchanged.
  • Slider arguments are f32 (the original clamps and tolerates non-finite values; so does this).
  • The root README.md crate table is not updated (outside this task's file list).

For the reviewer

  • The gate adds 10 ms of delay to mono capture (the lookahead), as in the original.
  • A full-scale input is compressed above 0.8 even at 100 % volume, because the soft clip is always in the chain, as in the original graph.
  • Malformed frames (wrong length or channel count) never panic; RNNoise and the gate skip them.

Closes #64

🤖 Generated with Claude Code

## What this builds New crate `crates/fluxer-audio` (no devices, no network, no dependency on `fluxer-core` or LiveKit), with the constants of the Electron client: - `Frame { samples: Vec<i16>, channels: u8 }` (48 kHz, 10 ms), `SAMPLE_RATE`, `FRAME_MS`, `FRAME_LEN`. - `Resampler::new(from_hz, to_hz, channels) -> Result<_, ResampleError>`: streaming over rubato's FFT resampler; `process(&[i16], &mut Vec<i16>)`, `flush`, `reset`. The filter delay is trimmed and `flush` ends a stream with exactly the input's duration. - `gain::user_volume_gain(percent)` (per-user curve) and `gain::percent(p)` (linear 0-200 % microphone volume). - `Gate::new(GateConfig { mode: Auto | Manual { threshold_rms }, attack_ms: 5, release_ms: 20, hold_ms: 180, lookahead_ms: 10 })`: a port of `voiceGateProcessor.worklet.js` (floor = 10th percentile of 2 s x 1.5, auto threshold = 3 x floor capped at 0.04, 300 ms seeding, release at 70 % of the threshold). `gate::threshold_rms_for_slider`. - `LevelDetector::new(DetectorConfig)` (30 ms attack, 220 ms release), `level::remote_threshold_rms_for_slider`, `level::rms`. - `soft_clip(&mut [f32])` (knee 0.8, tanh shoulder). - `Denoiser::{Rnnoise, None}` (a plain selector, so `ChainConfig` is `Copy + PartialEq`) and `denoise::DenoiseProcessor` holding the RNNoise state per channel. - `CaptureChain::new(ChainConfig { denoise, gate, input_volume_percent, stereo })`, `process(&mut Frame) -> ChainOutput { rms, gate_open }`, plus `set_config` (keeps the gate's noise floor when only the volume or threshold changes) for Voice 4's `set_chain_config`. Workspace `Cargo.toml` gains `nnnoiseless` (no default features: the defaults build its CLI) and `rubato`. `deny.toml` is unchanged: every new licence was already allowed. ## How it was tested `crates/fluxer-audio/tests/dsp.rs`, synthetic signals only: the nine tests the plan names plus `gate_attack_ramp_takes_5ms`, `rms_of_reference_signals` and `chain_never_gates_stereo_capture`. Written first against a signatures-only stub (12 failed), then implemented (12 pass). `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. ## Deviations from the plan - **`user_volume_gain(0.0)` is `0.0`, not `10^-1.2`.** The plan's reference point came from the facts report, which left out the original's special case (`VoiceVolumeUtils.ts:24`: `if (clamped === 0) return 0`). Review focus 5 asks to match the original, so 0 % is silence; the test checks the `10^-1.2` limit just above zero (0.001 %) and exact zero separately. Voice 8/10 should not expect a non-zero gain at 0 %. - **`rnnoise_reduces_white_noise_energy` asserts at least 3 dB**, measured over seconds 3-4 of white noise at RMS 0.02. RNNoise's attenuation of pure white noise varies a lot with level and seed (measured: 3 % to 12 % of the energy left at this level, but 55-75 % left at RMS 0.05 or 0.2), so a stricter bound would be fragile. - **Remote branch name is `voice-3-audio-dsp`.** The remote has a branch `voice`, so git refuses `voice/3-audio-dsp` there (refname conflict); same as Voice 1 (#102). The local branch keeps the plan's name. - The gate analyses 2.5 ms blocks (120 samples) where the worklet uses the 128-sample render quantum; time constants are unchanged. - Slider arguments are `f32` (the original clamps and tolerates non-finite values; so does this). - The root `README.md` crate table is not updated (outside this task's file list). ## For the reviewer - The gate adds 10 ms of delay to mono capture (the lookahead), as in the original. - A full-scale input is compressed above 0.8 even at 100 % volume, because the soft clip is always in the chain, as in the original graph. - Malformed frames (wrong length or channel count) never panic; RNNoise and the gate skip them. Closes #64 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(audio): frames and microphone DSP chain
All checks were successful
ci / Linux (x86_64) (pull_request) Successful in 7m37s
ci / Windows (x64) (pull_request) Successful in 8m11s
ci / Linux (x86_64) (push) Successful in 7m13s
ci / Windows (x64) (push) Successful in 8m5s
fc38bd2e78
New crate fluxer-audio: Frame (48 kHz, 10 ms), streaming Resampler (rubato),
per-user volume curve, voice-activity gate (port of the fork's worklet),
speaking level detector, soft clip, RNNoise denoiser (nnnoiseless) and the
CaptureChain that runs them in the original's order.

Refs #64

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Author
Owner

Independent review: Voice 3 (audio-dsp), commit fc38bd2

Reviewed 8b37da4..origin/voice-3-audio-dsp cold, against the plan (Task 3, Global Constraints, Review Focus 5), spec §6, facts §3/§4 and the fork sources.

1. Tests named by the plan — all nine exist in crates/fluxer-audio/tests/dsp.rs and assert the spec values: gate 5/20/180/10 ms, slider points 0.0003 / 0.008 / 0.04 (log below 50) and 0.002 / 0.006 / 0.035, detector 30/220 ms, soft clip bound, 44.1 k → 48 k duration (exactly 48 000 frames), RNNoise energy drop, volume applied after the gate. Three extra tests (gate_attack_ramp_takes_5ms, rms_of_reference_signals, chain_never_gates_stereo_capture).

2. Interfaces block — every name exists as written: Frame { samples, channels }, Resampler::new(from_hz, to_hz, channels) (returns Result), gain::user_volume_gain, gain::percent, Gate::new(GateConfig { mode: GateMode::{Auto, Manual { threshold_rms }}, attack_ms, release_ms, hold_ms, lookahead_ms }) with defaults 5/20/180/10, gate::threshold_rms_for_slider, LevelDetector::new(DetectorConfig), level::remote_threshold_rms_for_slider, soft_clip(&mut [f32]), Denoiser::{Rnnoise, None}, CaptureChain::new(ChainConfig { denoise, gate, input_volume_percent, stereo }), process(&mut self, &mut Frame) -> ChainOutput { rms, gate_open }.

3. Constraints / invariants — fluxer-audio depends only on nnnoiseless, rubato, thiserror (no fluxer-core, no LiveKit); no other crate touched; no secrets or fixtures; malformed frames (0/1/479/481/5000 samples, channels 0/3/255, NaN config) do not panic in my probe. Gate, thresholds, volume curve and soft clip compared line by line with voiceGateProcessor.worklet.js, VoiceSpeakingThreshold.ts, VoiceVolumeUtils.ts, VoiceSharedAudioContext.ts: faithful port (only difference: 120-sample analysis blocks instead of 128).

4. Review Focus 5 — curve matches the original at 50/100/150/200 % to 1e-4. At 0 % the original returns 0 (VoiceVolumeUtils.ts: if (clamped === 0) return 0), so the implementation is right and the plan's 0 → 10^-1.2 is wrong (see finding 1). The soundboard cap of 3 is Voice 10's.

5. cargo xtask ci in the review worktree: green (fmt, clippy, deny, core-isolation, parity, test; 12/12 in dsp.rs; screenshot step skipped because FLUXER_SCREENSHOT_TESTS was not set). Remote runners not checked.

Findings

  1. minor — gain.rs / plan: user_volume_gain(0.0) == 0.0 deviates from the plan's test text but matches the fork. Voice 8 and Voice 10 (and the plan / facts §3) must expect silence at 0 %, not 10^-1.2.
  2. minor — resample.rs: only rate pairs with a large common divisor work. Probe (1 s of input, then flush): 8 k, 11.025 k, 16 k, 22.05 k, 32 k, 44.1 k, 88.2 k, 96 k, 176.4 k, 192 k → exactly 48 000 frames, mono and stereo. But Resampler::new(47_999, 48_000, _) delivers nothing while streaming and only 24 000 of 48 000 frames after flush (the doc promise "exactly the input's duration" does not hold), and Resampler::new(u32::MAX, 48_000, 1) did not finish within 60 s instead of returning an error. Rates come from the local device, not the server, and real devices use the rates that work, so not blocking; suggest a follow-up issue to reject rates outside a sane range before Voice 4 passes device rates in.
  3. minor — level.rs: LevelDetector::update accumulates dt_ms, so speaking turns on one tick earlier than the original's timestamp difference (advanceSpeakingHysteresis): on the 3rd 10 ms frame instead of the 4th, and on the first 50 ms window instead of the second. Not audible; note it if Voice 8 compares indicator timing with the original.
  4. minor — rnnoise_reduces_white_noise_energy asserts only 3 dB. Acceptable: fixed seed, pure-Rust network, passes locally; I did not verify the Windows runner.
  5. minor — root README.md crate table has no fluxer-audio row (outside this task's file list; the crate README itself is complete).

Verdict: APPROVE — zero blocking findings, local ci green.

## Independent review: Voice 3 (audio-dsp), commit fc38bd2 Reviewed `8b37da4..origin/voice-3-audio-dsp` cold, against the plan (Task 3, Global Constraints, Review Focus 5), spec §6, facts §3/§4 and the fork sources. **1. Tests named by the plan** — all nine exist in `crates/fluxer-audio/tests/dsp.rs` and assert the spec values: gate 5/20/180/10 ms, slider points 0.0003 / 0.008 / 0.04 (log below 50) and 0.002 / 0.006 / 0.035, detector 30/220 ms, soft clip bound, 44.1 k → 48 k duration (exactly 48 000 frames), RNNoise energy drop, volume applied after the gate. Three extra tests (`gate_attack_ramp_takes_5ms`, `rms_of_reference_signals`, `chain_never_gates_stereo_capture`). **2. Interfaces block** — every name exists as written: `Frame { samples, channels }`, `Resampler::new(from_hz, to_hz, channels)` (returns `Result`), `gain::user_volume_gain`, `gain::percent`, `Gate::new(GateConfig { mode: GateMode::{Auto, Manual { threshold_rms }}, attack_ms, release_ms, hold_ms, lookahead_ms })` with defaults 5/20/180/10, `gate::threshold_rms_for_slider`, `LevelDetector::new(DetectorConfig)`, `level::remote_threshold_rms_for_slider`, `soft_clip(&mut [f32])`, `Denoiser::{Rnnoise, None}`, `CaptureChain::new(ChainConfig { denoise, gate, input_volume_percent, stereo })`, `process(&mut self, &mut Frame) -> ChainOutput { rms, gate_open }`. **3. Constraints / invariants** — `fluxer-audio` depends only on `nnnoiseless`, `rubato`, `thiserror` (no `fluxer-core`, no LiveKit); no other crate touched; no secrets or fixtures; malformed frames (0/1/479/481/5000 samples, channels 0/3/255, NaN config) do not panic in my probe. Gate, thresholds, volume curve and soft clip compared line by line with `voiceGateProcessor.worklet.js`, `VoiceSpeakingThreshold.ts`, `VoiceVolumeUtils.ts`, `VoiceSharedAudioContext.ts`: faithful port (only difference: 120-sample analysis blocks instead of 128). **4. Review Focus 5** — curve matches the original at 50/100/150/200 % to 1e-4. At 0 % the original returns 0 (`VoiceVolumeUtils.ts`: `if (clamped === 0) return 0`), so the implementation is right and the plan's `0 → 10^-1.2` is wrong (see finding 1). The soundboard cap of 3 is Voice 10's. **5. `cargo xtask ci`** in the review worktree: green (fmt, clippy, deny, core-isolation, parity, test; 12/12 in `dsp.rs`; screenshot step skipped because `FLUXER_SCREENSHOT_TESTS` was not set). Remote runners not checked. ### Findings 1. **minor** — `gain.rs` / plan: `user_volume_gain(0.0) == 0.0` deviates from the plan's test text but matches the fork. Voice 8 and Voice 10 (and the plan / facts §3) must expect silence at 0 %, not `10^-1.2`. 2. **minor** — `resample.rs`: only rate pairs with a large common divisor work. Probe (1 s of input, then `flush`): 8 k, 11.025 k, 16 k, 22.05 k, 32 k, 44.1 k, 88.2 k, 96 k, 176.4 k, 192 k → exactly 48 000 frames, mono and stereo. But `Resampler::new(47_999, 48_000, _)` delivers nothing while streaming and only 24 000 of 48 000 frames after `flush` (the doc promise "exactly the input's duration" does not hold), and `Resampler::new(u32::MAX, 48_000, 1)` did not finish within 60 s instead of returning an error. Rates come from the local device, not the server, and real devices use the rates that work, so not blocking; suggest a follow-up issue to reject rates outside a sane range before Voice 4 passes device rates in. 3. **minor** — `level.rs`: `LevelDetector::update` accumulates `dt_ms`, so speaking turns on one tick earlier than the original's timestamp difference (`advanceSpeakingHysteresis`): on the 3rd 10 ms frame instead of the 4th, and on the first 50 ms window instead of the second. Not audible; note it if Voice 8 compares indicator timing with the original. 4. **minor** — `rnnoise_reduces_white_noise_energy` asserts only 3 dB. Acceptable: fixed seed, pure-Rust network, passes locally; I did not verify the Windows runner. 5. **minor** — root `README.md` crate table has no `fluxer-audio` row (outside this task's file list; the crate README itself is complete). **Verdict: APPROVE** — zero blocking findings, local ci green.
charles merged commit c2454549c2 into voice 2026-11-11 18:30:54 +00:00
Sign in to join this conversation.
No description provided.