Voice 5: fluxer-audio clips and chimes (#66) #118
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "voice-5-audio-clips-chimes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this adds
fluxer-audiocan now play short sounds into the mixer (Voice spec §6 "Clips", §5.2 "Chimes").clip:clip::decode(bytes, content_type) -> Result<Clip, ClipError>decodes mp3, ogg/vorbis, m4a/aac and wav withsymphonia0.6 into 48 kHz stereo. Files over 20 MiB are refused before decoding and decoding stops once a sound passes 30 s. The content type is only a hint. A decoder panic on server data is caught and becomesClipError::Malformed.ClipCache::new(32)is an LRU keyed by string.chime:Chime::{UserJoin, UserLeave, Mute, Unmute, Deafen, Undeafen, VoiceDisconnect, ViewerJoin, ViewerLeave, IncomingCall}, with the ten mp3 files copied unchanged from the fork'sfluxer_app/src/media/sounds/intoassets/sounds/(AGPL-3.0-or-later note in the crate README). Each is decoded once, outside the LRU.engine:AudioEngine::play_clip(key, bytes: Option<Bytes>, gain, tag) -> Result<Duration, ClipError>,play_chime(Chime, gain),stop_clips(), andEngineEvent::ClipFinished { tag }. Decoding runs on the caller's thread; the worker feeds each clip to the mixer 10 ms at a time as its own source, so clips, chimes and voices overlap and share the soft clipper.How it was tested
crates/fluxer-audio/tests/clips.rs, all onNullHost:decodes_mp3_ogg_m4a_wav_fixtures_to_48k_stereo(duration, pitch, level, both channels; also with no, parameterised and wrong content types),rejects_overlong_clip(31 s, over 20 MiB, non-audio, truncated, header-only),cache_evicts_least_recently_used_at_33,play_clip_mixes_into_output_and_reports_finished(mixed with a remote voice, level and audible duration checked, replay from cache,NotCached),every_chime_asset_decodes;stop_clips, replaying a tag, no output device, chimes report noClipFinished.The fixtures in
tests/fixtures/are 0.5 s 440 Hz tones generated with ffmpeg.FLUXER_SCREENSHOT_TESTS=1 cargo xtask cipasses locally (clippy, cargo deny, all tests).Deviations from the plan, and decisions it leaves open
mixer.rsgainsSourceId::Chime(u64)(chimes have no caller tag, so they cannot shareClip(tag)) andMixer::forget(drops a finished clip's gain, whichremovekeeps on purpose for participants)..gitattributesmarks*.mp3 *.ogg *.m4a *.wavas binary.bytesbecomes a dependency offluxer-audiobecause the interface takesBytes; it was already a workspace dependency.play_chimereturnsResult<Duration, ClipError>; the plan gives no return type.AudioEngine::has_clip(key)is an addition, so the runtime can check the cache without playing.ClipFinishedis also sent for clips cut bystop_clips, and at once when no output device can be opened, so a caller waiting on a tag always gets its event. Replaying a tag that is still playing replaces the clip and reports once.stop_clipsdoes not stop chimes, so the disconnect chime survives the soundboard being stopped.symphoniais built without its ID3 metadata features; an mp3 with ID3v2 and ID3v1 tags was checked by hand and decodes.Chime::IncomingCallplays one 4.5 s ring; repeating it is left to the caller (Voice 22).Stacked on: voice-4-audio-devices-mixer
Closes #66
🤖 Generated with Claude Code
Independent review: Voice 5 (audio clips and chimes), commit
e0757c6Verdict: changes requested (1 blocking, 4 minor). Local
FLUXER_SCREENSHOT_TESTS=1 cargo xtask cion the review worktree: all steps passed. Remote runners not checked.Blocking
B1. A sound file with an unusual sample rate hangs or aborts the client (
clip.rs,decode_inner). The file's sample rate goes straight intoResampler::new(rate, 48_000, 2). The FFT resampler sizes its buffers from the two rates divided by their gcd, so a rate that shares no factor with 48 000 costs time and memory proportional to the rate itself. Measured with a 4 KB 8-bit mono WAV whose header states the rate (debug build, 8 GB address-space cap):memory allocation of 408146688 bytes failed, SIGABRTmemory allocation of 16000000028 bytes failed, SIGABRTAn allocation failure is an abort, which the
catch_unwinddoes not catch. The server would accept such a file:SoundboardService.tsstores the uploaded bytes and checks only size (20 MiB) and duration (100 ms to 30 s), and 100 ms of 8-bit mono at 200 MHz is 20 MB. So any member who may upload a sound can take down every native client that plays it, which breaks "server data must never panic the client". Suggested fix: refuse rates outside a sane range (for example 8 kHz to 192 kHz) withClipError::Unsupportedbefore building the resampler, with a test.Minor
M1. Short or odd-rate clips can decode to an empty or truncated
Clipand still returnOk. Same probe: 4000 frames at 44 101 Hz gaveOkwith 0 frames, 60 000 frames (1.36 s) gave 24 000 frames (0.5 s). Standard rates are fine (44.1 k, 22.05 k, 11.025 k, 8 k checked, down to 3 frames). The truncation comes from the Voice 3 resampler'sflushround bound, not this PR, butdecodeshould returnClipError::Emptywhen the resampled clip is empty. The range check of B1 does not remove this; rejecting rates whose gcd with 48 000 is small would.M2. Padding in
finish_played_clipswith a large device period. Confirmed as the implementer described: the engine sets the prefill to the device period rounded up plus 10 ms (capped at 120 ms), so with a period above about 100 ms a clip shorter than the prefill is padded pastMAX_QUEUE_MSand loses up to its first 10 ms. Only clips shorter than 120 ms are affected; every chime is longer than 300 ms. No clip is lost andClipFinishedis still sent.M3. Cache memory. 32 clips of up to 30 s are about 5.8 MB each decoded, so about 184 MB worst case. That is what the spec asks for (LRU of 32); worth a byte budget later.
M4.
play_clipignoresbytesfor a cached key. Fine while a sound's audio never changes under one id; the caller must use a new key if that ever changes. Documented in the rustdoc.Checked and fine
tests/clips.rsand assert the plan's values: all four fixtures to 48 kHz stereo (pitch, level, both channels), 31 s refused and 29 s accepted, 20 MiB + 1 refused, eviction at the 33rd insert with LRU order, clip mixed with a remote voice and exactly oneClipFinished, all 10 chimes decode. All run onNullHost.clip::decode(&[u8], Option<&str>) -> Result<Clip, ClipError>,ClipCache::new(32), the tenChimevariants,play_clip(&str, Option<Bytes>, f32, u64) -> Result<Duration, ClipError>,play_chime(Chime, f32),stop_clips(),EngineEvent::ClipFinished { tag: u64 }all exist as written.fluxer-audiodepends on neitherfluxer-corenor LiveKit; decoding and the cache lock are on the caller's thread, the device callbacks are untouched; no secrets in code or fixtures;symphoniais the crate the plan names,byteswas already a workspace dependency.fluxer_app/src/media/sounds/in the fork; the AGPL note is in the README.Review round 1, fixed in
313e865(localFLUXER_SCREENSHOT_TESTS=1 cargo xtask cigreen).Blocking: unchecked sample rate in
decode_inner— fixed.clip::decodenow refuses rates outsideMIN_CLIP_RATE..=MAX_CLIP_RATE(8 kHz to 192 kHz) withClipError::Unsupportedat the first decoded packet, before anything is sized by the rate and before the resampler is built. Testrefuses_sample_rates_outside_8_to_192_khz: 7 999, 192 001, 48 000 001, 200 000 003, 4 000 000 007, 1 andu32::MAXHz are each refused in under a second; 8 000 and 192 000 Hz decode. Residual cost inside the range: a rate sharing no factor with 48 kHz still makes the resampler work in one-second blocks; the worst case I measured is a 29 s mono clip at 191 999 Hz, 8.9 s in a debug build on the calling thread (bounded, no crash; release not measured). Recorded in #119.Minor: odd-rate clips empty or truncated — fixed. Cause: the FFT resampler takes about 10 ms per call but emits nothing until a block of
from_hz / gcdframes is full (one second at 44 101 Hz, plus half a block of delay), andResampler::flushgave up after 64 rounds. The bound is now computed from the block size and the output still owed (flush_rounds); standard rates behave as before.decodealso returnsClipError::Emptyinstead ofOkwith an empty clip. Testodd_sample_rates_keep_their_whole_duration(44 101, 11 027, 191 999, 47 999 Hz: duration within 1 ms, level at the end, pitch). This touchesresample.rsfrom Voice 3, same crate.Minor:
finish_played_clipspadding pastMAX_QUEUE_MS— fixed.feed_clipsfills a clip's last frame up to a whole 10 ms, so the queue holds whole frames and the padding lands exactly on the prefill; the padding is also skipped when it would exceedMAX_QUEUE_MS. Testa_short_clip_is_played_whole_on_a_device_with_a_long_perioduses a test host with a 110 ms period (it reproduced the loss: a 115 ms clip was heard for 110 ms) and checks 115, 95 and 25 ms clips sample for sample, withClipFinished.Minor: cache can hold about 184 MB — not changed (matches the spec's LRU of 32). Follow-up: #119.
Minor:
play_clipignores bytes for a cached key — behaviour not changed; the rustdoc now says a sound whose audio changes needs a new key. Follow-up for the callers: #119.Independent review, round 2 — Voice 5 (clips and chimes) at
313e865Scope:
git diff 65cd5a3..313e865. Read cold; implementer's notes verified, not trusted.Verdict: APPROVE — 0 blocking, 3 minor. Local
FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciis green (all steps passed); remote runners not checked.Checks
crates/fluxer-audio/tests/clips.rsand assert what the plan says:decodes_mp3_ogg_m4a_wav_fixtures_to_48k_stereo(four fixtures, duration, pitch, level, both channels),rejects_overlong_clip(31 s refused, 29 s accepted, 20 MiB + 1 refused),cache_evicts_least_recently_used_at_33(ClipCache::new(32), the 33rd insert evicts the least recently used,getcounts as a use),play_clip_mixes_into_output_and_reports_finished(mixed with a remote voice, exactly oneClipFinished, cached replay without bytes),every_chime_asset_decodes(all 10). Eight further tests cover the round 1 fixes. The clips test binary passed 15 runs in a row.clip::decode(&[u8], Option<&str>) -> Result<Clip, ClipError>,ClipCache::new(32)keyed by string, the tenChimevariants,AudioEngine::play_clip(&str, Option<Bytes>, f32, u64) -> Result<Duration, ClipError>,play_chime(Chime, f32),stop_clips()andEngineEvent::ClipFinished { tag }all exist as written. Additions beyond the plan (has_clip,SourceId::Chime,Mixer::forget,play_chimereturning the duration) are additive.fluxer-audiogains onlysymphonia(named by the task) andbytes(already in the workspace, required by the signature); nofluxer-coreor LiveKit dependency. Device callbacks are untouched; the newMutex<ClipCache>is locked only on the caller's thread. No secrets: fixtures carry only ffmpeg/LAME encoder tags. The ten chime files are byte-identical to the fork'sfluxer_app/src/media/sounds/, and the README has the licence note. Commits carryRefs #66and the attribution line.Findings
crates/fluxer-audio/src/clip.rs: thecatch_unwindindecodeis load-bearing, not only defensive, and no test exercises it. A 44-byte-header WAV claiming 65535 channels makessymphonia-format-riff0.6.1 panic (wave/chunks.rs:100, "attempt to multiply with overflow") in a debug build;decodeturns it intoClipError::Malformed("the decoder gave up"). A regression test with that file would catch a laterpanic = "abort"profile or a refactor that drops the guard. The default panic hook still prints the message to stderr, which is harmless here.crates/fluxer-audio/src/engine.rs(play_clip): decoding runs on the calling thread and, inside the accepted range, a rate sharing no factor with 48 kHz resamples in one-second blocks (the implementer measured 8.9 s for a 29 s clip at 191999 Hz in a debug build). Any member can upload such a file, so Voice 14 must callplay_clipoff the UI thread. Already recorded in #119; noting it so the integrator carries it to Voice 14.crates/fluxer-audio/src/engine.rs(tick): when no output device can be opened, chimes are dropped silently and clips reportClipFinishedat once, althoughplay_clipreturned the full duration. Core's "speaking for the clip duration + 80 ms" (spec §5.2) should therefore key onClipFinished, not only on the returned duration. Documented in the rustdoc and README; no code change needed here.Not findings: the cache is bounded by count, not bytes (matches the spec; in #119), and
play_clipignores bytes for a cached key (documented; in #119).