DeepFilterNet: a late frame no longer ends it; the warm-up runs the networks (#910) #912

Closed
charles wants to merge 0 commits from fix-910-deepfilter-late-frame into textchat
Owner

Closes #910

Cause

  1. Why the first frame was slow. The loader warmed DeepFilterNet up with 8 silent frames, as the original does. But libDF's DfTract::process returns before the networks when the input is silent (energy < 1e-7). So the warm-up ran none of the three networks. The first frame of sound (the check's white noise, or a microphone's room noise) was the first tract run on that session and thread. That run allocates tract's per-session, per-thread matmul scratch, packs operands cold (packs are kept only from the second and third run), and runs the kernel code for the first time. On Linux this makes the first frame about 4–5× slower (0.5–0.6 ms vs 0.12 ms). On a loaded Windows runner, where page faults on new memory and on the executable's code pages cost much more, that is the likely source of the 250 ms frame. The worker thread waiting in recv() is not the cause. Neither is debug vs release: tract and libDF build at opt-level 3 in dev, and release run 382 hit the problem too.
  2. Why one late frame ended it. Remote::process blocked the engine thread for up to 250 ms. On a timeout it ended DeepFilterNet for the whole session. The engine thread also refills playback, so during that wait the user heard about 220 ms of silence on the loudspeaker (the playback ring holds about 30 ms). The microphone's 250 ms then went out as one late burst.

Fix

  • Warm-up (deep_filter.rs): a noisy tone goes through process, the same path real frames take. It runs until the encoder, the gain decoder and the deep-filter decoder have each run 3 times (5 frames here). This happens on the loader thread, which is also the thread that cleans the call. The stream state is then restored, so the call starts clean. The restore uses DfTract::stream/set_stream, added to the vendored libDF (listed in PATCHES.md). Restoring keeps the tract sessions. A clone of the model would start new sessions, cold again.
  • Real-time rule (denoise.rs): the chain waits at most 8 ms for a frame. If the frame is not back by then, it goes out through a stand-in: the unprocessed input, delayed by DeepFilterNet's own 3 frames. The audio stays continuous and in order, with no backlog and no added latency. DeepFilterNet still processes that frame, so its own state stays continuous, and the late result is discarded. RNNoise is not the stand-in because its state is stale and its delay is different, so swapping it in would repeat 20 ms.
  • Giving up happens only after N = 50 late frames in a row, or when frames take more than 8 ms on average over 200 frames (timed on the worker). 50 frames is 0.5 s. A shorter stall (a page-in, the scheduler, a virus scan) is covered by the stand-in. A thread that still hasn't caught up after 0.5 s is stuck, or the machine can't keep up, and RNNoise is better than more unprocessed sound. A worker that never answers fails with a new DeepFilterError::Stalled ("it did not answer for N ms"), not "too slow".
  • --check-noise-suppression fails only when the chain gives DeepFilterNet up. It measures noise reduction only on the frames DeepFilterNet processed, and it reports the first frame's time and the number of late frames.
  • xtask package: the three-attempt workaround is removed. It worked around the single-frame failure, which no longer exists. A "too slow to keep up" result is still only a warning, since it now means at least 0.5 s of sustained slowness on the build runner, which is not a fault in the package. Stalled and every other failure still fail the package.

Verification

  • Scripted slow worker (denoise::tests), each test checking that every output frame is the input from 3 frames earlier (in order, no repeats or gaps) and that the caller never waits much longer than the budget:

    • late first frame: stand-in for that frame, DeepFilterNet carries on;
    • one late frame mid-stream: same;
    • 12 ms per frame: given up on the 50th late frame, "takes 12 ms … too slow";
    • slow average: given up after the window;
    • worker that never answers: given up after 50 frames with Stalled.

    Against the old wait logic (temporarily restored), 4 of these fail: the old code waited 120 ms instead of standing in, and failed after one frame with "250 ms".

  • silence_runs_no_network shows the root cause. the_warm_up_runs_every_network (mono and stereo) checks the new warm-up. the_warm_up_leaves_no_trace_in_the_stream checks that a warmed instance gives the same output as a cold one (within 1e-5). the_reported_delay_is_the_real_one measures the 3-frame delay by cross-correlation.

  • flaxative --check-noise-suppression, 3 runs on Linux: started in 123 ms, first frame 161µs, 134µs per 10 ms frame, 0 of 300 frames late, white noise lowered by 30.0 dB.

  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: all steps passed.

Not verified: Windows. Point 1 is reasoned from the code and measured on Linux only. Whether the first-run cost on the Windows runners was the whole 250 ms can only be confirmed there. The check now prints first frame to show it.

🤖 Generated with Claude Code

Closes #910 ## Cause 1. **Why the first frame was slow.** The loader warmed DeepFilterNet up with 8 silent frames, as the original does. But libDF's `DfTract::process` returns before the networks when the input is silent (energy < 1e-7). So the warm-up ran **none** of the three networks. The first frame of sound (the check's white noise, or a microphone's room noise) was the first tract run on that session and thread. That run allocates tract's per-session, per-thread matmul scratch, packs operands cold (packs are kept only from the second and third run), and runs the kernel code for the first time. On Linux this makes the first frame about 4–5× slower (0.5–0.6 ms vs 0.12 ms). On a loaded Windows runner, where page faults on new memory and on the executable's code pages cost much more, that is the likely source of the 250 ms frame. The worker thread waiting in `recv()` is not the cause. Neither is debug vs release: tract and libDF build at opt-level 3 in dev, and release run 382 hit the problem too. 2. **Why one late frame ended it.** `Remote::process` blocked the engine thread for up to 250 ms. On a timeout it ended DeepFilterNet for the whole session. The engine thread also refills playback, so during that wait the user heard about 220 ms of silence on the loudspeaker (the playback ring holds about 30 ms). The microphone's 250 ms then went out as one late burst. ## Fix - **Warm-up** (`deep_filter.rs`): a noisy tone goes through `process`, the same path real frames take. It runs until the encoder, the gain decoder and the deep-filter decoder have each run 3 times (5 frames here). This happens on the loader thread, which is also the thread that cleans the call. The stream state is then restored, so the call starts clean. The restore uses `DfTract::stream`/`set_stream`, added to the vendored libDF (listed in PATCHES.md). Restoring keeps the tract sessions. A clone of the model would start new sessions, cold again. - **Real-time rule** (`denoise.rs`): the chain waits at most **8 ms** for a frame. If the frame is not back by then, it goes out through a stand-in: the unprocessed input, delayed by DeepFilterNet's own 3 frames. The audio stays continuous and in order, with no backlog and no added latency. DeepFilterNet still processes that frame, so its own state stays continuous, and the late result is discarded. RNNoise is not the stand-in because its state is stale and its delay is different, so swapping it in would repeat 20 ms. - **Giving up** happens only after **N = 50 late frames in a row**, or when frames take more than 8 ms on average over 200 frames (timed on the worker). 50 frames is 0.5 s. A shorter stall (a page-in, the scheduler, a virus scan) is covered by the stand-in. A thread that still hasn't caught up after 0.5 s is stuck, or the machine can't keep up, and RNNoise is better than more unprocessed sound. A worker that never answers fails with a new `DeepFilterError::Stalled` ("it did not answer for N ms"), not "too slow". - **`--check-noise-suppression`** fails only when the chain gives DeepFilterNet up. It measures noise reduction only on the frames DeepFilterNet processed, and it reports the first frame's time and the number of late frames. - **`xtask package`**: the three-attempt workaround is **removed**. It worked around the single-frame failure, which no longer exists. A "too slow to keep up" result is still only a warning, since it now means at least 0.5 s of sustained slowness on the build runner, which is not a fault in the package. `Stalled` and every other failure still fail the package. ## Verification - Scripted slow worker (`denoise::tests`), each test checking that every output frame is the input from 3 frames earlier (in order, no repeats or gaps) and that the caller never waits much longer than the budget: - late first frame: stand-in for that frame, DeepFilterNet carries on; - one late frame mid-stream: same; - 12 ms per frame: given up on the 50th late frame, "takes 12 ms … too slow"; - slow average: given up after the window; - worker that never answers: given up after 50 frames with `Stalled`. Against the old wait logic (temporarily restored), 4 of these fail: the old code waited 120 ms instead of standing in, and failed after one frame with "250 ms". - `silence_runs_no_network` shows the root cause. `the_warm_up_runs_every_network` (mono and stereo) checks the new warm-up. `the_warm_up_leaves_no_trace_in_the_stream` checks that a warmed instance gives the same output as a cold one (within 1e-5). `the_reported_delay_is_the_real_one` measures the 3-frame delay by cross-correlation. - `flaxative --check-noise-suppression`, 3 runs on Linux: `started in 123 ms, first frame 161µs, 134µs per 10 ms frame, 0 of 300 frames late, white noise lowered by 30.0 dB`. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci`: all steps passed. **Not verified:** Windows. Point 1 is reasoned from the code and measured on Linux only. Whether the first-run cost on the Windows runners was the whole 250 ms can only be confirmed there. The check now prints `first frame` to show it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The loader warmed DeepFilterNet up with silent frames, but libDF returns before
the networks for silence, so the first frame of sound paid for tract's first
runs (per-session, per-thread scratch and packing, first touch of the kernels).
The warm-up now runs a noisy tone through all three networks three times on
the thread and instance that clean the call, then restores the stream state
(DfTract::stream/set_stream, added to the vendored libDF).

The chain waits at most 8 ms for a frame. A frame not back by then goes out as
the unprocessed input delayed by DeepFilterNet's 3 frames, so the sound stays in
order with no added latency; the late result is dropped and DeepFilterNet
carries on. It is given up after 50 late frames in a row or an average above
8 ms over 200 frames. Scripted-worker tests cover a late first frame, a late
frame mid-stream, sustained slowness, a slow average and a worker that never
answers. The packaged check runs once (no more three attempts) and measures
the noise only on processed frames.

Refs #910

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Merged into textchat by hand (the merge API returns 405); the commits are in the branch.

Merged into `textchat` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-18 19:50:54 +00:00
charles referenced this pull request from a commit 2026-11-18 22:34:35 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.