Screen share 1: fluxer-video crate — frames, conversions, FrameBus, test pattern, preview JPEG (#148) #188

Manually merged
charles merged 0 commits from screenshare-1-video-frames into screenshare 2026-11-12 05:17:14 +00:00
Owner

What this builds

New crate crates/fluxer-video (pure Rust; depends only on image and thiserror; no OS, LiveKit or iced dependency):

  • frame: VideoFrame { width, height, timestamp_us, data: FrameData }, FrameData::{I420 { y, u, v, strides }, Nv12 { y, uv, strides }, Bgra { data, stride }, Rgba { data, stride }} with Arc<[u8]> planes and u32 byte strides; VideoFrame::{i420, nv12, bgra, rgba} constructors for tightly packed frames; validate(), FrameError, PixelFormat.
  • convert: bgra_to_i420, i420_to_rgba, nv12_to_i420, plus to_i420 / to_rgba from any format. They return Result<VideoFrame, FrameError>.
  • scale::area_downscale(&VideoFrame, max_w, max_h) -> VideoFrame: a box filter that keeps the aspect ratio and returns even dimensions.
  • pattern::TestPattern::new(w, h).frame(n) and pattern::read_counter: a row of 2 sync cells and 16 counter bits, above 8 moving colour bars.
  • jpeg::preview_jpeg / preview_jpeg_with(&frame, &PreviewBudget): the fork's previewEncoding.ts loop with the StreamConstants.ts values.
  • bus: FrameBus (Clone + Send + Sync) with publish, get, generation, remove and keys, and VideoKey::{Remote { sid }, LocalCapture, Preview { stream_key }, Source { id }}.

How it was tested

  • cargo test -p fluxer-video: 20 tests in tests/{convert,bus,jpeg}.rs. They cover every test the plan names, plus odd sizes and padded strides, NV12, malformed frames (they return errors and never panic), a tight budget that lowers quality and then size, an impossible budget, the counter read back from a decoded JPEG, and checks that the bus does not keep replaced frames and that removed keys never reuse a generation.
  • I checked the conversions against libyuv. I linked a small C++ program against the prebuilt libwebrtc.a (webrtc-a545de4) and dumped I420ToABGR over a 232k-point YUV grid and ARGBToI420 over a 140k-point RGB grid. I also dumped ARGBToI420 chroma for 2,000 random 64x2 images, so the 2x2 chroma averaging was checked too. Our output matched every value exactly. The known-colour tests use values taken from that program.
  • I checked that the stall detector works. I made get take the writer lock while publish slept 6 ms inside it, and bus_reader_never_blocks_writer then failed with "329 reads blocked for 5 ms or more". The real implementation passes repeatedly at load average about 38.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passes locally. No real-device tests apply to this task.

Deviations from the plan

  1. bgra_i420_round_trip_within_2_levels: "within 2 levels" cannot hold for every colour while staying bit-exact with libyuv, as the plan requires. libyuv's own BT.601 limited-range round trip loses up to 3 levels of red and 5 of blue on some saturated colours (measured on libyuv itself: about 12% of random colours are off by more than 2). The main causes are that libyuv caps UB at 128 instead of 2.018 × 64, and chroma is rounded to 8 bits. The test keeps its name and asserts:
    • every grey comes back within 2 levels;
    • muted colours are within (2, 2, 3);
    • any colour is within libyuv's own bounds (3, 2, 5);
    • the mean error is below 1 level for greys and muted colours, and below 2 levels for any colour.
  2. bus_reader_never_blocks_writer: on Linux the test measures blocked time (wall time minus on-CPU time and run-queue time from /proc/thread-self/schedstat) instead of wall time. A plain 5 ms wall-clock limit flaked on this loaded machine because of preemption, not because of locks. On other OSes it uses wall time and tolerates 3 slow calls.
  3. The FrameBus uses std only. The spec says "lock-free for the reader (one atomic swap per write)", but no task adds arc-swap. Each key has two RwLock slots and an atomic current index. A writer fills the spare slot and flips the index, so a reader takes an uncontended read lock just long enough to clone an Arc. Generations come from one counter for the whole bus and are taken under the per-key writer lock, so they only grow, even after a key is removed and published again.
  4. Integer JPEG qualities 70/55/40. The fork's float loop (0.7 − 0.15 − 0.15 = 0.4000000000000001) makes one extra attempt at "0.4". Here the steps are exact decimals; the attempt limit (6) and the ×0.8 size step are unchanged.
  5. Things the Interfaces block left open, decided here:
    • conversions return Result;
    • area_downscale returns a frame that already fits, or a malformed frame, unchanged;
    • YUV input comes out as I420, packed input stays packed;
    • the counter holds 16 bits, so read_counter gives n mod 65536.
  6. The workspace Cargo.toml needed no change: crates/* is already a member glob, and image already has jpeg. Only Cargo.lock changed outside the crate.

Commit attribution is Claude Opus 5.5, the model that did the work; the repo CLAUDE.md names Fable 5.2.

Closes #148

🤖 Generated with Claude Code

## What this builds New crate `crates/fluxer-video` (pure Rust; depends only on `image` and `thiserror`; no OS, LiveKit or iced dependency): - `frame`: `VideoFrame { width, height, timestamp_us, data: FrameData }`, `FrameData::{I420 { y, u, v, strides }, Nv12 { y, uv, strides }, Bgra { data, stride }, Rgba { data, stride }}` with `Arc<[u8]>` planes and `u32` byte strides; `VideoFrame::{i420, nv12, bgra, rgba}` constructors for tightly packed frames; `validate()`, `FrameError`, `PixelFormat`. - `convert`: `bgra_to_i420`, `i420_to_rgba`, `nv12_to_i420`, plus `to_i420` / `to_rgba` from any format. They return `Result<VideoFrame, FrameError>`. - `scale::area_downscale(&VideoFrame, max_w, max_h) -> VideoFrame`: a box filter that keeps the aspect ratio and returns even dimensions. - `pattern::TestPattern::new(w, h).frame(n)` and `pattern::read_counter`: a row of 2 sync cells and 16 counter bits, above 8 moving colour bars. - `jpeg::preview_jpeg` / `preview_jpeg_with(&frame, &PreviewBudget)`: the fork's `previewEncoding.ts` loop with the `StreamConstants.ts` values. - `bus`: `FrameBus` (`Clone + Send + Sync`) with `publish`, `get`, `generation`, `remove` and `keys`, and `VideoKey::{Remote { sid }, LocalCapture, Preview { stream_key }, Source { id }}`. ## How it was tested - `cargo test -p fluxer-video`: 20 tests in `tests/{convert,bus,jpeg}.rs`. They cover every test the plan names, plus odd sizes and padded strides, NV12, malformed frames (they return errors and never panic), a tight budget that lowers quality and then size, an impossible budget, the counter read back from a decoded JPEG, and checks that the bus does not keep replaced frames and that removed keys never reuse a generation. - I checked the conversions against libyuv. I linked a small C++ program against the prebuilt `libwebrtc.a` (webrtc-a545de4) and dumped `I420ToABGR` over a 232k-point YUV grid and `ARGBToI420` over a 140k-point RGB grid. I also dumped `ARGBToI420` chroma for 2,000 random 64x2 images, so the 2x2 chroma averaging was checked too. Our output matched every value exactly. The known-colour tests use values taken from that program. - I checked that the stall detector works. I made `get` take the writer lock while `publish` slept 6 ms inside it, and `bus_reader_never_blocks_writer` then failed with "329 reads blocked for 5 ms or more". The real implementation passes repeatedly at load average about 38. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passes locally. No real-device tests apply to this task. ## Deviations from the plan 1. **`bgra_i420_round_trip_within_2_levels`**: "within 2 levels" cannot hold for every colour while staying bit-exact with libyuv, as the plan requires. libyuv's own BT.601 limited-range round trip loses up to 3 levels of red and 5 of blue on some saturated colours (measured on libyuv itself: about 12% of random colours are off by more than 2). The main causes are that libyuv caps UB at 128 instead of 2.018 × 64, and chroma is rounded to 8 bits. The test keeps its name and asserts: - every grey comes back within 2 levels; - muted colours are within (2, 2, 3); - any colour is within libyuv's own bounds (3, 2, 5); - the mean error is below 1 level for greys and muted colours, and below 2 levels for any colour. 2. **`bus_reader_never_blocks_writer`**: on Linux the test measures *blocked* time (wall time minus on-CPU time and run-queue time from `/proc/thread-self/schedstat`) instead of wall time. A plain 5 ms wall-clock limit flaked on this loaded machine because of preemption, not because of locks. On other OSes it uses wall time and tolerates 3 slow calls. 3. **The `FrameBus` uses `std` only.** The spec says "lock-free for the reader (one atomic swap per write)", but no task adds `arc-swap`. Each key has two `RwLock` slots and an atomic current index. A writer fills the spare slot and flips the index, so a reader takes an uncontended read lock just long enough to clone an `Arc`. Generations come from one counter for the whole bus and are taken under the per-key writer lock, so they only grow, even after a key is removed and published again. 4. **Integer JPEG qualities 70/55/40.** The fork's float loop (0.7 − 0.15 − 0.15 = 0.4000000000000001) makes one extra attempt at "0.4". Here the steps are exact decimals; the attempt limit (6) and the ×0.8 size step are unchanged. 5. **Things the Interfaces block left open, decided here:** - conversions return `Result`; - `area_downscale` returns a frame that already fits, or a malformed frame, unchanged; - YUV input comes out as I420, packed input stays packed; - the counter holds 16 bits, so `read_counter` gives `n mod 65536`. 6. The workspace `Cargo.toml` needed no change: `crates/*` is already a member glob, and `image` already has `jpeg`. Only `Cargo.lock` changed outside the crate. Commit attribution is Claude Opus 5.5, the model that did the work; the repo `CLAUDE.md` names Fable 5.2. Closes #148 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(video): frames, conversions, frame bus, test pattern and preview JPEG
All checks were successful
ci / Linux (x86_64) (push) Successful in 15m16s
ci / Windows (x64) (push) Successful in 16m21s
8b8a359082
Refs #148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 1 (fluxer-video), commit 8b8a359

I read the diff 53e3d60..origin/screenshare-1-video-frames cold. It adds the fluxer-video crate and a Cargo.lock entry.

1. Plan tests. All eight named tests exist and pass. Plain reading of each:

  • i420_to_rgba_known_colours matches libyuv. I checked the blue value by hand: Y41/U239 gives B=251.
  • area_downscale_keeps_aspect_and_even_dims covers 1920x1080, 1366x768, 1001x3001, 4K and 7x5.
  • pattern_counter_survives_blur_and_downscale reads the counter within ±1 after a blur and a downscale. preview_keeps_the_picture also checks it after a real JPEG round trip.
  • preview_jpeg_respects_size_and_bytes_budget uses 4K noise and gets 640x360 within 1 000 000 bytes.
  • bus_latest_frame_wins and bus_remove_forgets_key pass.
  • bus_reader_never_blocks_writer writes at 1 kHz for 2 s and checks monotonic generations and no stall over 5 ms.

2. Interfaces. Every name in the Interfaces block exists as written: VideoFrame, FrameData, convert::{bgra_to_i420, i420_to_rgba, nv12_to_i420}, scale::area_downscale, pattern::{TestPattern, read_counter}, jpeg::{preview_jpeg, JpegError}, VideoKey and the five FrameBus methods. FrameBus is Clone + Send + Sync. Choices the plan left open:

  • Conversions return Result<VideoFrame, FrameError>.
  • Strides are u32.
  • Added: PreviewBudget, preview_jpeg_with, to_i420 and to_rgba.

These are fine for Tasks 3, 6, 16, 17 and 21.

3. Invariants.

  • The only dependencies are image (workspace, jpeg already enabled) and thiserror. No OS, LiveKit or iced dependency, and no I/O.
  • Every pixel path calls validate() first, so a malformed frame is an Err or is passed through unchanged. I found no panic path for decoder or server data. I also checked sizes like 4000x1 and a max of 1 in area_plane's spans.
  • No secrets.

4. Review Focus 6. The reader takes no writer mutex. It holds a per-slot read lock only for an Arc clone, and frames are dropped outside the locks. One frame per key, and the latest wins. Generations come from a bus-wide counter, so a key that is removed and published again never reuses one.

I checked the Slot::publish/get interleavings, including two publishes during one read. The retry when an emptied slot is no longer current is correct, and generations seen by a reader are monotonic.

5. Local CI. FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green: "all steps passed", exit 0. A second run I started at the same time on the same target dir failed xtask --test package because both runs used the same appimagetool .partial path. Run alone, that test passes. It is unrelated to this diff.

Findings

  • minor (src/bus.rs, Slot::publish): generation is stored before current flips. During that window, generation(&key) can return g+1 while get(&key) still returns frame g. Task 6 will compare generation(). If a widget records the g+1 from generation() but uploads frame g, it will not redraw until the next publish. That matters for one-off keys (Preview, Source), where the next publish can be 60 s away or never. The window is tiny. Fix: store generation after the current flip (an early read then only causes a harmless extra redraw), or tell Task 6 to key its redraws on the generation get returns.
  • minor (src/bus.rs): a publish racing a remove of the same key may write into the removed slot, so the frame is lost silently. A publish after remove brings the key back. Both are acceptable, but the runtime (Task 21) must stop the producer before it removes a key. A line in the README would help.
  • minor (spec §4 wording): the spec says the reader is "lock-free". The reader uses short RwLock read sections instead. The README states this and it meets Focus 6. No change needed, just noting the deviation.
  • minor, documented deviations, accepted:
    • The "within 2 levels" round trip holds for greys and for the mean. Saturated colours get libyuv's own bounds (R 3, B 5), because exact libyuv matching takes priority.
    • JPEG qualities are integers 70/55/40. The fork makes an extra attempt at 0.4000000000000001.
    • On Linux, the stall test subtracts scheduler time.
  • minor (process): the commit trailer names Opus 5.5, but the repo's CLAUDE.md names Fable 5.2. The integrator should decide which is right.

Verdict: APPROVE. No blocking findings and local CI is green. I would like the generation store-order fix before Task 6 relies on it, but it does not block this PR.

## Independent review: Screen share 1 (fluxer-video), commit 8b8a359 I read the diff `53e3d60..origin/screenshare-1-video-frames` cold. It adds the `fluxer-video` crate and a Cargo.lock entry. **1. Plan tests.** All eight named tests exist and pass. Plain reading of each: - `i420_to_rgba_known_colours` matches libyuv. I checked the blue value by hand: Y41/U239 gives B=251. - `area_downscale_keeps_aspect_and_even_dims` covers 1920x1080, 1366x768, 1001x3001, 4K and 7x5. - `pattern_counter_survives_blur_and_downscale` reads the counter within ±1 after a blur and a downscale. `preview_keeps_the_picture` also checks it after a real JPEG round trip. - `preview_jpeg_respects_size_and_bytes_budget` uses 4K noise and gets 640x360 within 1 000 000 bytes. - `bus_latest_frame_wins` and `bus_remove_forgets_key` pass. - `bus_reader_never_blocks_writer` writes at 1 kHz for 2 s and checks monotonic generations and no stall over 5 ms. **2. Interfaces.** Every name in the Interfaces block exists as written: `VideoFrame`, `FrameData`, `convert::{bgra_to_i420, i420_to_rgba, nv12_to_i420}`, `scale::area_downscale`, `pattern::{TestPattern, read_counter}`, `jpeg::{preview_jpeg, JpegError}`, `VideoKey` and the five `FrameBus` methods. `FrameBus` is `Clone + Send + Sync`. Choices the plan left open: - Conversions return `Result<VideoFrame, FrameError>`. - Strides are `u32`. - Added: `PreviewBudget`, `preview_jpeg_with`, `to_i420` and `to_rgba`. These are fine for Tasks 3, 6, 16, 17 and 21. **3. Invariants.** - The only dependencies are `image` (workspace, `jpeg` already enabled) and `thiserror`. No OS, LiveKit or iced dependency, and no I/O. - Every pixel path calls `validate()` first, so a malformed frame is an `Err` or is passed through unchanged. I found no panic path for decoder or server data. I also checked sizes like 4000x1 and a max of 1 in `area_plane`'s spans. - No secrets. **4. Review Focus 6.** The reader takes no writer mutex. It holds a per-slot read lock only for an `Arc` clone, and frames are dropped outside the locks. One frame per key, and the latest wins. Generations come from a bus-wide counter, so a key that is removed and published again never reuses one. I checked the `Slot::publish`/`get` interleavings, including two publishes during one read. The retry when an emptied slot is no longer current is correct, and generations seen by a reader are monotonic. **5. Local CI.** `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green: "all steps passed", exit 0. A second run I started at the same time on the same target dir failed `xtask --test package` because both runs used the same appimagetool `.partial` path. Run alone, that test passes. It is unrelated to this diff. ### Findings - **minor** (`src/bus.rs`, `Slot::publish`): `generation` is stored *before* `current` flips. During that window, `generation(&key)` can return g+1 while `get(&key)` still returns frame g. Task 6 will compare `generation()`. If a widget records the g+1 from `generation()` but uploads frame g, it will not redraw until the next publish. That matters for one-off keys (`Preview`, `Source`), where the next publish can be 60 s away or never. The window is tiny. Fix: store `generation` after the `current` flip (an early read then only causes a harmless extra redraw), or tell Task 6 to key its redraws on the generation `get` returns. - **minor** (`src/bus.rs`): a `publish` racing a `remove` of the same key may write into the removed slot, so the frame is lost silently. A `publish` after `remove` brings the key back. Both are acceptable, but the runtime (Task 21) must stop the producer before it removes a key. A line in the README would help. - **minor** (spec §4 wording): the spec says the reader is "lock-free". The reader uses short `RwLock` read sections instead. The README states this and it meets Focus 6. No change needed, just noting the deviation. - **minor, documented deviations, accepted:** - The "within 2 levels" round trip holds for greys and for the mean. Saturated colours get libyuv's own bounds (R 3, B 5), because exact libyuv matching takes priority. - JPEG qualities are integers 70/55/40. The fork makes an extra attempt at 0.4000000000000001. - On Linux, the stall test subtracts scheduler time. - **minor (process):** the commit trailer names Opus 5.5, but the repo's `CLAUDE.md` names Fable 5.2. The integrator should decide which is right. **Verdict: APPROVE.** No blocking findings and local CI is green. I would like the `generation` store-order fix before Task 6 relies on it, but it does not block this PR.
charles manually merged commit 69edce3d34 into screenshare 2026-11-12 05:17:14 +00:00
Sign in to join this conversation.
No description provided.