Camera 4: fluxer-video camera formats and the camera pattern (#956) #977

Closed
charles wants to merge 0 commits from camera-4-video-camera-formats-and into camera
Owner

What was built (Camera plan, Task 4; spec §5.1, §7.4)

  • convert::yuyv_to_i420(width, height, stride, &[u8], timestamp_us): packed 4:2:2 to I420, chroma rows averaged with rounding as libyuv's YUY2ToI420, odd width/height handled, size and stride checked against the buffer (u64 arithmetic), errors never panics.
  • convert::mjpeg_to_i420(&[u8], timestamp_us): zune-jpeg 0.5.15 to YCbCr, 2x2 mean to 4:2:0, full range to BT.601 limited in the same pass. 4:2:2, 4:2:0, 4:4:4 and greyscale sources. Frames without DHT (as UVC cameras send them) get the ITU T.81 Annex K tables inserted before the first SOS, whether or not they carry the AVI1 marker. Size capped at 7680x4320; a decoder panic is caught and reported as an error.
  • convert::full_to_limited_range(&mut VideoFrame): I420 and NV12 (a shared plane is copied first); packed RGB left unchanged.
  • convert::PREFER_UNCOMPRESSED: bool = false: the measurement result (see below).
  • FrameError::Undecodable { reason } (new variant: the Result<VideoFrame, FrameError> signature of the interface needs it).
  • VideoKey::LocalCamera, appended.
  • TestPattern::camera(w, h): the same counter row over a dark ground with a face-sized oval in the middle and a red block on the left only. read_counter works on it; pattern::is_mirrored(frame) -> Option<bool> answers Some(false)/Some(true), and None for the screen-share bars, malformed frames and frames under 64x32. It still works after to_rgba and area_downscale.
  • Fixtures fixtures/media/camera_frame_422.jpg and camera_frame_420.jpg: frame 5 of TestPattern::camera(1920, 1080), encoded by FFmpeg's MJPEG encoder with -huffman default. They are generated, not photographs. The command is in the crate README. examples/pattern_yuv.rs gained a camera argument to produce them.
  • zune-jpeg = "0.5" is a workspace dependency and a direct dependency of fluxer-video. It was already in the tree through image, so Cargo.lock gains no new package. proptest is a dev-dependency.

Decision (spikes §4)

mjpeg_to_i420 at 1080p on one core, release build, 60 runs (mean):

Input Size Mean time
pattern fixtures 54-64 kB 2.7-2.8 ms
pattern + noise ±6, q85 (typical camera frame) 244 kB 5.1 ms
pattern + noise ±20, q85 (heavy camera frame) 618 kB 7.3 ms

The conversion after the decoder costs about 0.9 ms. All cases are under 8 ms, so PREFER_UNCOMPRESSED is false and no follow-up issue was opened.

Deviation: these numbers come from this machine (Ryzen 9 9950X, under load), not from the Linux runner. The runner does not run #[ignore] tests, and this task does not wait on CI. A runner core about 1.5x slower would put the heavy case over 8 ms. To confirm, run cargo test --release -p fluxer-video --test camera_formats -- --ignored --nocapture measure on the runner. If the ±6 row goes over 8 ms there, flip the constant and open the FFmpeg follow-up (spikes §4 says this).

Tests

crates/fluxer-video/tests/camera_formats.rs:

  • yuyv_known_pixels
  • yuyv_odd_stride
  • mjpeg_422_and_420_match_pattern (mean error under 1 level per plane in limited range; measured 0.03 to 0.09)
  • mjpeg_without_huffman_tables_decodes (the stripped frame decodes bit-identical to the original)
  • mjpeg_garbage_is_an_error_not_a_panic (proptest, 256 cases)
  • mjpeg_garbage_is_an_error_not_a_panic_on_fixture_damage
  • range_pass_maps_0_255_to_16_235
  • camera_pattern_is_asymmetric_and_counts
  • local_camera_key_round_trips_on_bus
  • measure_mjpeg_1080p (#[ignore], prints the timings)

FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally. No camera, display or device was opened, and no FLUXER_DESKTOP_TESTS test applies to this task. No frame is written or logged: the fixtures are the generated pattern.

For the reviewer

  • zune-jpeg decodes a truncated frame as Ok, with the missing blocks filled in. The damage test only asserts that it does not panic. Camera 6/10 should drop frames whose bytesused is short, if they want truncated frames dropped instead of shown.
  • is_mirrored samples three boxes: the oval centre, the marker and its mirror position. The palette distance threshold is 48.

Closes #956

🤖 Generated with Claude Code

https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K

## What was built (Camera plan, Task 4; spec §5.1, §7.4) - `convert::yuyv_to_i420(width, height, stride, &[u8], timestamp_us)`: packed 4:2:2 to I420, chroma rows averaged with rounding as libyuv's `YUY2ToI420`, odd width/height handled, size and stride checked against the buffer (u64 arithmetic), errors never panics. - `convert::mjpeg_to_i420(&[u8], timestamp_us)`: `zune-jpeg` 0.5.15 to YCbCr, 2x2 mean to 4:2:0, full range to BT.601 limited in the same pass. 4:2:2, 4:2:0, 4:4:4 and greyscale sources. Frames without DHT (as UVC cameras send them) get the ITU T.81 Annex K tables inserted before the first SOS, whether or not they carry the `AVI1` marker. Size capped at 7680x4320; a decoder panic is caught and reported as an error. - `convert::full_to_limited_range(&mut VideoFrame)`: I420 and NV12 (a shared plane is copied first); packed RGB left unchanged. - `convert::PREFER_UNCOMPRESSED: bool = false`: the measurement result (see below). - `FrameError::Undecodable { reason }` (new variant: the `Result<VideoFrame, FrameError>` signature of the interface needs it). - `VideoKey::LocalCamera`, appended. - `TestPattern::camera(w, h)`: the same counter row over a dark ground with a face-sized oval in the middle and a red block on the **left only**. `read_counter` works on it; `pattern::is_mirrored(frame) -> Option<bool>` answers `Some(false)`/`Some(true)`, and `None` for the screen-share bars, malformed frames and frames under 64x32. It still works after `to_rgba` and `area_downscale`. - Fixtures `fixtures/media/camera_frame_422.jpg` and `camera_frame_420.jpg`: frame 5 of `TestPattern::camera(1920, 1080)`, encoded by FFmpeg's MJPEG encoder with `-huffman default`. They are generated, not photographs. The command is in the crate README. `examples/pattern_yuv.rs` gained a `camera` argument to produce them. - `zune-jpeg = "0.5"` is a workspace dependency and a direct dependency of `fluxer-video`. It was already in the tree through `image`, so `Cargo.lock` gains no new package. `proptest` is a dev-dependency. ## Decision (spikes §4) `mjpeg_to_i420` at 1080p on one core, release build, 60 runs (mean): | Input | Size | Mean time | |---|---|---| | pattern fixtures | 54-64 kB | 2.7-2.8 ms | | pattern + noise ±6, q85 (typical camera frame) | 244 kB | 5.1 ms | | pattern + noise ±20, q85 (heavy camera frame) | 618 kB | 7.3 ms | The conversion after the decoder costs about 0.9 ms. All cases are under 8 ms, so `PREFER_UNCOMPRESSED` is false and no follow-up issue was opened. **Deviation: these numbers come from this machine (Ryzen 9 9950X, under load), not from the Linux runner.** The runner does not run `#[ignore]` tests, and this task does not wait on CI. A runner core about 1.5x slower would put the heavy case over 8 ms. To confirm, run `cargo test --release -p fluxer-video --test camera_formats -- --ignored --nocapture measure` on the runner. If the ±6 row goes over 8 ms there, flip the constant and open the FFmpeg follow-up (spikes §4 says this). ## Tests `crates/fluxer-video/tests/camera_formats.rs`: - `yuyv_known_pixels` - `yuyv_odd_stride` - `mjpeg_422_and_420_match_pattern` (mean error under 1 level per plane in limited range; measured 0.03 to 0.09) - `mjpeg_without_huffman_tables_decodes` (the stripped frame decodes bit-identical to the original) - `mjpeg_garbage_is_an_error_not_a_panic` (proptest, 256 cases) - `mjpeg_garbage_is_an_error_not_a_panic_on_fixture_damage` - `range_pass_maps_0_255_to_16_235` - `camera_pattern_is_asymmetric_and_counts` - `local_camera_key_round_trips_on_bus` - `measure_mjpeg_1080p` (`#[ignore]`, prints the timings) `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. No camera, display or device was opened, and no `FLUXER_DESKTOP_TESTS` test applies to this task. No frame is written or logged: the fixtures are the generated pattern. ## For the reviewer - `zune-jpeg` decodes a **truncated** frame as `Ok`, with the missing blocks filled in. The damage test only asserts that it does not panic. Camera 6/10 should drop frames whose `bytesused` is short, if they want truncated frames dropped instead of shown. - `is_mirrored` samples three boxes: the oval centre, the marker and its mirror position. The palette distance threshold is 48. Closes #956 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
convert: yuyv_to_i420 (packed 4:2:2, chroma rows averaged as libyuv's
YUY2ToI420), mjpeg_to_i420 (zune-jpeg to YCbCr, 4:2:0, limited range;
frames without Huffman tables get the Annex K tables; never panics) and
full_to_limited_range; PREFER_UNCOMPRESSED = false from the measurement
of spikes §4. FrameError::Undecodable. VideoKey::LocalCamera appended.
TestPattern:📷 the counter row over an oval and a left-only marker,
with pattern::is_mirrored for tests. zune-jpeg 0.5 becomes a direct
dependency (already in the tree through image).

Fixtures camera_frame_422.jpg / camera_frame_420.jpg: frame 5 of
TestPattern::camera(1920, 1080), encoded by FFmpeg's MJPEG encoder with
the standard Huffman tables (command in the crate README); generated,
not photographs.

Refs #956

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
mjpeg_to_i420 at 1080p on one core: 2.7-2.8 ms on the pattern fixtures,
5.1 ms on a typical (240 kB) and 7.3 ms on a heavy (600 kB) frame, measured
here, not on the runner; under the 8 ms budget, so PREFER_UNCOMPRESSED
stays false and no follow-up is opened. How to confirm on the runner.

Refs #956

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Author
Owner

Merged into camera by hand (the merge API returns 405); the commits are in the branch.

Merged into `camera` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-19 23:15:50 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.