Screen share 6: wgpu video widget (#153) #191

Manually merged
charles merged 0 commits from screenshare-6-ui-video-widget into screenshare 2026-11-12 05:17:18 +00:00
Owner

Stacked on: screenshare-1-video-frames

What this builds (Screen share plan, Task 6; spec §8.2)

  • fluxer_ui::widgets::video: video(&bus, key, Fit::{Contain, Cover}) -> Element<'_, M> draws the latest frame of a FrameBus key, filling its space, letterboxed in --voice-surface-0 (Contain) or cropped (Cover). No frame or an invalid frame shows only the surface, so server data never panics.
    • wgpu: an iced shader primitive. Each key has three R8Unorm textures that are re-uploaded only when the bus generation moves. They are recreated on a size change and dropped once the key leaves the bus. Frames larger than the device's max texture size are downscaled first. video.wgsl does BT.601 limited range with libyuv's own constants, with bilinear sampling. It has an sRGB-target variant.
    • tiny-skia fallback: fluxer_video::{scale, convert} gives RGBA at the drawn size, drawn as image::Handle::from_rgba. It is cached per generation and size.
    • redraw_subscription(&bus, keys) -> Subscription<()> uses window::frames().with(Watch). It emits only when a watched generation moved. Watch is public and tested.
    • placement() is the pure geometry, and upload_count(&key) is the counting hook the tests use.
  • testing::Offscreen::{wgpu, tiny_skia}() keeps a renderer and widget tree across frames. testing::render now wraps it.
  • CLAUDE.md: the FrameBus is the second sanctioned exception, worded like the image one. The fluxer-ui README is updated.
  • spikes §4 is in docs/superpowers/research/2026-11-11-screenshare-spikes.md. The file is new on this base; Task 2's branch creates the same file with §1, so the merge needs both sections.

How it was tested

  • tests/video_widget.rs (cfg(not(windows))):
    • Tests that always run on tiny-skia: placement geometry, missing_key_draws_surface, malformed frames, contain_letterboxes_and_cover_crops, uploads_only_on_new_generation, and Watch.
    • *_wgpu variants, fallback_matches_wgpu_within_tolerance (both paths are under 2 levels mean error against libyuv, and the counter reads back), and upload_1080p_frame_time are #[ignore]d and run with FLUXER_SCREENSHOT_TESTS=1.
  • Screenshot video_widget_pattern_matches_golden is in tests/screenshots.rs, with a row in the docs/screenshots/README.md table. The golden was generated on lavapipe in an ubuntu:22.04 container (as CI does). There all 13 screenshot tests and all 13 video widget tests passed.
  • Ran on: lavapipe (container) and the RTX 5090 host. FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green locally.
  • 1080p I420 upload measured at about 0.5 ms (RTX 5090) and 0.7 ms (lavapipe) per frame, see spikes §4.2.
  • No capture or encoder tests apply to this task.

Deviations from the plan

  • Redraws: window::frames() only fires on RedrawRequested, so a subscription that emits only on a moved generation stalls whenever the video is slower than the display, and nothing wakes it. So the widget also requests a redraw at most 16.7 ms (video::POLL) after each frame while its key has a frame on the bus. With no key it requests nothing, so an idle call does not spin. Spikes §4.5 has the details; a runtime-side notifier is an option for Screen share 21.
  • How the primitive reaches iced: iced_wgpu::primitive::Renderer is not public through iced, so the widget draws an inner iced::widget::Shader on the frame's rectangle. It picks wgpu or the fallback by matching iced::Renderer::{Primary, Secondary}.
  • Files outside the Files list:
    • src/testing/{mod,offscreen}.rs (Offscreen, needed for the multi-frame upload test and the tiny-skia path).
    • tests/screenshots.rs and docs/screenshots/README.md, which screenshot_suite.rs requires for every golden.
    • fixtures/golden/video_widget_pattern.png.
  • Extra tests beyond the plan's list: malformed-frame handling, the pure placement tests, Watch, and the timing measurement.
  • Plan's test names: the plan names a single uploads_only_on_new_generation. It exists on both renderers: the tiny-skia version plus a _wgpu variant.

Closes #153

🤖 Generated with Claude Code

Stacked on: screenshare-1-video-frames ## What this builds (Screen share plan, Task 6; spec §8.2) - `fluxer_ui::widgets::video`: `video(&bus, key, Fit::{Contain, Cover}) -> Element<'_, M>` draws the latest frame of a `FrameBus` key, filling its space, letterboxed in `--voice-surface-0` (Contain) or cropped (Cover). No frame or an invalid frame shows only the surface, so server data never panics. - wgpu: an iced `shader` primitive. Each key has three `R8Unorm` textures that are re-uploaded only when the bus generation moves. They are recreated on a size change and dropped once the key leaves the bus. Frames larger than the device's max texture size are downscaled first. `video.wgsl` does BT.601 limited range with libyuv's own constants, with bilinear sampling. It has an sRGB-target variant. - tiny-skia fallback: `fluxer_video::{scale, convert}` gives RGBA at the drawn size, drawn as `image::Handle::from_rgba`. It is cached per generation and size. - `redraw_subscription(&bus, keys) -> Subscription<()>` uses `window::frames().with(Watch)`. It emits only when a watched generation moved. `Watch` is public and tested. - `placement()` is the pure geometry, and `upload_count(&key)` is the counting hook the tests use. - `testing::Offscreen::{wgpu, tiny_skia}()` keeps a renderer and widget tree across frames. `testing::render` now wraps it. - `CLAUDE.md`: the `FrameBus` is the second sanctioned exception, worded like the image one. The `fluxer-ui` README is updated. - *spikes §4* is in `docs/superpowers/research/2026-11-11-screenshare-spikes.md`. The file is new on this base; Task 2's branch creates the same file with §1, so the merge needs both sections. ## How it was tested - `tests/video_widget.rs` (`cfg(not(windows))`): - Tests that always run on tiny-skia: placement geometry, `missing_key_draws_surface`, malformed frames, `contain_letterboxes_and_cover_crops`, `uploads_only_on_new_generation`, and `Watch`. - `*_wgpu` variants, `fallback_matches_wgpu_within_tolerance` (both paths are under 2 levels mean error against libyuv, and the counter reads back), and `upload_1080p_frame_time` are `#[ignore]`d and run with `FLUXER_SCREENSHOT_TESTS=1`. - Screenshot `video_widget_pattern_matches_golden` is in `tests/screenshots.rs`, with a row in the `docs/screenshots/README.md` table. The golden was generated on lavapipe in an `ubuntu:22.04` container (as CI does). There all 13 screenshot tests and all 13 video widget tests passed. - Ran on: lavapipe (container) and the RTX 5090 host. `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is green locally. - 1080p I420 upload measured at about 0.5 ms (RTX 5090) and 0.7 ms (lavapipe) per frame, see spikes §4.2. - No capture or encoder tests apply to this task. ## Deviations from the plan - **Redraws:** `window::frames()` only fires on RedrawRequested, so a subscription that emits only on a moved generation stalls whenever the video is slower than the display, and nothing wakes it. So the widget also requests a redraw at most 16.7 ms (`video::POLL`) after each frame while its key has a frame on the bus. With no key it requests nothing, so an idle call does not spin. Spikes §4.5 has the details; a runtime-side notifier is an option for Screen share 21. - **How the primitive reaches iced:** `iced_wgpu::primitive::Renderer` is not public through `iced`, so the widget draws an inner `iced::widget::Shader` on the frame's rectangle. It picks wgpu or the fallback by matching `iced::Renderer::{Primary, Secondary}`. - **Files outside the Files list:** - `src/testing/{mod,offscreen}.rs` (`Offscreen`, needed for the multi-frame upload test and the tiny-skia path). - `tests/screenshots.rs` and `docs/screenshots/README.md`, which `screenshot_suite.rs` requires for every golden. - `fixtures/golden/video_widget_pattern.png`. - **Extra tests** beyond the plan's list: malformed-frame handling, the pure placement tests, `Watch`, and the timing measurement. - **Plan's test names:** the plan names a single `uploads_only_on_new_generation`. It exists on both renderers: the tiny-skia version plus a `_wgpu` variant. Closes #153 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(video): frames, conversions, frame bus, test pattern and preview JPEG
All checks were successful
ci / Linux (x86_64) (push) Successful in 15m16s
ci / Windows (x64) (push) Successful in 16m21s
8b8a359082
Refs #148

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(ui): wgpu video widget
All checks were successful
ci / Linux (x86_64) (push) Successful in 36m24s
ci / Windows (x64) (push) Successful in 28m53s
87239806ff
widgets::video draws the latest frame of a FrameBus key: an iced shader
primitive with three R8Unorm planes re-uploaded only on a new generation
and a libyuv BT.601 limited-range WGSL shader, letterboxed (Contain) or
cropped (Cover) in --voice-surface-0, with a tiny-skia RGBA fallback.
redraw_subscription, testing::Offscreen, the FrameBus exception in
CLAUDE.md, spikes §4 and the video_widget_pattern golden (lavapipe).

Refs #153

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Owner

Independent review: Screen share 6 (wgpu video widget), commit 8723980 on 8b8a359

Reviewed git diff 8b8a359..origin/screenshare-6-ui-video-widget cold.

Plan tests (Task 6 Step 1): all present and asserting what the plan says.

  • uploads_only_on_new_generation (tiny-skia + _wgpu): counts 1 upload for the first frame, still 1 after two more draws of the same generation, 2 after a new publish (Review Focus 6, widget half: satisfied).
  • contain_letterboxes_and_cover_crops (both renderers) plus pure placement_* geometry tests.
  • missing_key_draws_surface (both renderers; every pixel is VOICE_SURFACE_0), plus malformed_frame_draws_surface (short planes, 0x0 frame).
  • fallback_matches_wgpu_within_tolerance: both paths < 2 levels mean error against fluxer_video::convert::i420_to_rgba, and the counter reads back 42.
  • Screenshot video_widget_pattern_matches_golden with golden + docs/screenshots/README.md row.

Interfaces: video(&FrameBus, VideoKey, Fit) -> Element<'_, M>, Fit::{Contain, Cover}, redraw_subscription(bus, keys) -> Subscription<()> (window frames, emits only on a moved generation) exist as written. Three R8Unorm planes, written with queue.write_texture only on a new generation, recreated on a size change, dropped in trim once the key leaves the bus; BT.601 limited range in libyuv's fixed point; letterbox in VOICE_SURFACE_0; tiny-skia fallback through scale + convert + Handle::from_rgba.

Invariants: fluxer-ui reads FrameBus only in widgets::video (grep confirms). fluxer-video still depends only on image + thiserror. CLAUDE.md gets the second exception, worded like the image one. No secrets. Server data cannot panic: validate() (stride >= row bytes, len >= stride*(rows-1)+row) runs before any write_texture, so wgpu validation can't fire; invalid or empty frames draw the surface.

CI (review worktree): cargo xtask ci green, and FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green (RTX 5090 host): golden matches, the 6 ignored wgpu video tests pass.

Findings

  1. minor: Redraw deviation (spikes §4.5). While a widget's key has any frame, it requests a redraw every 16.7 ms, even when the frame never changes. Spec §8.2 says to redraw only when a visible key's generation moved, and to send stills (preview thumbnails, picker thumbnails) through this widget. So a visible preview thumbnail or picker grid keeps the whole window redrawing at 60 Hz indefinitely. The reasoning about window::frames() holds, and this does not break any interface. Suggest a follow-up issue for Screen share 21/22/26: a publish-side notifier next to the FrameBus, or an iced::time::every check that emits only on a generation change, so a still does not spin.
  2. minor: UPLOADS (the counter behind upload_count, which only tests use) is a process-global Mutex<HashMap<VideoKey, u64>>. Production updates it on every upload and never removes entries. That is one entry per key ever seen (small, but unbounded over a long session) and a global lock on the hot path. Consider #[cfg(any(test, feature = ...))] or pruning it in trim.
  3. minor (integration note): docs/superpowers/research/2026-11-11-screenshare-spikes.md is created here with §4 only. Task 2's branch creates the same file with §1, so whoever merges second must keep both sections.

Verdict: APPROVED. No blocking findings; ci green locally, including the screenshot step.

## Independent review: Screen share 6 (wgpu video widget), commit 8723980 on 8b8a359 Reviewed `git diff 8b8a359..origin/screenshare-6-ui-video-widget` cold. **Plan tests (Task 6 Step 1):** all present and asserting what the plan says. - `uploads_only_on_new_generation` (tiny-skia + `_wgpu`): counts 1 upload for the first frame, still 1 after two more draws of the same generation, 2 after a new publish (Review Focus 6, widget half: satisfied). - `contain_letterboxes_and_cover_crops` (both renderers) plus pure `placement_*` geometry tests. - `missing_key_draws_surface` (both renderers; every pixel is `VOICE_SURFACE_0`), plus `malformed_frame_draws_surface` (short planes, 0x0 frame). - `fallback_matches_wgpu_within_tolerance`: both paths < 2 levels mean error against `fluxer_video::convert::i420_to_rgba`, and the counter reads back 42. - Screenshot `video_widget_pattern_matches_golden` with golden + `docs/screenshots/README.md` row. **Interfaces:** `video(&FrameBus, VideoKey, Fit) -> Element<'_, M>`, `Fit::{Contain, Cover}`, `redraw_subscription(bus, keys) -> Subscription<()>` (window frames, emits only on a moved generation) exist as written. Three `R8Unorm` planes, written with `queue.write_texture` only on a new generation, recreated on a size change, dropped in `trim` once the key leaves the bus; BT.601 limited range in libyuv's fixed point; letterbox in `VOICE_SURFACE_0`; tiny-skia fallback through `scale` + `convert` + `Handle::from_rgba`. **Invariants:** `fluxer-ui` reads `FrameBus` only in `widgets::video` (grep confirms). `fluxer-video` still depends only on `image` + `thiserror`. CLAUDE.md gets the second exception, worded like the image one. No secrets. Server data cannot panic: `validate()` (stride >= row bytes, len >= stride*(rows-1)+row) runs before any `write_texture`, so wgpu validation can't fire; invalid or empty frames draw the surface. **CI (review worktree):** `cargo xtask ci` green, and `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green (RTX 5090 host): golden matches, the 6 ignored wgpu video tests pass. ### Findings 1. **minor**: Redraw deviation (spikes §4.5). While a widget's key has *any* frame, it requests a redraw every 16.7 ms, even when the frame never changes. Spec §8.2 says to redraw only when a visible key's generation moved, and to send stills (preview thumbnails, picker thumbnails) through this widget. So a visible preview thumbnail or picker grid keeps the whole window redrawing at 60 Hz indefinitely. The reasoning about `window::frames()` holds, and this does not break any interface. Suggest a `follow-up` issue for Screen share 21/22/26: a publish-side notifier next to the `FrameBus`, or an `iced::time::every` check that emits only on a generation change, so a still does not spin. 2. **minor**: `UPLOADS` (the counter behind `upload_count`, which only tests use) is a process-global `Mutex<HashMap<VideoKey, u64>>`. Production updates it on every upload and never removes entries. That is one entry per key ever seen (small, but unbounded over a long session) and a global lock on the hot path. Consider `#[cfg(any(test, feature = ...))]` or pruning it in `trim`. 3. **minor (integration note)**: `docs/superpowers/research/2026-11-11-screenshare-spikes.md` is created here with §4 only. Task 2's branch creates the same file with §1, so whoever merges second must keep both sections. Verdict: **APPROVED**. No blocking findings; ci green locally, including the screenshot step.
charles manually merged commit 69edce3d34 into screenshare 2026-11-12 05:17:17 +00:00
Sign in to join this conversation.
No description provided.