Video 64: fluxer-media-playback crate: demux, decode, A/V sync, seeking (FFmpeg through rsmpeg) #404

Closed
opened 2026-11-13 02:50:45 +00:00 by charles · 0 comments
Owner

Plan: docs/superpowers/plans/2026-11-12-textchat.md, section "Task 64" (added 2026-11-13 at Charles's request: rename the client to Flaxative; in-client video playback with bundled LGPL FFmpeg, spec §19).

Task 64: fluxer-media-playback crate: demux, decode, A/V sync, seeking (FFmpeg through rsmpeg)

Issue: (to be created)
Closes: none.

Touches fluxer-video (one enum variant, first commit) and the new crate — serialised.

Files:

  • Create: crates/fluxer-media-playback/{Cargo.toml,README.md,build.rs,src/{lib.rs,source.rs,avio.rs,demux.rs,video.rs,audio.rs,clock.rs,hw.rs,player.rs,error.rs}}, crates/fluxer-media-playback/tests/{player.rs,decode.rs,support/clips.rs}, crates/fluxer-media-playback/tests/fixtures/{h264_aac_64x64.mp4,vp9_opus_64x64.webm,av1_64x64.webm} (≤ 20 KB each), tools/ffmpeg/make-fixtures.sh (the exact commands that produced them, with any ffmpeg; the fixtures are data, no GPL code ships)
  • Modify: crates/fluxer-video/src/{bus.rs,lib.rs}, crates/fluxer-video/README.md; workspace Cargo.toml (member, rsmpeg pinned with =), deny.toml only if a new licence appears (none expected: rsmpeg and rusty_ffmpeg are MIT, bindgen is already in the graph)
  • Test: as above, plus crates/fluxer-video/src/bus.rs (unit)

Interfaces:

  • Produces (fluxer-video): VideoKey::Media { id: u64 } — a playing video attachment, video embed or GIFV, by the player id the store hands out.
  • Produces (crate, depends on rsmpeg, fluxer-video, thiserror, tracing; not on fluxer-audio, tokio or reqwest):
    • trait ByteSource: Send + Sync { fn len(&self) -> Option<u64>; fn read_at(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize>; } — blocking, called only from the player's demux thread; MemorySource(Arc<[u8]>) implements it. avio.rs wraps it in a custom AVIOContext (read + seek, AVSEEK_SIZE from len), 64 KiB buffer.
    • trait AudioOut: Send + Sync { fn push(&self, interleaved_stereo_48k: &[i16]) -> bool /* false = full, retry later */; fn played(&self) -> Duration; fn queued(&self) -> Duration; fn pause(&self, paused: bool); fn flush(&self); } — Task 65's MediaAudio fits it; the app adapts.
    • type VideoOut = Box<dyn Fn(fluxer_video::VideoFrame) + Send> (the app publishes on the FrameBus).
    • OpenOptions { autoplay: bool, looping: bool, start: Duration, rate: f32, hardware: HwPreference, max_size: (u32, u32) /* default 1920×1080 */ }, HwPreference::{Auto, Off}, HwKind::{Vaapi, Nvdec, D3d11va}.
    • Player::open(source: Arc<dyn ByteSource>, options: OpenOptions, video: VideoOut, audio: Option<Arc<dyn AudioOut>>, events: std::sync::mpsc::Sender<PlayerEvent>) -> Player (returns at once; a demux thread and a video thread do the work); Player::{play(&self), pause(&self), seek(&self, to: Duration), set_looping(&self, bool), set_rate(&self, rate: f32 /* 0.75, 1, 1.5, 2 */)}; Drop stops and joins the threads within 200 ms.
    • PlayerEvent::{Opened(MediaInfo), Position(Duration) /* at most 4 per second and after every seek */, Buffering(bool), Ended, Error(PlaybackError)}; MediaInfo { duration: Option<Duration>, width: u32, height: u32, has_video: bool, has_audio: bool, video_codec: String, hardware: Option<HwKind> }; PlaybackError::{Open(String), Unsupported(String), Io(String), Decode(String)} (no URL in any string).
    • fn ffmpeg_info() -> FfmpegInfo { avcodec: String, avformat: String, avutil: String, license: String }, fn hardware_decoders() -> Vec<HwKind> (probes device creation; never panics, empty without a GPU) — for --debug-info.
  • Rules: video decoded with the first HwKind whose av_hwdevice_ctx_create and codec hw config succeed (Linux: VAAPI, then NVDEC; Windows: D3D11VA, then NVDEC), else software (threads = min(4, cores))

🤖 Generated with Claude Code

Plan: `docs/superpowers/plans/2026-11-12-textchat.md`, section "Task 64" (added 2026-11-13 at Charles's request: rename the client to Flaxative; in-client video playback with bundled LGPL FFmpeg, spec §19). ### Task 64: `fluxer-media-playback` crate: demux, decode, A/V sync, seeking (FFmpeg through `rsmpeg`) **Issue:** (to be created) **Closes:** none. Touches `fluxer-video` (one enum variant, first commit) and the new crate — serialised. **Files:** - Create: `crates/fluxer-media-playback/{Cargo.toml,README.md,build.rs,src/{lib.rs,source.rs,avio.rs,demux.rs,video.rs,audio.rs,clock.rs,hw.rs,player.rs,error.rs}}`, `crates/fluxer-media-playback/tests/{player.rs,decode.rs,support/clips.rs}`, `crates/fluxer-media-playback/tests/fixtures/{h264_aac_64x64.mp4,vp9_opus_64x64.webm,av1_64x64.webm}` (≤ 20 KB each), `tools/ffmpeg/make-fixtures.sh` (the exact commands that produced them, with any ffmpeg; the fixtures are data, no GPL code ships) - Modify: `crates/fluxer-video/src/{bus.rs,lib.rs}`, `crates/fluxer-video/README.md`; workspace `Cargo.toml` (member, `rsmpeg` pinned with `=`), `deny.toml` only if a new licence appears (none expected: `rsmpeg` and `rusty_ffmpeg` are MIT, `bindgen` is already in the graph) - Test: as above, plus `crates/fluxer-video/src/bus.rs` (unit) **Interfaces:** - Produces (`fluxer-video`): `VideoKey::Media { id: u64 }` — a playing video attachment, video embed or GIFV, by the player id the store hands out. - Produces (crate, depends on `rsmpeg`, `fluxer-video`, `thiserror`, `tracing`; not on `fluxer-audio`, `tokio` or `reqwest`): - `trait ByteSource: Send + Sync { fn len(&self) -> Option<u64>; fn read_at(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize>; }` — blocking, called only from the player's demux thread; `MemorySource(Arc<[u8]>)` implements it. `avio.rs` wraps it in a custom `AVIOContext` (read + seek, `AVSEEK_SIZE` from `len`), 64 KiB buffer. - `trait AudioOut: Send + Sync { fn push(&self, interleaved_stereo_48k: &[i16]) -> bool /* false = full, retry later */; fn played(&self) -> Duration; fn queued(&self) -> Duration; fn pause(&self, paused: bool); fn flush(&self); }` — Task 65's `MediaAudio` fits it; the app adapts. - `type VideoOut = Box<dyn Fn(fluxer_video::VideoFrame) + Send>` (the app publishes on the `FrameBus`). - `OpenOptions { autoplay: bool, looping: bool, start: Duration, rate: f32, hardware: HwPreference, max_size: (u32, u32) /* default 1920×1080 */ }`, `HwPreference::{Auto, Off}`, `HwKind::{Vaapi, Nvdec, D3d11va}`. - `Player::open(source: Arc<dyn ByteSource>, options: OpenOptions, video: VideoOut, audio: Option<Arc<dyn AudioOut>>, events: std::sync::mpsc::Sender<PlayerEvent>) -> Player` (returns at once; a demux thread and a video thread do the work); `Player::{play(&self), pause(&self), seek(&self, to: Duration), set_looping(&self, bool), set_rate(&self, rate: f32 /* 0.75, 1, 1.5, 2 */)}`; `Drop` stops and joins the threads within 200 ms. - `PlayerEvent::{Opened(MediaInfo), Position(Duration) /* at most 4 per second and after every seek */, Buffering(bool), Ended, Error(PlaybackError)}`; `MediaInfo { duration: Option<Duration>, width: u32, height: u32, has_video: bool, has_audio: bool, video_codec: String, hardware: Option<HwKind> }`; `PlaybackError::{Open(String), Unsupported(String), Io(String), Decode(String)}` (no URL in any string). - `fn ffmpeg_info() -> FfmpegInfo { avcodec: String, avformat: String, avutil: String, license: String }`, `fn hardware_decoders() -> Vec<HwKind>` (probes device creation; never panics, empty without a GPU) — for `--debug-info`. - Rules: **video** decoded with the first `HwKind` whose `av_hwdevice_ctx_create` and codec hw config succeed (Linux: VAAPI, then NVDEC; Windows: D3D11VA, then NVDEC), else software (threads = min(4, cores)) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
charles added this to the Text chat milestone 2026-11-13 02:50:45 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#404
No description provided.