Text chat 64: fluxer-media-playback crate: demux, decode, A/V sync, seeking (#404) #460
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "textchat-64-media-playback-crate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this builds
fluxer-video:VideoKey::Media { id: u64 }(first commit,media_key_is_distinct_from_other_keys).crates/fluxer-media-playback(spec §19.2/§19.3), with the Interfaces of plan Task 64:ByteSource+MemorySource, a customAVIOContext(read/seek,AVSEEK_SIZEfromlen, 64 KiB, no FFmpeg protocol),AudioOut,VideoOut,OpenOptions,HwPreference/HwKind,Player::{open, play, pause, seek, set_looping, set_rate}+Drop(joins within 200 ms; detaches only a thread stuck insideread_at),PlayerEvent,MediaInfo,PlaybackError(no URL in any string; I/O errors keep only their kind),ffmpeg_info(),hardware_decoders(),MAX_PLAYERS = 8.max_sizekeeping aspect; BT.601 limited output; audio to 48 kHz stereo i16 throughatempowhen rate != 1; audio master clock, else a monotonic clock that stops while paused (injectable with featuretesting:clock::ManualTime,Player::open_with_time); publish at <= clock + 10 ms, drop at < clock - 40 ms, at most 2 held frames; demuxer stops at 2 s of queued audio or 32 video packets; exact seek (keyframe, discard to target,AudioOut::flush,Position(target)); looping withoutEnded;Buffering(true)after a 250 ms blocked read.tests/support/clips.rsmake_clip(ClipSpec { width, height, fps, frames, gop, audio_hz }): TestPattern frames + a tone, MPEG-4 Part 2 + AAC in MP4, in memory with the bundled LGPL encoders (plusfaststartand a flat-colour clip). Fixturesh264_aac_64x64.mp4(7.9 KB),vp9_opus_64x64.webm(6.6 KB),av1_64x64.webm(2.7 KB) made bytools/ffmpeg/make-fixtures.sh.How it was tested
FLUXER_SCREENSHOT_TESTS=1 cargo xtask cigreen locally (Linux, CachyOS, clang 23).tests/player.rs(19),tests/decode.rs(4), unit tests (clock, source, fit, atempo).FLUXER_HW_DECODER_TESTS=1 cargo test -p fluxer-media-playback --test hwranvaapi_decodes_h264_fixtureon this desktop (VAAPI) and passed.d3d11va_decodes_h264_fixturewas not run (no Windows machine here). CI does not depend on either.rusty_ffmpegagainst the MSVC import libraries (FFMPEG_DLL_PATH), and it needs libclang for bindgen (the FFmpeg archive ships no pre-generatedbinding.rs). Please watch that job.Deviations from the plan
rsmpeg0.18.0 andrusty_ffmpeg0.16.7 underthird_party/([patch.crates-io], same layout aslivekit/,PATCHES.mdin each,third_party/README.mdupdated), outside the task's Files list, because neither published crate builds here:--disable-avdevice(Task 63's flags) butrusty_ffmpeglinks/binds libavdevice unconditionally andrsmpeghas anavdevicemodule, so both drop it;rusty_ffmpegasks for bindgen 0.71, which with libclang >= 22 makesAVFormatContext& co. opaque; it now uses bindgen 0.72.1, already in the graph forpipewire(so the lockfile gains no second bindgen).Follow-up #458 tracks dropping the patch (FFmpeg r2 with an empty avdevice, or upstream). New crates in the graph via rsmpeg:
bon,darling 0.24,camino,prettyplease,paste(cargo deny green).Error(Open); a faststart MP4 cut short plays what it has and may end withEnded(the demuxer sees a clean EOF). The test accepts either for that case.fluxer_video::convert::bgra_to_i420(libyuv-exact BT.601 limited), not byswscalealone, so the result matches the shader exactly; BT.601 sources go straight to I420. Costs a second pass on HD video.Endedleaves the player paused at the end (browser behaviour);playrestarts from 0, a seek shows that frame.tests/support/mod.rs(harness, fakeAudioOut, counting/slow sources) was added next toclips.rs; tests that open players run one at a time per binary becauseMAX_PLAYERSis process-wide.send_packetdropping a packet), both fixed.Stacked on: textchat-63-ffmpeg-lgpl-packaging
Closes #404
🤖 Generated with Claude Code
VideoKey::Media { id } for a playing video attachment, video embed or GIFV. Refs #404 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Independent review: Text chat 64 (
fluxer-media-playback)Reviewed
978245c..ae3c1f7cold (the fluxer-video commit, the vendoring commits, the crate).Checked and fine
seek_lands_on_the_exact_frameis 30 fps, gop 30, 1.5 s → counter 45;bt709_input_comes_out_bt601_limitedis ±2 against libyuv BT.601 limited;ninth_player_is_refusedchecks the exactUnsupported("too many players"); the hw tests are gated onFLUXER_HW_DECODER_TESTS=1).media_key_is_distinct_from_other_keysis influxer-video/src/bus.rs.VideoKey::Media { id: u64 },ByteSource/MemorySource,AudioOut,VideoOut,OpenOptions(default 1920×1080),HwPreference,HwKind,Player::{open, play, pause, seek, set_looping, set_rate}+Drop,PlayerEvent,MediaInfo,PlaybackError,ffmpeg_info()/FfmpegInfo,hardware_decoders(),MAX_PLAYERSall match the plan exactly.make_clip(ClipSpec { width, height, fps, frames, gop, audio_hz })matches.rsmpeg,fluxer-video,thiserrorandtracing(nofluxer-audio,tokioorreqwest). No change tofluxer-core,fluxer-uiorfluxer-emoji.PlaybackErrorstrings carry only FFmpeg error text or anio::ErrorKind, never a URL. FFmpeg callbacks are wrapped incatch_unwind. Server data cannot reach a panic path:expects are only on checked invariants, and sizes are bounded bymax_size. Fixtures are 2.7/6.6/7.9 KB and made by the script.rsmpeg,rusty_ffmpeg,bon,darling,camino,prettyplease.cargo denyis green. The vendoring is documented (PATCHES.md,third_party/README.md, follow-up #458).cargo xtask ciin a fresh review worktree: all steps passed (Linux). Media-playback results: 4 unit tests, decode 4/4, player 19/19. hw 1/1:vaapi_decodes_h264_fixtureran and passed on this desktop.Findings
rusty_ffmpeg's bindgen.binding.rsreused throughFFMPEG_BINDING_PATH(so it needs no libclang), or a pinned LLVM withLIBCLANG_PATH.tools/ffmpeg/build.shputs nobinding.rsin the win-x64 archive, soxtask ffmpeg envnever setsFFMPEG_BINDING_PATH. Thewindowsjob inci.yamlinstalls git, rustup, cargo-deny and .NET, but no LLVM.cargo test --workspaceon the Windows runner will therefore fail inrusty_ffmpeg's build script unless that machine happens to have libclang. The PR itself says Windows was never built.binding.rsin the Linux job (or ship it in the archive) and pointFFMPEG_BINDING_PATHat it on Windows, or add a pinned LLVM to the Windows bootstrap. If the Windows job turns out green as it stands, downgrade this finding and record why libclang was there.Endedinstead ofError.Error", andtruncated_file_is_an_error_not_a_panicaccepts either result for this case. There is no panic, but the store will treat a broken file as fully played.durationat EOF, or open a follow-up issue.Tempo::processreadsdata[0]as interleavednb_samples*2values. It is correct only because format negotiation happens to keep packed s16.aformat=sample_fmts=s16:channel_layouts=stereobeforeabuffersinkmakes that guaranteed.aformatis already in the configure flags.MediaInfo.hardwarecan report a hardware decoder that is not in use.Some(kind)whenever the hardware context opened. Whenget_formatlater falls back to a software format,--debug-info/UI still claim hardware.autoplay: falseshows no frame untilplayorseek.seek(Duration::ZERO)to get one. The runtime must do the same if it wants a first frame instead of the poster.audio: Noneends at once withEndedand noError.AudioOut, it reads to EOF and reportsEnded. The store should not open it that way.Verdict: changes requested. The only blocker is finding 1, the Windows bindgen/libclang path. Everything else matches the plan, and local CI is green.
Review round 1 (commits
d0dba6b,336aeea): what changed for each finding.Blocking: the Windows CI cannot run rusty_ffmpeg's bindgen (
.forgejo/workflows/ci.yaml): fixed by shipping pre-generated bindings, so the Windows job needs no libclang.tools/ffmpeg/bindings/8.0.3-r1-win-x64.rsholds the bindings of the Windows archive's headers, generated on Linux by the newtools/ffmpeg/gen-binding.sh. The script runs clang for the MSVC target, with the mingw-w64 C headers standing in for the Windows SDK and-fgnuc-version=4.9. That gives 32-bitlong, achar *va_listandintenums. rsmpeg relies on theintenums (av_parser_init): a first try with the windows-gnu target producedunsignedenums, and rsmpeg failed to compile against them. The script then type-checks rusty_ffmpeg forx86_64-pc-windows-msvc.cargo xtask ffmpeg fetch(and soci,envandpackage, which call it) now installs the committed bindings for the target as the install'sbinding.rs, copying only when the bytes changed. The existingenv_varsthen setsFFMPEG_BINDING_PATH. The workflow is unchanged.PATCHES.md):build.rsnow emitscargo:rerun-if-changed=<FFMPEG_BINDING_PATH>. Without it, a regenerated binding at the same path was ignored by a warm target dir, and the Windows runner's target dir persists. I hit this locally.install_binding_copies_only_when_it_changedandcommitted_windows_binding_matches_the_lock. The second checks that the file exists for the locked build, has the Windowsva_listand has MSVCintenums. It fails whenffmpeg.lockis bumped without regenerating the file.env --target windowsvariables and no libclang involvement,cargo checkandcargo clippy -D warningsoffluxer-media-playback --tests --target x86_64-pc-windows-msvcpass. Clippy flagged one cast that is a no-op only on Windows (hw.rs), now allowed with a comment. Nothing was built or run on the Windows runner itself: linking and the tests there are still unverified until the integrator's run.Minor: a truncated faststart MP4 ends in
Ended(tests/player.rs): fixed, with the test tightened first. At EOF,Demuxer::readnow checks the played streams' index entries (avformat_index_get_entry). If any entry points past the source's length, the read givesError(Decode("the file is cut short")). The test now requiresError(Decode)for the cut faststart file andEndedfor the whole one.Minor:
Tempo::processwithoutaformat(audio.rs): fixed. The graph is nowabuffer → atempo → aformat(sample_fmts=s16:channel_layouts=stereo:sample_rates=48000) → abuffersink.processalso skips any frame that is not 2-channel s16 instead of reading it. New test:tempo_sink_is_pinned_to_interleaved_stereo_s16(it failed before the fix).Minor:
MediaInfo.hardwarereportsSomeafter a software fallback (video.rs): deferred to #464 (follow-up).Openedis sent before anything is decoded. A correct value needs eitherOpenedto wait for the first frame or a new event, which is an interface choice for Task 67's consumer. The field only feeds--debug-info.Minor: no first frame with
autoplay=false; audio-only withoutAudioOutends withEnded(player.rs): both fixed, with the tests written first.<video>does. Task 67 no longer needsseek(0). New test:paused_open_shows_the_first_frame_only(shows frame 0, no more frames while paused, no duplicate afterplay).seek_lands_on_the_exact_framenow expects[0, 45].AudioOutis nowError(Unsupported("audio-only media needs an audio output"))beforeOpened. New test:audio_only_without_an_output_is_an_error, using an in-memory WAV. With an output, the same file opens.READMEs updated: the crate's,
tools/ffmpeg/README.mdandxtask/README.md. LocalFLUXER_SCREENSHOT_TESTS=1 cargo xtask ciis green.Independent review: Text chat 64 (round 2, head
336aeea)Scope:
git diff 978245c..origin/textchat-64-media-playback-crate. I read the code without relying on the implementer's notes.Checked and OK
tests/hw.rstests exist, and they assert the plan's values: seek 1.5 s lands on counter 45 (gop 30, 30 fps); the +10 ms / −40 ms window shows as 15 frames at 0.5 s and only 44–45 after a jump to 1.5 s; rate 2 reaches frame 30 at 0.5 s of wall time; BT.709 and BT.601 patches match the expected value within ±2; a 64×48 clip is downscaled to 32×24; the 9th player getsUnsupported("too many players"); drop takes under 200 ms and releases theAudioOut; LGPL licence. Each fixture is 20 KB or smaller, andmake-fixtures.shholds the commands that made them.VideoKey::Media { id: u64 },ByteSource,MemorySource,AudioOut,VideoOut,OpenOptions(default 1920×1080),HwPreference,HwKind,Player::openplus play, pause, seek, set_looping, set_rate,PlayerEvent,MediaInfo,PlaybackError,ffmpeg_info()/FfmpegInfo,hardware_decoders()andMAX_PLAYERSmatch the plan exactly. The crate's dependencies are only rsmpeg, fluxer-video, thiserror and tracing.PATCHES.md.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciin a clean review worktree: all steps passed (rc=0).Findings
demux::to_durationcallsDuration::from_secs_f64(ts * num / den), which panics when the value is more than u64::MAX seconds (about 1.8e19). Bothtsand the streamtime_basecome from the file. In Matroska, FFmpeg sets the time base toTimestampScale·TrackTimestampScale / 1e9, with the numerator truncated to 32 bits, so num/den can be about 4. A crafted WebM with 64-bit cluster timecodes near i64::MAX then reachesto_durationfromVideoThread::media_time,Demuxer::media_time(audio trim) and the container duration, and panics. The thread dies without anErrorevent, so the store sees a player that opened and then stalls. The same applies toat + Duration::from_millis(1)intake_frameswhenatis nearDuration::MAX. Fix: useDuration::try_from_secs_f64(..).ok()(or clamp), usechecked_add/saturating_addfor the frame arithmetic, and add a table test withto_duration(i64::MAX, {num: 4, den: 1})and similar.should_readkeeps reading past 32 video packets while less than 200 ms of audio is buffered. Once the audio stream is exhausted before EOF, nothing refills it, so the demuxer reads the rest of the file into the video channel. Also, the clock stays onAudioOut::played(), which no longer advances, so video freezes until EOF switches the clock to wall time. This is harmless for small files, but a long video with a short sound track buffers its whole remainder in memory. Suggestion: treat "audio stream gave nothing for N packets" (or audio EOS) as no audio for the margin rule, and switch to wall time then.audio.rs,skip.as_micros() as u64 * VALUES_PER_SECONDoverflows whenskipis above about 1.9e8 s, which needs a file that claims an absurd duration and a seek into it. It panics in debug builds and wraps in release. Usesaturating_mul, or compute fromas_secs_f64.MediaInfo.hardwarecan say hardware whenget_formatsilently fell back to software. This is tracked as #464; the deferral is fine.cargo check/clippy for x86_64-pc-windows-msvc, never linked or run on the Windows runner. The integrator should watch the Windows job after merge, as the implementer says.Verdict: changes requested. One blocking finding (1); ci is green locally.
Review round 2, addressed in
041dda0:Blocking:
to_durationpanic (demux.rs).to_durationnow goes throughDuration::try_from_secs_f64and clamps to a newMAX_MEDIA_TIME(i64::MAXµs, about 292,000 years), so an absurd timestamp or time base yields a capped time instead of a panic. The cap also keeps every lateras_micros()→i64conversion and addition in range. In addition,Demux::seekclamps its target toMAX_MEDIA_TIME.VideoThread::take_frames(at + 1 ms) andpresent(now + EARLY,at + LATE) usesaturating_add.PlaybackClock::position_ofreplacesmul_f64(which panics on overflow) withtry_from_secs_f64andsaturating_add. Test first: the table testdemux::tests::to_duration_never_panics_and_stays_in_rangepanicked insidefrom_secs_f64before the fix. It covers NOPTS, a zero denominator, negative values,i64::MAXwith 4/1 andi32::MAX/1, andi64::MIN.Minor: audio trim overflow (audio.rs). The new
values_in(Duration) -> usizecomputes in u128 (it cannot overflow, even atDuration::MAX) and saturates tousize. The table testvalues_in_saturates_on_absurd_skipsincludes the 2e8 s case andDuration::MAX.Minor:
should_readqueues all video once the audio ends first (player.rs). Not cheap to fix properly: it needs a hard cap on queued video plus a switch to wall time while the audio is drained, and a clip whose audio is shorter than its video. Deferred to #469 (follow-up).Minor:
MediaInfo.hardwareafter a silent software fallback (hw.rs). Left as is. The deferral to #464 stands.Minor: Windows bindings from mingw-w64 headers. No code change. They pass
cargo checkand clippy forx86_64-pc-windows-msvcbut have not been linked or run on the Windows runner yet. Integrator: please watch the Windows job after merge.Local
FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci: all steps passed.Independent review, Task 64 (round 3, head
041dda0)Reviewed
git diff 978245c..origin/textchat-64-media-playback-crate:crates/fluxer-media-playback, theVideoKey::Mediavariant influxer-video, the vendoredrsmpeg/rusty_ffmpeg(PATCHES.md read), the committed Windows bindings and thextask ffmpegbinding install.What I checked
media_key_is_distinct_from_other_keysinfluxer-video/src/bus.rsand the twoFLUXER_HW_DECODER_TESTS=1tests intests/hw.rs) and assert what the plan says. For example,seek_lands_on_the_exact_frameuses 30 fps, gop 30, seek 1.5 s and expects counter 45.bt709_input_comes_out_bt601_limitedallows ±2.ninth_player_is_refusedexpectsUnsupported("too many players").drop_joins_threads_quicklyexpects < 200 ms. The 3 fixtures are 2.7, 6.6 and 7.9 KB (limit 20 KB). Their metadata holds only encoder tags, with no paths.make-fixtures.shis present.ByteSource,MemorySource(Arc<[u8]>),AudioOut(push/played/queued/pause/flush),VideoOut,OpenOptions(default max 1920×1080),HwPreference::{Auto, Off},HwKind::{Vaapi, Nvdec, D3d11va},Player::open(...)and its play/pause/seek/set_looping/set_rate,PlayerEvent,MediaInfo,PlaybackError::{Open, Unsupported, Io, Decode},ffmpeg_info() -> FfmpegInfo,hardware_decoders(),MAX_PLAYERS = 8andVideoKey::Media { id: u64 }. The crate depends only onrsmpeg,fluxer-video,thiserrorandtracing.rsmpegis pinned=0.18.0.fluxer-core,fluxer-uiorfluxer-emoji. No URLs, paths or secrets appear in code, fixtures or logs.PlaybackErrorstrings are FFmpeg error text orio::ErrorKindonly. The round-2 overflow fixes hold:to_durationusestry_from_secs_f64capped atMAX_MEDIA_TIME, seek targets are capped, the clock usestry_from_secs_f64plussaturating_add, video timestamps use saturating adds, andvalues_inis computed in u128. Each fix has a table test. The FFmpeg callbacks are wrapped incatch_unwind. Deferrals #464 and #469 exist as openfollow-upissues.FLUXER_SCREENSHOT_TESTS=1 cargo xtask ciin a fresh review worktree: all steps passed (ffmpeg fetch, fmt, clippy, deny, core-isolation, parity, tests). I did not check the remote Windows run. The committedwin-x64bindings still need the integrator to watch it, as the PR already says.Findings
audio.rsResampler::convertsizes its output withswr_get_out_samples, which scales by 48000 /sample_rate. A crafted file with an absurdly low sample rate (for example a 1 Hz WAV, and thewavdemuxer is enabled) makes each decoded frame allocate hundreds of MB, then queue it inpending. This does not panic, but server data can cause a large memory spike. Suggestion: reject audio withsample_ratebelow about 4000 Hz (play the video without sound, orUnsupportedwhen the file is audio-only), or caproom. This can be a follow-up.player.rs,DropdecrementsOPEN_PLAYERSeven when it detaches a thread that is still blocked inread_at. With a stalledHttpRangeSource, more thanMAX_PLAYERSsets of player threads can exist at once. The cap is a soft limit in that case. Worth a note in the README or a follow-up.No blocking findings.
Verdict: approve (0 blocking, 2 minor; local ci green).