Camera 9: fluxer-voice the camera track (#961) #982

Closed
charles wants to merge 0 commits from camera-9-voice-the-camera-track into camera
Owner

What this builds (Camera plan Task 9, spec §5.4, §8.1)

fluxer-voice publishes one camera track beside the microphone and the screen share.

  • Room gains publish_camera(CameraPublish { codec, encode }) -> sid, update_camera_encoding(CameraEncode), unpublish_camera() and push_camera(VideoFrame). Each has a default for a room without video, like the screen methods. The contract paragraph gains the camera's rules: at most one camera, independent of the screen share, the SDK's own encoder, taken down by disconnect.
  • LiveKitRoom (rtc.rs) publishes NativeVideoSource::new(_, false) (CAMERA_IS_SCREENCAST) as source camera, with track name camera and no stream name, using camera_video_options, a pure function with its own test, like screen_video_options:
    • the encoding is set explicitly (the Rust SDK would otherwise pick the next preset up, spikes §3.2);
    • VP9 and AV1 are sent as one encoding in the CameraEncode's scalability mode (L3T3_KEY);
    • VP8 and H264 are sent as simulcast with livekit-client 2.22.3's lower layers. The 16:9 layers are h180/h360; the 4:3 layers are the VideoPresets43 ones; the nearer aspect decides. Both layers are used from 960 px on the long side, the lowest alone below that. No layer runs faster than the source;
    • video_encoder: Auto, never PreEncoded; degradation_preference is left unset (the SDK's camera default, maintain-framerate).
    • Frames are paced to the encoding's frame rate (FramePace). A camera that the server removes, or that a full reconnect republishes, is reported like the share's. No encoder statistics, self-test or GPU order touch the camera.
  • Session: carries out PublishCamera, UnpublishCamera and UpdateCameraEncoding:
    • It reports LocalTrackPublished { source: Camera, publication: LocalPublication::Camera(id) }, also for a publication the room restores after a reconnect.
    • A second publish replaces the first.
    • Commands run in order, which covers the #921 case for the camera.
    • Disconnect, the room going away and a move to another room report the camera's LocalTrackUnpublished.
    • VoiceHandle::push_camera uses a watch channel of its own (newest frame wins, only while published, never sent to the share's track).
    • RemoteVideo::source is new.
  • FakeRoom: adds FakeCall::{PublishCamera, UnpublishCamera, UpdateCameraEncoding} (appended), FakeCamera, camera(), camera_frames(), fail_next_publish_camera, hold_camera_publish/release_camera_publish, remove_local_camera and restore_local_camera. play_video sends TestPattern::camera on a track that was published as a camera.
  • README: the "Camera" section now describes what was built (the spike findings are kept below).

How it was tested

  • tests/session_camera.rs (10 tests, fake room, paused clock): publish_camera_reports_local_track_with_camera_id, camera_commands_need_a_room_and_a_camera, failed_camera_publication_reports_nothing_and_takes_no_frames, camera_and_screen_are_independent, unpublish_is_harmless_when_none, disconnect_takes_camera_down, late_publication_after_unpublish_is_taken_down, frames_pushed_only_while_published, reconnect_restores_camera, remote_video_carries_source.
  • rtc unit tests:
    • camera_options_per_codec_and_layers puts the options through the SDK's own compute_video_encodings and checks the result against Camera 3's table: VP9 gives one L3T3_KEY encoding at 1.445 Mbit/s; VP8 and H264 give f/h/q at 1.7M@30, 450k@20 and 160k@20. It also checks 1080p, 640×360 (two layers), under 480 px (one layer), 4:3 (330k/125k), portrait, a slower source and Single.
    • camera_never_uses_pre_encoded and camera_without_a_room_refuses_politely are the other two.
  • Interop, FLUXER_LIVEKIT_TESTS=1 (the Linux runner runs these): camera_round_trip_per_codec passed with AV1, H264, VP9 and VP8 (the camera pattern arrives unmirrored with its source); camera_beside_share passed (camera and share side by side, and the share outlives the camera). I ran the whole livekit_interop suite locally against tools/livekit-test-server.sh (loopback, test pattern only): 20 passed.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passed locally.
  • No camera, microphone or screen was opened, and the client was not run. No frame is written or logged: the camera logs only the codec, size, rate and bitrate.

Deviations and notes

  • update_camera_encoding behaves like update_screen_encoding. A new frame rate applies at once and frames of a new size go through immediately. A new bitrate or new layers are logged and apply from the next publish, because the SDK does not expose the RTP sender.
  • For a source between 480 and 960 px, simulcast_layers gets only the lowest preset. The Rust SDK would otherwise send h360 beside the source, where livekit-client sends h180 (publishUtils.ts:175).
  • The plan says "play_video with a source". I implemented it as the fake reading the source from the track it published, rather than adding a parameter. This keeps every existing caller unchanged.
  • A camera Svc layering given with a codec that has no scalability mode (VP8, H264, H265) is published as a single layer.
  • Outside fluxer-voice: one test literal in apps/fluxer/src/voice_rt.rs gains RemoteVideo::source. The runtime's handling of camera frames is Task 12.
  • No cfg(windows) code was added.

Closes #961

🤖 Generated with Claude Code

https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K

## What this builds (Camera plan Task 9, spec §5.4, §8.1) `fluxer-voice` publishes one camera track beside the microphone and the screen share. - `Room` gains `publish_camera(CameraPublish { codec, encode }) -> sid`, `update_camera_encoding(CameraEncode)`, `unpublish_camera()` and `push_camera(VideoFrame)`. Each has a default for a room without video, like the screen methods. The contract paragraph gains the camera's rules: at most one camera, independent of the screen share, the SDK's own encoder, taken down by `disconnect`. - `LiveKitRoom` (rtc.rs) publishes `NativeVideoSource::new(_, false)` (`CAMERA_IS_SCREENCAST`) as source `camera`, with track name `camera` and no stream name, using `camera_video_options`, a pure function with its own test, like `screen_video_options`: - the encoding is set explicitly (the Rust SDK would otherwise pick the next preset up, spikes §3.2); - VP9 and AV1 are sent as one encoding in the `CameraEncode`'s scalability mode (`L3T3_KEY`); - VP8 and H264 are sent as simulcast with `livekit-client` 2.22.3's lower layers. The 16:9 layers are `h180`/`h360`; the 4:3 layers are the `VideoPresets43` ones; the nearer aspect decides. Both layers are used from 960 px on the long side, the lowest alone below that. No layer runs faster than the source; - `video_encoder: Auto`, never `PreEncoded`; `degradation_preference` is left unset (the SDK's camera default, maintain-framerate). - Frames are paced to the encoding's frame rate (`FramePace`). A camera that the server removes, or that a full reconnect republishes, is reported like the share's. No encoder statistics, self-test or GPU order touch the camera. - Session: carries out `PublishCamera`, `UnpublishCamera` and `UpdateCameraEncoding`: - It reports `LocalTrackPublished { source: Camera, publication: LocalPublication::Camera(id) }`, also for a publication the room restores after a reconnect. - A second publish replaces the first. - Commands run in order, which covers the #921 case for the camera. - `Disconnect`, the room going away and a move to another room report the camera's `LocalTrackUnpublished`. - `VoiceHandle::push_camera` uses a watch channel of its own (newest frame wins, only while published, never sent to the share's track). - `RemoteVideo::source` is new. - `FakeRoom`: adds `FakeCall::{PublishCamera, UnpublishCamera, UpdateCameraEncoding}` (appended), `FakeCamera`, `camera()`, `camera_frames()`, `fail_next_publish_camera`, `hold_camera_publish`/`release_camera_publish`, `remove_local_camera` and `restore_local_camera`. `play_video` sends `TestPattern::camera` on a track that was published as a camera. - README: the "Camera" section now describes what was built (the spike findings are kept below). ## How it was tested - `tests/session_camera.rs` (10 tests, fake room, paused clock): `publish_camera_reports_local_track_with_camera_id`, `camera_commands_need_a_room_and_a_camera`, `failed_camera_publication_reports_nothing_and_takes_no_frames`, `camera_and_screen_are_independent`, `unpublish_is_harmless_when_none`, `disconnect_takes_camera_down`, `late_publication_after_unpublish_is_taken_down`, `frames_pushed_only_while_published`, `reconnect_restores_camera`, `remote_video_carries_source`. - rtc unit tests: - `camera_options_per_codec_and_layers` puts the options through the SDK's own `compute_video_encodings` and checks the result against Camera 3's table: VP9 gives one `L3T3_KEY` encoding at 1.445 Mbit/s; VP8 and H264 give f/h/q at 1.7M@30, 450k@20 and 160k@20. It also checks 1080p, 640×360 (two layers), under 480 px (one layer), 4:3 (330k/125k), portrait, a slower source and `Single`. - `camera_never_uses_pre_encoded` and `camera_without_a_room_refuses_politely` are the other two. - Interop, `FLUXER_LIVEKIT_TESTS=1` (the Linux runner runs these): `camera_round_trip_per_codec` passed with AV1, H264, VP9 and VP8 (the camera pattern arrives unmirrored with its source); `camera_beside_share` passed (camera and share side by side, and the share outlives the camera). I ran the whole `livekit_interop` suite locally against `tools/livekit-test-server.sh` (loopback, test pattern only): 20 passed. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passed locally. - No camera, microphone or screen was opened, and the client was not run. No frame is written or logged: the camera logs only the codec, size, rate and bitrate. ## Deviations and notes - `update_camera_encoding` behaves like `update_screen_encoding`. A new frame rate applies at once and frames of a new size go through immediately. A new bitrate or new layers are logged and apply from the next publish, because the SDK does not expose the RTP sender. - For a source between 480 and 960 px, `simulcast_layers` gets only the lowest preset. The Rust SDK would otherwise send `h360` beside the source, where `livekit-client` sends `h180` (`publishUtils.ts:175`). - The plan says "`play_video` with a source". I implemented it as the fake reading the source from the track it published, rather than adding a parameter. This keeps every existing caller unchanged. - A camera `Svc` layering given with a codec that has no scalability mode (VP8, H264, H265) is published as a single layer. - Outside `fluxer-voice`: one test literal in `apps/fluxer/src/voice_rt.rs` gains `RemoteVideo::source`. The runtime's handling of camera frames is Task 12. - No `cfg(windows)` code was added. Closes #961 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Refs #961

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
The room publishes one camera track beside the screen share: a NativeVideoSource
that is not a screencast, source and track name "camera", VP9/AV1 in L3T3_KEY,
VP8/H264 simulcast with livekit-client 2.22.3's lower layers, the SDK's own
encoder. The session carries out PublishCamera, UnpublishCamera and
UpdateCameraEncoding, reports the camera's publications with their CameraId
(also after a reconnect), takes the camera down with the room, and passes
camera frames only while published. RemoteVideo carries its track's source.
FakeRoom records camera publications and frames and plays remote cameras in
the camera look. apps/fluxer: one test literal gains RemoteVideo::source.

Refs #961

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Author
Owner

Merged into camera by hand (the merge API returns 405); the commits are in the branch.

Merged into `camera` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-20 03:12:38 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.