Camera 3: Spike — a camera track through the publish path (#955) #978

Closed
charles wants to merge 0 commits from camera-3-spike-a-camera-track into camera
Owner

What was built

Spike for Camera 3: does the existing publish path take a camera track? Yes. The decision rule passes, so the default stays the original's rule: VP9 in one L3T3_KEY encoding. VP8 and H264 go out as simulcast in three layers. The findings are in docs/superpowers/research/2026-11-19-camera-spikes.md §3, and crates/fluxer-voice/README.md has a short new "Camera" section. No production code changed.

  • crates/fluxer-voice/tests/livekit_camera_spike.rs:
    • camera_encodings_as_the_js_sdk needs no server and runs in cargo xtask ci. It checks the SDK's encodings for the options Camera 9 should use, against livekit-client 2.22.3 as vendored in the fork.
    • Four server cases, #[ignore] and gated on FLUXER_LIVEKIT_TESTS=1: camera_pattern_is_received_per_codec, layers_reported_per_codec, camera_beside_share_keeps_share_rate (also reads per-thread CPU from /proc/self/task/*/stat over 20 s) and camera_frame_decrypts_with_e2ee (includes a wrong-key negative control).

Results (local livekit-server 1.12.0, test pattern 1280×720@30)

  • Frames received: the viewer decodes the camera in every codec: VP9 and VP8 at 30 fps, H264 at 25–29 fps (encoded with VAAPI on this machine).
  • Layers: when the viewer asks for Low, Medium and High it decodes 320×180, 640×360 and 1280×720 in all three codecs. The remote publication reports simulcasted for each.
  • Camera beside a share: a 1080p30 software VP9 share decodes at 30.0 fps both alone and beside a VP9 or VP8 camera.
  • CPU: the camera's encoder threads use 29.5–34.7 % of one core with VP9 L3T3_KEY and 16–20 % with VP8 simulcast. Measured on a Ryzen 9 9950X while other builds were running.
  • E2EE: frames decrypt with the client's key parameters and the server still switches layers. With a different key nothing decodes.

Findings for Camera 9 (deviations from the spec's assumptions)

  • The Rust SDK picks h1080's 3 Mbit/s for a 1280×720 source (it compares > where the JS SDK compares >=). It also runs the h180 layer at 15 fps where the JS SDK uses 20. So video_encoding and simulcast_layers must be set explicitly; the values are in spikes §3.4.
  • Spec §8.1 says the SDK's default degradation for a camera is "balanced". It is actually maintain-framerate, in both SDKs. Leaving it unset still matches the original.
  • VP9 SVC works only with is_screencast: false. On a screencast source libvpx encodes nothing in L3T3_KEY (the Screen share spike's svc_matrix).
  • The scalibility_mode stats field in libwebrtc 0.4.0 is misspelled, so it is always empty. To check that layers exist, switch layers on the viewer side.
  • Not measured: H264 simulcast (three VAAPI sessions) beside a hardware share, and AV1.

How it was tested

  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci passes locally.
  • I ran the server cases myself against tools/livekit-test-server.sh on loopback, with FLUXER_CAMERA_FORBID=1 FLUXER_LIVEKIT_TESTS=1 cargo test -p fluxer-voice --test livekit_camera_spike -- --ignored --nocapture --test-threads=1.
  • No camera, microphone or screen was opened, and no frame was written anywhere. The client was not run.
  • No new dependencies. Nothing needs Charles.

Closes #955

🤖 Generated with Claude Code

## What was built Spike for Camera 3: does the existing publish path take a camera track? **Yes. The decision rule passes, so the default stays the original's rule: VP9 in one `L3T3_KEY` encoding. VP8 and H264 go out as simulcast in three layers.** The findings are in `docs/superpowers/research/2026-11-19-camera-spikes.md` §3, and `crates/fluxer-voice/README.md` has a short new "Camera" section. No production code changed. - `crates/fluxer-voice/tests/livekit_camera_spike.rs`: - `camera_encodings_as_the_js_sdk` needs no server and runs in `cargo xtask ci`. It checks the SDK's encodings for the options Camera 9 should use, against `livekit-client` 2.22.3 as vendored in the fork. - Four server cases, `#[ignore]` and gated on `FLUXER_LIVEKIT_TESTS=1`: `camera_pattern_is_received_per_codec`, `layers_reported_per_codec`, `camera_beside_share_keeps_share_rate` (also reads per-thread CPU from `/proc/self/task/*/stat` over 20 s) and `camera_frame_decrypts_with_e2ee` (includes a wrong-key negative control). ## Results (local livekit-server 1.12.0, test pattern 1280×720@30) - **Frames received**: the viewer decodes the camera in every codec: VP9 and VP8 at 30 fps, H264 at 25–29 fps (encoded with VAAPI on this machine). - **Layers**: when the viewer asks for Low, Medium and High it decodes 320×180, 640×360 and 1280×720 in all three codecs. The remote publication reports `simulcasted` for each. - **Camera beside a share**: a 1080p30 software VP9 share decodes at 30.0 fps both alone and beside a VP9 or VP8 camera. - **CPU**: the camera's encoder threads use 29.5–34.7 % of one core with VP9 `L3T3_KEY` and 16–20 % with VP8 simulcast. Measured on a Ryzen 9 9950X while other builds were running. - **E2EE**: frames decrypt with the client's key parameters and the server still switches layers. With a different key nothing decodes. ## Findings for Camera 9 (deviations from the spec's assumptions) - The Rust SDK picks h1080's 3 Mbit/s for a 1280×720 source (it compares `>` where the JS SDK compares `>=`). It also runs the h180 layer at 15 fps where the JS SDK uses 20. So `video_encoding` and `simulcast_layers` must be set explicitly; the values are in spikes §3.4. - Spec §8.1 says the SDK's default degradation for a camera is "balanced". It is actually maintain-framerate, in both SDKs. Leaving it unset still matches the original. - VP9 SVC works only with `is_screencast: false`. On a screencast source libvpx encodes nothing in `L3T3_KEY` (the Screen share spike's `svc_matrix`). - The `scalibility_mode` stats field in `libwebrtc` 0.4.0 is misspelled, so it is always empty. To check that layers exist, switch layers on the viewer side. - Not measured: H264 simulcast (three VAAPI sessions) beside a hardware share, and AV1. ## How it was tested - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` passes locally. - I ran the server cases myself against `tools/livekit-test-server.sh` on loopback, with `FLUXER_CAMERA_FORBID=1 FLUXER_LIVEKIT_TESTS=1 cargo test -p fluxer-voice --test livekit_camera_spike -- --ignored --nocapture --test-threads=1`. - No camera, microphone or screen was opened, and no frame was written anywhere. The client was not run. - No new dependencies. Nothing needs Charles. Closes #955 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The camera options the original's livekit-client 2.22.3 computes
(fluxer_app/pkgs/livekit-client: options.ts:274, publishUtils.ts:43,105,236,
LocalParticipant.ts:1068), checked against the pinned SDK without a server,
and four server cases behind FLUXER_LIVEKIT_TESTS=1: decodable frames per
codec, layers on request, a camera beside a 1080p30 VP9 share with the
encoder threads' CPU, and E2EE. Test pattern only; nothing opens a camera.

Refs #955

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Spikes §3 and the README's Camera section: the decision rule passes on a
local livekit-server 1.12.0 with the test pattern. Default camera codec VP9
in one L3T3_KEY encoding (as the original), VP8 and H264 simulcast in three
layers; a 1080p30 software VP9 share keeps 30 fps beside the camera; the
camera's encoder threads use about 30 % (VP9) and 18 % (VP8) of one core;
frames decrypt with E2EE. Camera 9 must set video_encoding and
simulcast_layers itself (the Rust SDK picks a larger preset for 720p and
runs h180 at 15 fps).

Refs #955

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161qMQMsCyJhjYw4gqa6e4K
Author
Owner

Merged into camera by hand (the merge API returns 405); the commits are in the branch.

Merged into `camera` by hand (the merge API returns 405); the commits are in the branch.
charles closed this pull request 2026-11-19 23:15:53 +00:00

Pull request closed

Sign in to join this conversation.
No description provided.