Screen share 12: core-codec-negotiation (#159) #235

Manually merged
charles merged 2 commits from screenshare-12-core-codec-negotiation into screenshare 2026-11-12 09:48:18 +00:00
Owner

Stacked on: screenshare-11-core-stream-params

What this builds

fluxer-core::voice::stream::codec (spec §3.4, §6.3 "Codec negotiation (sharer)"):

  • rank_codecs(caps, &VideoSettings) -> Vec<VideoCodec>: port of rankScreenShareCodecs (default order, software-H.264 order, clamped-H.264 order; hardware first except on the software encoder path; AV1/H.265 opt-ins; H.265 only in hardware or pinned; pin in front; VP8 last resort).
  • select_codec(local, viewers, published, previous, suppressed) -> Selection { codec, reason: Unchanged | Changed | Deferred { wanted } }: the fork's computeNegotiatedVideoCodec + never-upgrade + suppression rules.
  • CodecNegotiation in Streams.negotiation: adverts per identity (op 1 and op 14 codecs), per-participant 3000 ms grace timers, selected codec, media_session_id (derived like rtc_connection_id, new per change), 10 000 ms suppression timer and the deferred reason.
  • Viewers: room participants whose voice state lists the own key or no list (or no voice state); every participant before the share is published.
  • The codec is selected when the capture starts (op 14, then PublishScreenShare { codec }); it is re-evaluated on adverts, participant join/leave (kept when the last one leaves), grace end, deferred change, Reconnected, CodecCapabilities, settings change and voice-state change. A change sends op 14, then RepublishScreenShare { codec } and the capture/encoder updates for the new codec (layers, software-H.264 clamp). The republished video keeps the share live under its new sid without a new op 4.
  • codec_advert: voice_only until CodecCapabilities arrived, then with_capabilities (encode = ranked codecs actually reported, decode = exactly the reported decoders, so no H.264 decode is ever claimed); re-advertised when this changes (capabilities, settings).
  • Migration-topic messages are never looked at.

Tests

crates/fluxer-core/tests/stream_codec.rs: selection_matches_fork_rules (18 table cases), ranking_hardware_first_and_h264_not_hardware_order, advert_switches_to_capabilities_and_never_claims_missing_decoder, op14_on_change_only, republish_command_on_codec_change, viewers_are_those_watching_or_unknown, migration_messages_ignored. FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green locally. No real-device tests apply.

Deviations from the plan

  • select_codec takes previous: Option<VideoCodec>, suppressed: bool and ViewerCaps { decode, grace_over } instead of now, last_change / recheck_at: Option<Instant>: the store has no clock (#109), so the grace and suppression windows are Effect::Schedule timers, as in typing.rs and audio_prefs.rs. Selection.reason says Unchanged/Changed/Deferred.
  • Negotiation (op 14) runs only while this client shares; the fork also evaluates and sends op 14 while not sharing. Adverts and grace timers are kept for the whole room so a later share knows them.
  • Re-evaluation on voice-state changes is an addition: the fork evaluates on room events only, so a participant that starts watching later could keep a codec it cannot decode.
  • Not ported: runtime encode-failure and decoder-exclusion denials of isVideoCodecAllowedForPublish, and the "frozen" ranking while published (the ranking is recomputed; it changes only with settings).
  • tests/voice_connection.rs::events_of_later_tasks_are_accepted now also expects the grace timer on ParticipantJoined.

Closes #159

🤖 Generated with Claude Code

Stacked on: screenshare-11-core-stream-params ## What this builds `fluxer-core::voice::stream::codec` (spec §3.4, §6.3 "Codec negotiation (sharer)"): - `rank_codecs(caps, &VideoSettings) -> Vec<VideoCodec>`: port of `rankScreenShareCodecs` (default order, software-H.264 order, clamped-H.264 order; hardware first except on the software encoder path; AV1/H.265 opt-ins; H.265 only in hardware or pinned; pin in front; VP8 last resort). - `select_codec(local, viewers, published, previous, suppressed) -> Selection { codec, reason: Unchanged | Changed | Deferred { wanted } }`: the fork's `computeNegotiatedVideoCodec` + never-upgrade + suppression rules. - `CodecNegotiation` in `Streams.negotiation`: adverts per identity (op 1 and op 14 codecs), per-participant 3000 ms grace timers, selected codec, `media_session_id` (derived like `rtc_connection_id`, new per change), 10 000 ms suppression timer and the deferred reason. - Viewers: room participants whose voice state lists the own key or no list (or no voice state); every participant before the share is published. - The codec is selected when the capture starts (op 14, then `PublishScreenShare { codec }`); it is re-evaluated on adverts, participant join/leave (kept when the last one leaves), grace end, deferred change, `Reconnected`, `CodecCapabilities`, settings change and voice-state change. A change sends op 14, then `RepublishScreenShare { codec }` and the capture/encoder updates for the new codec (layers, software-H.264 clamp). The republished video keeps the share live under its new sid without a new op 4. - `codec_advert`: `voice_only` until `CodecCapabilities` arrived, then `with_capabilities` (encode = ranked codecs actually reported, decode = exactly the reported decoders, so no H.264 decode is ever claimed); re-advertised when this changes (capabilities, settings). - Migration-topic messages are never looked at. ## Tests `crates/fluxer-core/tests/stream_codec.rs`: `selection_matches_fork_rules` (18 table cases), `ranking_hardware_first_and_h264_not_hardware_order`, `advert_switches_to_capabilities_and_never_claims_missing_decoder`, `op14_on_change_only`, `republish_command_on_codec_change`, `viewers_are_those_watching_or_unknown`, `migration_messages_ignored`. `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green locally. No real-device tests apply. ## Deviations from the plan - `select_codec` takes `previous: Option<VideoCodec>, suppressed: bool` and `ViewerCaps { decode, grace_over }` instead of `now, last_change` / `recheck_at: Option<Instant>`: the store has no clock (#109), so the grace and suppression windows are `Effect::Schedule` timers, as in `typing.rs` and `audio_prefs.rs`. `Selection.reason` says Unchanged/Changed/Deferred. - Negotiation (op 14) runs only while this client shares; the fork also evaluates and sends op 14 while not sharing. Adverts and grace timers are kept for the whole room so a later share knows them. - Re-evaluation on voice-state changes is an addition: the fork evaluates on room events only, so a participant that starts watching later could keep a codec it cannot decode. - Not ported: runtime encode-failure and decoder-exclusion denials of `isVideoCodecAllowedForPublish`, and the "frozen" ranking while published (the ranking is recomputed; it changes only with settings). - `tests/voice_connection.rs::events_of_later_tasks_are_accepted` now also expects the grace timer on `ParticipantJoined`. Closes #159 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Owner

Independent review: Screen share 12 (core codec negotiation)

Scope: git diff 63b98de..origin/screenshare-12-core-codec-negotiation (a944662). I checked it against the fork's ScreenShareCodecSelection.ts and ScreenShareCodecNegotiation.ts.

Local gate: FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci in a fresh worktree: all steps passed (fmt, clippy, deny, core isolation, parity, tests, screenshot tests; livekit-interop skipped as usual).

Checks

  1. Plan tests. All six named tests exist: selection_matches_fork_rules (18 cases, at least 12 required), ranking_hardware_first_and_h264_not_hardware_order, advert_switches_to_capabilities_and_never_claims_missing_decoder, op14_on_change_only, republish_command_on_codec_change and migration_messages_ignored, plus viewers_are_those_watching_or_unknown. The table covers every case the plan lists: all advertised, unknown viewer inside and after its grace, an H264-less native viewer forcing VP9, opt-ins off, a pinned codec, the suppression window, never upgrade, and the last viewer leaving.
  2. Interfaces. rank_codecs(&[CodecCap], &VideoSettings) -> Vec<VideoCodec> matches the plan exactly. select_codec takes previous and suppressed instead of now and last_change, ViewerCaps holds {decode, grace_over}, and Selection has no recheck_at. This is justified because the merged store has no clock (#109), and no other task's Interfaces block uses these names. CodecNegotiation is in Streams.negotiation, RepublishScreenShare { codec } is sent on each change, and PublishScreenShare uses the codec select_codec picks.
  3. Invariants. No new dependency. fluxer-core gets no iced, wgpu, winit, reqwest or tokio-tungstenite. No pixels or samples reach the core. No LiveKit types appear in a public signature. Server data cannot cause a panic here: payloads go through CodecMessage::parse, unknown identities are dropped, and there is no unwrap or indexing on server data. The code and fixtures contain no secrets.
  4. Review Focus 3 (advertisement). The decode flags are exactly the reported decoders, and encode lists only codecs the build reported, with the VP8 last-resort entry filtered out. The test asserts (H264, encode=true, decode=false). Voice-only is advertised until CodecCapabilities arrives.
  5. Review Focus 4 (fork rules). These match the fork:
    • only viewers count, using viewer_stream_keys; a participant with no list or no voice state counts as a viewer, and every participant counts before publication;
    • candidates must be encodable here and decodable by every viewer; a viewer silent past its grace restricts the choice to H264, VP9 and VP8; the fallback order and the VP8 last resort are the fork's;
    • is_upgrade uses the same rank comparison as the fork;
    • suppression is armed on each change and on each new publication sid, the first selection is never suppressed, and a deferred change is re-evaluated when the window ends;
    • the codec is kept when the last remote participant leaves (live.participants() lists remote participants only and is reduced first);
    • the ranking orders, including the clamped software H.264 order, match rankScreenShareCodecs and the fork's isH264SoftwareClamped.

Findings

  • minor: The tests never pin the spec's values. CODEC_ADVERTISEMENT_GRACE and CODEC_CHANGE_SUPPRESSION appear only by name (timers(&effects, CODEC_ADVERTISEMENT_GRACE)). If someone changed either constant, the tests would still pass. Add assert_eq!(CODEC_ADVERTISEMENT_GRACE, Duration::from_millis(3000)) and the matching assertion for 10 000 ms. The constants themselves are correct.
  • minor: Deferred items have no follow-up issues. The PR lists, as not ported, the encode-failure and decoder-exclusion denials of isVideoCodecAllowedForPublish and the frozen ranking while published. CLAUDE.md asks for deferred work to become follow-up issues linked from the PR before merge.
  • minor: Two divergences from the fork are not in spec §14: re-evaluating on VoiceStateUpdate (the fork does not), and sending op 14 only while sharing. Both are reasonable and listed in the PR body, but Review Focus 4 asks for an exact port. Record them in the spec's divergence list, or in a follow-up issue, so the acceptance checks expect them.
  • minor (for Task 21): A stop during a republish can wait for the wrong unpublish. If the user stops after the old video was unpublished but before the new one is published, Stopping waits for the old sid's LocalTrackUnpublished, which has already arrived, so it runs into STOP_TIMEOUT (5 s). A late LocalTrackPublished of the new track is then ignored by the core. The runtime should drop a republish that is still in flight when it gets UnpublishScreenShare.

Verdict: approve. No blocking findings and the local CI is green.

## Independent review: Screen share 12 (core codec negotiation) Scope: `git diff 63b98de..origin/screenshare-12-core-codec-negotiation` (a944662). I checked it against the fork's `ScreenShareCodecSelection.ts` and `ScreenShareCodecNegotiation.ts`. **Local gate:** `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` in a fresh worktree: **all steps passed** (fmt, clippy, deny, core isolation, parity, tests, screenshot tests; livekit-interop skipped as usual). ### Checks 1. **Plan tests.** All six named tests exist: `selection_matches_fork_rules` (18 cases, at least 12 required), `ranking_hardware_first_and_h264_not_hardware_order`, `advert_switches_to_capabilities_and_never_claims_missing_decoder`, `op14_on_change_only`, `republish_command_on_codec_change` and `migration_messages_ignored`, plus `viewers_are_those_watching_or_unknown`. The table covers every case the plan lists: all advertised, unknown viewer inside and after its grace, an H264-less native viewer forcing VP9, opt-ins off, a pinned codec, the suppression window, never upgrade, and the last viewer leaving. 2. **Interfaces.** `rank_codecs(&[CodecCap], &VideoSettings) -> Vec<VideoCodec>` matches the plan exactly. `select_codec` takes `previous` and `suppressed` instead of `now` and `last_change`, `ViewerCaps` holds `{decode, grace_over}`, and `Selection` has no `recheck_at`. This is justified because the merged store has no clock (#109), and no other task's Interfaces block uses these names. `CodecNegotiation` is in `Streams.negotiation`, `RepublishScreenShare { codec }` is sent on each change, and `PublishScreenShare` uses the codec `select_codec` picks. 3. **Invariants.** No new dependency. `fluxer-core` gets no iced, wgpu, winit, reqwest or tokio-tungstenite. No pixels or samples reach the core. No LiveKit types appear in a public signature. Server data cannot cause a panic here: payloads go through `CodecMessage::parse`, unknown identities are dropped, and there is no unwrap or indexing on server data. The code and fixtures contain no secrets. 4. **Review Focus 3 (advertisement).** The `decode` flags are exactly the reported decoders, and `encode` lists only codecs the build reported, with the VP8 last-resort entry filtered out. The test asserts `(H264, encode=true, decode=false)`. Voice-only is advertised until `CodecCapabilities` arrives. 5. **Review Focus 4 (fork rules).** These match the fork: - only viewers count, using `viewer_stream_keys`; a participant with no list or no voice state counts as a viewer, and every participant counts before publication; - candidates must be encodable here and decodable by every viewer; a viewer silent past its grace restricts the choice to H264, VP9 and VP8; the fallback order and the VP8 last resort are the fork's; - `is_upgrade` uses the same rank comparison as the fork; - suppression is armed on each change and on each new publication sid, the first selection is never suppressed, and a deferred change is re-evaluated when the window ends; - the codec is kept when the last remote participant leaves (`live.participants()` lists remote participants only and is reduced first); - the ranking orders, including the clamped software H.264 order, match `rankScreenShareCodecs` and the fork's `isH264SoftwareClamped`. ### Findings - **minor:** The tests never pin the spec's values. `CODEC_ADVERTISEMENT_GRACE` and `CODEC_CHANGE_SUPPRESSION` appear only by name (`timers(&effects, CODEC_ADVERTISEMENT_GRACE)`). If someone changed either constant, the tests would still pass. Add `assert_eq!(CODEC_ADVERTISEMENT_GRACE, Duration::from_millis(3000))` and the matching assertion for 10 000 ms. The constants themselves are correct. - **minor:** Deferred items have no follow-up issues. The PR lists, as not ported, the encode-failure and decoder-exclusion denials of `isVideoCodecAllowedForPublish` and the frozen ranking while published. `CLAUDE.md` asks for deferred work to become `follow-up` issues linked from the PR before merge. - **minor:** Two divergences from the fork are not in spec §14: re-evaluating on `VoiceStateUpdate` (the fork does not), and sending op 14 only while sharing. Both are reasonable and listed in the PR body, but Review Focus 4 asks for an exact port. Record them in the spec's divergence list, or in a follow-up issue, so the acceptance checks expect them. - **minor (for Task 21):** A stop during a republish can wait for the wrong unpublish. If the user stops after the old video was unpublished but before the new one is published, `Stopping` waits for the old sid's `LocalTrackUnpublished`, which has already arrived, so it runs into `STOP_TIMEOUT` (5 s). A late `LocalTrackPublished` of the new track is then ignored by the core. The runtime should drop a republish that is still in flight when it gets `UnpublishScreenShare`. **Verdict: approve.** No blocking findings and the local CI is green.
charles manually merged commit f98d075b60 into screenshare 2026-11-12 09:48:18 +00:00
Sign in to join this conversation.
No description provided.