fix(coordinator): two-phase member-IP-refresh so permuted IPs converge instead of livelocking #193

Open
charles wants to merge 0 commits from fix/ip-refresh-two-phase-converge into main
Owner

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention).

Root cause (failure #2): after a power-cycle Docker permutes members' primary IPs. The leader's IP-refresh loop updated grid_members.docker_bridge_ip row-by-row and hit the grid_members_docker_bridge_ip_key UNIQUE index on every member of a swap/cycle, caught the error, and skipped — so it livelocked and never converged, leaving the DB stale (breaking liveness, config generation, and causing downstream /32 collisions). No member.ip_changed was emitted for skipped rows either.

Fix: compute the full desired mapping, detect permutation conflicts, phase 1 park conflicting members on a disjoint temp range (10.255.0.0/24), phase 2 set finals — provably collision-free. Relies on the verified serial+synchronous event-sourced materialization (design option A; see code comment). Emits member.ip_changed per member whose final IP changed.

Tests: new permutation tests (2-swap, 3-cycle, mixed) with a mock modeling the real unique constraint; 9/9 on the file, no regressions, tsc clean. DB-backed test deferred (no PG in sandbox).

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention). **Root cause (failure #2):** after a power-cycle Docker permutes members' primary IPs. The leader's IP-refresh loop updated `grid_members.docker_bridge_ip` row-by-row and hit the `grid_members_docker_bridge_ip_key` UNIQUE index on every member of a swap/cycle, caught the error, and skipped — so it **livelocked and never converged**, leaving the DB stale (breaking liveness, config generation, and causing downstream /32 collisions). No `member.ip_changed` was emitted for skipped rows either. **Fix:** compute the full desired mapping, detect permutation conflicts, **phase 1** park conflicting members on a disjoint temp range (`10.255.0.0/24`), **phase 2** set finals — provably collision-free. Relies on the verified serial+synchronous event-sourced materialization (design option A; see code comment). Emits `member.ip_changed` per member whose final IP changed. Tests: new permutation tests (2-swap, 3-cycle, mixed) with a mock modeling the real unique constraint; 9/9 on the file, no regressions, tsc clean. DB-backed test deferred (no PG in sandbox).
After a host power-cycle Docker PERMUTES floating members' primary lattice
IPs (A takes B's IP, B takes C's, ...). The leader-gated member-IP-refresh
loop wrote rows one-by-one: every member of a swap/cycle targets an address
still held by another member, so each write hit the
grid_members_docker_bridge_ip_key UNIQUE index, got caught, logged, and
SKIPPED — the loop LIVELOCKED and the DB never converged. It also emitted no
member.ip_changed for the skipped rows, so config-push never re-pushed peers.

Replace the naive loop with collision-free two-phase convergence:
  0. Resolve all live IPs and build the desired {member -> live_ip} moves.
  1. Detect conflicts (a target IP still held by a DIFFERENT mover = a
     permutation swap/cycle).
  2. Phase 1: park every conflicting member on a unique temp address in the
     reserved 10.255.0.0/24 range (disjoint from all real lattice CIDRs, so it
     can never collide).
  3. Phase 2: set every mover to its final live IP — all real targets are now
     vacated, so no unique-constraint collision is possible.
Non-conflicting members are set directly. member.ip_changed is emitted once
per member whose FINAL IP changed (never for a park address).

Design option A (serial, ordered event-sourced writes): emitEvent runs the
grid-member materializer (the real UPDATE) synchronously in the same committed
transaction before setDockerBridgeIp resolves, so awaiting all parks then all
finals guarantees targets are freed before any final write. This keeps every
write on the canonical event-sourced path — no raw-SQL transaction bypass
needed. Verified emitEvent/emitMemberPatch/materializeGridMember ordering in
services/event-emitter.ts and materializers/grid-member-materializer.ts.

Tests: the mock repo now models the UNIQUE constraint (rejects moving into an
IP another member still holds), so a naive row-by-row update WOULD collide.
Added failing-then-passing tests for a 2-member swap, a 3-member cycle, and a
mixed conflict+independent move, asserting full convergence and one
member.ip_changed per changed member. Leader/env gating unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This pull request has changes conflicting with the target branch.
  • packages/api/src/services/member-ip-refresh.ts
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/ip-refresh-two-phase-converge:fix/ip-refresh-two-phase-converge
git switch fix/ip-refresh-two-phase-converge

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/ip-refresh-two-phase-converge
git switch fix/ip-refresh-two-phase-converge
git rebase main
git switch main
git merge --ff-only fix/ip-refresh-two-phase-converge
git switch fix/ip-refresh-two-phase-converge
git rebase main
git switch main
git merge --no-ff fix/ip-refresh-two-phase-converge
git switch main
git merge --squash fix/ip-refresh-two-phase-converge
git switch main
git merge --ff-only fix/ip-refresh-two-phase-converge
git switch main
git merge fix/ip-refresh-two-phase-converge
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!193
No description provided.