fix(coordinator): self-heal docker_bridge_ip permutation after reshuffled reboot #183

Open
charles wants to merge 1 commit from fix/docker-bridge-ip-permutation-livelock into main
Owner

After an unclean reboot, float members come back on a reshuffled set of Docker lattice IPs (a permutation of the stored docker_bridge_ip values). The member-IP-refresh loop applied changes per-row, and under UNIQUE(docker_bridge_ip) each update transiently duplicated a peer and was rejected — livelock, grid stuck 'unreachable' until manual psql surgery.

Fix: constraint is now DEFERRABLE INITIALLY IMMEDIATE (squash + schema-repair for existing volumes); new MemberRepository.remapDockerBridgeIps applies the whole permutation in one txn with SET CONSTRAINTS DEFERRED; member-ip-refresh batches all changes. DB-gated permutation test reproduces the livelock then passes; 105 db+repo tests green.

🤖 Generated with Claude Code

After an unclean reboot, float members come back on a reshuffled set of Docker lattice IPs (a permutation of the stored docker_bridge_ip values). The member-IP-refresh loop applied changes per-row, and under UNIQUE(docker_bridge_ip) each update transiently duplicated a peer and was rejected — livelock, grid stuck 'unreachable' until manual psql surgery. Fix: constraint is now DEFERRABLE INITIALLY IMMEDIATE (squash + schema-repair for existing volumes); new MemberRepository.remapDockerBridgeIps applies the whole permutation in one txn with SET CONSTRAINTS DEFERRED; member-ip-refresh batches all changes. DB-gated permutation test reproduces the livelock then passes; 105 db+repo tests green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
After an unclean reboot, float members come back on a reshuffled set of
Docker lattice IPs, so the live map is a *permutation* of the values stored
in grid_members.docker_bridge_ip. The member-IP-refresh loop applied changes
per-row, and under the UNIQUE(docker_bridge_ip) constraint every single
update transiently duplicated a peer's current value and was rejected — so
the loop livelocked forever and the whole grid stayed stale ("unreachable"
everywhere: DB-liveness-vouch, config-push, log ingest all hit the wrong
container). Recovery required manual psql surgery.

Make it converge on its own:

- The UNIQUE(docker_bridge_ip) constraint is now DEFERRABLE INITIALLY
  IMMEDIATE (squash for fresh volumes; repairGridMembersDockerBridgeIpDeferrable
  for existing volumes, run from migrate.ts — DROP+ADD because Postgres only
  lets ALTER CONSTRAINT change FK deferrability). IMMEDIATE-by-default means
  the old code is unaffected until a caller explicitly defers.
- MemberRepository.remapDockerBridgeIps applies the whole change set in ONE
  transaction with SET CONSTRAINTS ... DEFERRED, so the permutation lands
  atomically and uniqueness is only enforced at commit.
- member-ip-refresh collects all changes first, then applies them via the
  batch method (was per-row setDockerBridgeIp).

Tests: DB-gated permutation swap (reproduces the livelock, then passes),
repair idempotency, and the existing unit suite updated to the batch API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012DiF99Lq5PbZAy18JZJsbr
This pull request has changes conflicting with the target branch.
  • packages/api/src/services/member-ip-refresh.ts
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/docker-bridge-ip-permutation-livelock:fix/docker-bridge-ip-permutation-livelock
git switch fix/docker-bridge-ip-permutation-livelock

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/docker-bridge-ip-permutation-livelock
git switch fix/docker-bridge-ip-permutation-livelock
git rebase main
git switch main
git merge --ff-only fix/docker-bridge-ip-permutation-livelock
git switch fix/docker-bridge-ip-permutation-livelock
git rebase main
git switch main
git merge --no-ff fix/docker-bridge-ip-permutation-livelock
git switch main
git merge --squash fix/docker-bridge-ip-permutation-livelock
git switch main
git merge --ff-only fix/docker-bridge-ip-permutation-livelock
git switch main
git merge fix/docker-bridge-ip-permutation-livelock
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!183
No description provided.