fix(coordinator): two-phase member-IP-refresh so permuted IPs converge instead of livelocking #193
Open
charles
wants to merge 0 commits from
fix/ip-refresh-two-phase-converge into main
pull from: fix/ip-refresh-two-phase-converge
merge into: charles:main
charles:main
charles:fix/coordinator-cpu-self-watchdog
charles:fix/lattice-aux-address-reserve-sidecar-ip
charles:fix/agent-reconcile-stale-bridge-32
charles:fix/coordinator-phantom-updating-badge
charles:fix/soa-serial-content-drift
charles:diag/debug-profile-token-file
charles:diag/coordinator-cpu-heap-profile
charles:fix/coordinator-cloud-credentials-key-seed
charles:fix/coordinator-process-stats-comm
charles:fix/lattice-dot2-aux-reservation
charles:fix/docker-bridge-ip-permutation-livelock
charles:design/member-mesh-dns-wiring
charles:design/mesh-dns-sidecar
charles:design/control-plane-resolver-adoption
charles:fix/reconcile-epoch-recreate-and-discovery-volume
charles:feat/mesh-resolver-enablement-wiring
charles:fix/mesh-resolver-coordinator-discovery
charles:fix/mesh-resolver-bind-loopback
charles:fix/mesh-resolver-publish-shadow-and-loopback
charles:design/sidecar-mesh-name-resolution
charles:design/split-fresh-install-provisioning
charles:design/sidecar-managed-coordinators
charles:feat/no-backwards-updates-coordinator
charles:feat/no-backwards-image-updates
charles:hotfix/liveness-flush-cadence
charles:hotfix/leader-election-uuid-and-holder-endpoint-repair
charles:feat/cheap-liveness
charles:design/edge-homing-routing
charles:design/core-edge-state-partitioning
charles:feat/discovery-vlan-attachments-stage3-ui
charles:feat/discovery-vlan-attachments-stage2
charles:fix/discovery-lattice-network
charles:feat/ci-closure-fingerprint
charles:feat/discovery-probe-routing
charles:feat/discovery-probe-role
charles:docs/discovery-probe-role-plan
charles:feat/discovery-synced-inventory
charles:fix/dns-record-ip-address-fk-safe
charles:fix/ddns-dedup-unique-index
charles:fix/discovery-addresses-sync
charles:fix/blocklist-fetch-fields-node-local
charles:fix/oci-operational-tags-node-local
charles:fix/secondary-database-replica-mode
charles:fix/database-member-id-env-persists
charles:fix/member-install-use-configured-registry
charles:fix/wireguard-keygen-nonavx2
charles:fix/reconciler-preserve-database-joiner
charles:fix/member-install-mesh-endpoint-split
charles:feat/overview-dns-flow-card
charles:fix/dns-push-per-zone-cache-convergence
charles:feat/kea-decline-probation-1h
charles:fix/oci-drift-null-served-false-positive
charles:ui/dnsdist-flow-trunk-fork
charles:ui/dnsdist-flow-merge-pool-bands
charles:ui/dnsdist-flow-sankey-ribbons
charles:fix/dns-traffic-links-fit-node-box
charles:feat/dns-traffic-flow-no-overlap
charles:feat/dns-traffic-ribbon-split
charles:feat/dns-traffic-sankey-view
charles:fix/sidecar-report-desired-digest
charles:fix/dns-traffic-peer-label-prefer-clean
charles:feat/recursor-dnssec-validate
charles:feat/dns-traffic-label-known-peers
charles:feat/dns-traffic-node-boxes
charles:feat/dns-traffic-group-by-node
charles:feat/dns-traffic-dist-lb-policy
charles:feat/dns-traffic-source-cidr-label
charles:feat/dns-traffic-source-subnets
charles:fix/ddns-trigger-idempotent-no-view-lock
charles:fix/ddns-local-first-server-order
charles:fix/dns-traffic-legend-idle-vs-down
charles:fix/dns-traffic-state-gauge-ui
charles:feat/member-detail-traffic-panel
charles:fix/remote-host-metrics-mesh-url
charles:fix/apex-ns-push-on-member-change
charles:fix/reconcile-transient-flood
charles:fix/log-server-report-real-ip
charles:fix/shared-log-read-token
charles:fix/sidecar-member-routes-hostname
charles:fix/ui-remote-member-ip-labels
charles:fix/remote-log-candidate-ip
charles:fix/remote-member-vpn-endpoint
charles:fix/host-install-live-cert
charles:fix/reconcile-adopt-unlabeled-orphan
charles:fix/agent-baseline-and-host-install-ca
charles:fix/dns-pdns-schema-apply-deadlock
charles:fix/dns-pgbouncer-backend-recycle
charles:fix/host-tile-24h-bucket-resolution
charles:fix/dns-pg-backend-recycle
charles:fix/ui-heartbeat-container-mem
charles:fix/whole-mem-ui-and-coordinator
charles:fix/heartbeat-schema-container-mem
charles:feat/host-tile-whole-service-memory
charles:fix/dns-postgres-malloc-arena
charles:fix/discovery-ui-response-schemas
charles:fix/discovery-sidebar-nav-privilege
charles:fix/discovery-neighbor-read-after-ping
charles:fix/discovery-test-fetch-mock-type
charles:fix/discovery-coordinator-base-fallback
charles:feature/auto-discovery
charles:docs/auto-discovery-spec
charles:fix/reconcile-transient-ddns-autoresolve
charles:fix/kea-ddns-apply-contention
charles:fix/chart-tooltip-timestamps
charles:feat/host-metrics
charles:feat/blocklist-entries-browser
charles:fix/ns-list-reconcile-idempotent
charles:fix/oci-drift-placeholder-soa
charles:fix/reconcile-dynamic-ns-blindspot
charles:fix/errors-tile-full-width-text
charles:fix/sparkline-range-tooltip
charles:feat/dashboard-row-shuffle
charles:fix/reconcile-envelope-and-auth-noise
charles:fix/noc-tiles-live-data
charles:fix/topology-gutter-routing
charles:feature/topology-circuit-map
charles:fix/ddns-zone-push-coalesce
charles:fix/vector-clock-loose-index-scan
charles:fix/event-log-retention-ddns-batching
charles:fix/update-image-reprobe-and-ddns-timeout
charles:fix/noc-board-height-tune
charles:fix/noc-board-style-match
charles:fix/stale-push-error-noise
charles:fix/ci-image-digest-churn
charles:worktree-dashboard-noc-overhaul
charles:fix/sidecar-self-swap-helper-auth
charles:fix/bootstrap-slim-image-bun-paths
charles:feat/mesh-hosts-container-errors
charles:fix/octet-log-exc-info
charles:ci/add-metrics-logs-images
charles:fix/sidecar-swap-latch-reset
charles:fix/registry-poller-namespace
charles:ci/fix-build-images-singlejob-main
charles:fix/dhcp-config-reconcile-false-positives
charles:ci/build-images-singlejob
charles:feature/ui-closest-coordinator-api
charles:fix/dhcp-kea-socket-retry-and-lattice-ip-race
charles:feat/tls-multi-cert-reconcile
charles:feat/tls-cert-management
charles:fix/sidecar-logs-streaming-no-timeout
charles:fix/entity-table-and-permanent-env
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
charles/projectoctet!193
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/ip-refresh-two-phase-converge"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention).
Root cause (failure #2): after a power-cycle Docker permutes members' primary IPs. The leader's IP-refresh loop updated
grid_members.docker_bridge_iprow-by-row and hit thegrid_members_docker_bridge_ip_keyUNIQUE index on every member of a swap/cycle, caught the error, and skipped — so it livelocked and never converged, leaving the DB stale (breaking liveness, config generation, and causing downstream /32 collisions). Nomember.ip_changedwas emitted for skipped rows either.Fix: compute the full desired mapping, detect permutation conflicts, phase 1 park conflicting members on a disjoint temp range (
10.255.0.0/24), phase 2 set finals — provably collision-free. Relies on the verified serial+synchronous event-sourced materialization (design option A; see code comment). Emitsmember.ip_changedper member whose final IP changed.Tests: new permutation tests (2-swap, 3-cycle, mixed) with a mock modeling the real unique constraint; 9/9 on the file, no regressions, tsc clean. DB-backed test deferred (no PG in sandbox).
After a host power-cycle Docker PERMUTES floating members' primary lattice IPs (A takes B's IP, B takes C's, ...). The leader-gated member-IP-refresh loop wrote rows one-by-one: every member of a swap/cycle targets an address still held by another member, so each write hit the grid_members_docker_bridge_ip_key UNIQUE index, got caught, logged, and SKIPPED — the loop LIVELOCKED and the DB never converged. It also emitted no member.ip_changed for the skipped rows, so config-push never re-pushed peers. Replace the naive loop with collision-free two-phase convergence: 0. Resolve all live IPs and build the desired {member -> live_ip} moves. 1. Detect conflicts (a target IP still held by a DIFFERENT mover = a permutation swap/cycle). 2. Phase 1: park every conflicting member on a unique temp address in the reserved 10.255.0.0/24 range (disjoint from all real lattice CIDRs, so it can never collide). 3. Phase 2: set every mover to its final live IP — all real targets are now vacated, so no unique-constraint collision is possible. Non-conflicting members are set directly. member.ip_changed is emitted once per member whose FINAL IP changed (never for a park address). Design option A (serial, ordered event-sourced writes): emitEvent runs the grid-member materializer (the real UPDATE) synchronously in the same committed transaction before setDockerBridgeIp resolves, so awaiting all parks then all finals guarantees targets are freed before any final write. This keeps every write on the canonical event-sourced path — no raw-SQL transaction bypass needed. Verified emitEvent/emitMemberPatch/materializeGridMember ordering in services/event-emitter.ts and materializers/grid-member-materializer.ts. Tests: the mock repo now models the UNIQUE constraint (rejects moving into an IP another member still holds), so a naive row-by-row update WOULD collide. Added failing-then-passing tests for a 2-member swap, a 3-member cycle, and a mixed conflict+independent move, asserting full convergence and one member.ip_changed per changed member. Leader/env gating unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.