fix(agent): reconcile the member bridge /32 (remove stale secondaries) #194

Open
charles wants to merge 0 commits from fix/agent-reconcile-stale-bridge-32 into main
Owner

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention).

Root cause (failure #3): octet-agent applied the coordinator-echoed docker_bridge_ip as a /32 secondary but never removed stale ones. When the echoed value was stale (failure #2) or a member renumbered, the stale /32 belonged to another container's current IP → duplicate IPs on the bridge → ARP blackholes (dhcp claimed the coordinator's .6; metrics claimed dist-1's .10).

Fix: turn the apply into a reconcile — enumerate addrs on the bridge iface, delete every in-range (172.16.0.0/12) /32 that isn't the intended IP, add the intended one if missing. Never touches the primary /24 or out-of-range addresses; idempotent; error-tolerant (retries next tick). Adds an injectable exec seam for unit testing.

Tests: 10/10 on the file, full agent suite green (329), tsc + biome clean, binary builds.

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention). **Root cause (failure #3):** octet-agent applied the coordinator-echoed `docker_bridge_ip` as a `/32` secondary but **never removed stale ones**. When the echoed value was stale (failure #2) or a member renumbered, the stale /32 belonged to another container's current IP → duplicate IPs on the bridge → ARP blackholes (dhcp claimed the coordinator's .6; metrics claimed dist-1's .10). **Fix:** turn the apply into a reconcile — enumerate addrs on the bridge iface, delete every in-range (172.16.0.0/12) `/32` that isn't the intended IP, add the intended one if missing. Never touches the primary /24 or out-of-range addresses; idempotent; error-tolerant (retries next tick). Adds an injectable exec seam for unit testing. Tests: 10/10 on the file, full agent suite green (329), tsc + biome clean, binary builds.
The agent claimed its member docker_bridge_ip (echoed by the coordinator as
mesh_member_docker_bridge_ip) as a /32 secondary on the docker bridge iface but
only ever ADDED it, never removing stale ones. After a host power-cycle docker
permutes primary IPs and the echoed value can transiently be another
container's current IP -> duplicate /32s on the bridge -> ARP blackholes. In
steady state a renumbered member also left its old /32 behind.

Make the /32 application a proper reconcile: enumerate current addresses on the
bridge iface, delete every in-range (172.16.0.0/12) /32 that is not the intended
IP, then ensure the intended /32 is present (idempotent; File exists = ok). The
docker-assigned primary /24 and any out-of-range addresses are never touched.

Refactors an injectable MeshBridgeExec seam (listBridgeAddrs + runIp) and a
resolveDevice override so the reconcile logic is unit-testable without iproute2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch is already included in the target branch. There is nothing to merge.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/agent-reconcile-stale-bridge-32:fix/agent-reconcile-stale-bridge-32
git switch fix/agent-reconcile-stale-bridge-32

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/agent-reconcile-stale-bridge-32
git switch fix/agent-reconcile-stale-bridge-32
git rebase main
git switch main
git merge --ff-only fix/agent-reconcile-stale-bridge-32
git switch fix/agent-reconcile-stale-bridge-32
git rebase main
git switch main
git merge --no-ff fix/agent-reconcile-stale-bridge-32
git switch main
git merge --squash fix/agent-reconcile-stale-bridge-32
git switch main
git merge --ff-only fix/agent-reconcile-stale-bridge-32
git switch main
git merge fix/agent-reconcile-stale-bridge-32
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!194
No description provided.