fix(sidecar): reserve lattice .2 via docker --ip-range on network create #192

Open
charles wants to merge 0 commits from fix/lattice-aux-address-reserve-sidecar-ip into main
Owner

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention).

Root cause (failure #1): the mesh-sidecar is pinned to <lattice>.2, but the Python runtime network-create paths (_ensure_lattice_bridge, _ensure_macvlan_network) created octet_lattice with only subnet+gateway and no .2 reservation. On a cold boot a floating container grabbed .2 first, so the sidecar crash-looped on 'Address already in use' and nothing drove mesh reconcile. This is the path that created the live networks (members at .3-.12), which is why the squat hit both hosts.

Fix: reserve the sidecar IP (dns_ip_from_subnet(subnet) = .2) via IPAMPool aux_addresses in both create paths, so IPAM never auto-assigns it (dynamic pool starts at .3, matching nextBridgeMemberIp). Existing networks are immutable, so the lattice path now also warns when a present network lacks the reservation (operators must recreate in a maintenance window to close the race on current hosts).

Supersedes #184 (which patched only the shell scripts and has gone stale against main).

Tests: full mesh-sidecar suite green (282); new test_lattice_bridge_aux.py + extended macvlan test.

Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention). **Root cause (failure #1):** the mesh-sidecar is pinned to `<lattice>.2`, but the Python runtime network-create paths (`_ensure_lattice_bridge`, `_ensure_macvlan_network`) created `octet_lattice` with only subnet+gateway and **no .2 reservation**. On a cold boot a floating container grabbed .2 first, so the sidecar crash-looped on 'Address already in use' and nothing drove mesh reconcile. This is the path that created the live networks (members at .3-.12), which is why the squat hit both hosts. **Fix:** reserve the sidecar IP (`dns_ip_from_subnet(subnet)` = .2) via IPAMPool `aux_addresses` in both create paths, so IPAM never auto-assigns it (dynamic pool starts at .3, matching nextBridgeMemberIp). Existing networks are immutable, so the lattice path now also **warns** when a present network lacks the reservation (operators must recreate in a maintenance window to close the race on current hosts). **Supersedes #184** (which patched only the shell scripts and has gone stale against main). Tests: full mesh-sidecar suite green (282); new `test_lattice_bridge_aux.py` + extended macvlan test.
After a host power-cycle the mesh-sidecar (pinned to <lattice>.2) could
crash-loop on "Address already in use" because the lattice bridge (and
static macvlan networks) were created without reserving .2, so Docker's
IPAM freely handed it to a floating container that raced to start first.

Reserve <subnet>.2 (via mesh_dns_ip.dns_ip_from_subnet) as an IPAM
aux_addresses entry when creating:
  - the host-local lattice bridge (_ensure_lattice_bridge)
  - static-subnet macvlan networks (_ensure_macvlan_network)

Docker IPAM is immutable on existing networks, so this only protects
freshly-created ones; for an already-present lattice bridge lacking the
reservation, warn the operator that a maintenance-window recreate is
needed to close the race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An IPAM aux-address reservation makes the address unusable even for an
explicit --ip (docker rejects 'connect --ip .2' against an aux-reserved
.2 with 'Address already in use'), so reserving the sidecar's .2 as an
aux address would STOP the sidecar binding its pin — verified live on
both hosts 2026-10-05. Switch both network-create paths to shrink the
dynamic pool to the subnet's upper half via --ip-range instead: floaters
allocate only from .128+, keeping .2 out of the dynamic pool, while the
sidecar's explicit --ip .2 still binds. Add dynamic_iprange_from_subnet()
helper; warn on existing networks that lack an --ip-range.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
charles changed title from fix(sidecar): reserve lattice .2 as an IPAM aux address on network create to fix(sidecar): reserve lattice .2 via docker --ip-range on network create 2026-10-05 20:44:53 +00:00
Author
Owner

Corrected approach (commit 39c6d8c4): switched from aux_addresses to docker --ip-range.

Verified LIVE on both hosts 2026-10-05 that an IPAM aux-address reservation makes the address unusable even for an explicit --ip — docker network connect --ip 172.16.0.2 against an aux-reserved .2 fails Address already in use, so reserving .2 as an aux address would STOP the sidecar binding its pin (worse than the bug). --ip-range instead shrinks only the dynamic pool to the subnet upper half (.128/25), so floaters cannot grab .2 while the sidecar explicit --ip .2 still binds. This matches the shell-script bootstrap path and what was applied to the live networks. Added dynamic_iprange_from_subnet(); existing-network branch now warns when no --ip-range is set. 285 sidecar tests pass.

Corrected approach (commit 39c6d8c4): switched from `aux_addresses` to docker `--ip-range`. Verified LIVE on both hosts 2026-10-05 that an IPAM **aux-address** reservation makes the address unusable even for an **explicit `--ip`** — `docker network connect --ip 172.16.0.2` against an aux-reserved `.2` fails `Address already in use`, so reserving `.2` as an aux address would STOP the sidecar binding its pin (worse than the bug). `--ip-range` instead shrinks only the *dynamic* pool to the subnet upper half (`.128/25`), so floaters cannot grab `.2` while the sidecar explicit `--ip .2` still binds. This matches the shell-script bootstrap path and what was applied to the live networks. Added `dynamic_iprange_from_subnet()`; existing-network branch now warns when no `--ip-range` is set. 285 sidecar tests pass.
This branch is already included in the target branch. There is nothing to merge.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/lattice-aux-address-reserve-sidecar-ip:fix/lattice-aux-address-reserve-sidecar-ip
git switch fix/lattice-aux-address-reserve-sidecar-ip

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/lattice-aux-address-reserve-sidecar-ip
git switch fix/lattice-aux-address-reserve-sidecar-ip
git rebase main
git switch main
git merge --ff-only fix/lattice-aux-address-reserve-sidecar-ip
git switch fix/lattice-aux-address-reserve-sidecar-ip
git rebase main
git switch main
git merge --no-ff fix/lattice-aux-address-reserve-sidecar-ip
git switch main
git merge --squash fix/lattice-aux-address-reserve-sidecar-ip
git switch main
git merge --ff-only fix/lattice-aux-address-reserve-sidecar-ip
git switch main
git merge fix/lattice-aux-address-reserve-sidecar-ip
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!192
No description provided.