fix(agent): reconcile the member bridge /32 (remove stale secondaries) #194
Open
charles
wants to merge 0 commits from
fix/agent-reconcile-stale-bridge-32 into main
pull from: fix/agent-reconcile-stale-bridge-32
merge into: charles:main
charles:main
charles:fix/coordinator-cpu-self-watchdog
charles:fix/lattice-aux-address-reserve-sidecar-ip
charles:fix/ip-refresh-two-phase-converge
charles:fix/coordinator-phantom-updating-badge
charles:fix/soa-serial-content-drift
charles:diag/debug-profile-token-file
charles:diag/coordinator-cpu-heap-profile
charles:fix/coordinator-cloud-credentials-key-seed
charles:fix/coordinator-process-stats-comm
charles:fix/lattice-dot2-aux-reservation
charles:fix/docker-bridge-ip-permutation-livelock
charles:design/member-mesh-dns-wiring
charles:design/mesh-dns-sidecar
charles:design/control-plane-resolver-adoption
charles:fix/reconcile-epoch-recreate-and-discovery-volume
charles:feat/mesh-resolver-enablement-wiring
charles:fix/mesh-resolver-coordinator-discovery
charles:fix/mesh-resolver-bind-loopback
charles:fix/mesh-resolver-publish-shadow-and-loopback
charles:design/sidecar-mesh-name-resolution
charles:design/split-fresh-install-provisioning
charles:design/sidecar-managed-coordinators
charles:feat/no-backwards-updates-coordinator
charles:feat/no-backwards-image-updates
charles:hotfix/liveness-flush-cadence
charles:hotfix/leader-election-uuid-and-holder-endpoint-repair
charles:feat/cheap-liveness
charles:design/edge-homing-routing
charles:design/core-edge-state-partitioning
charles:feat/discovery-vlan-attachments-stage3-ui
charles:feat/discovery-vlan-attachments-stage2
charles:fix/discovery-lattice-network
charles:feat/ci-closure-fingerprint
charles:feat/discovery-probe-routing
charles:feat/discovery-probe-role
charles:docs/discovery-probe-role-plan
charles:feat/discovery-synced-inventory
charles:fix/dns-record-ip-address-fk-safe
charles:fix/ddns-dedup-unique-index
charles:fix/discovery-addresses-sync
charles:fix/blocklist-fetch-fields-node-local
charles:fix/oci-operational-tags-node-local
charles:fix/secondary-database-replica-mode
charles:fix/database-member-id-env-persists
charles:fix/member-install-use-configured-registry
charles:fix/wireguard-keygen-nonavx2
charles:fix/reconciler-preserve-database-joiner
charles:fix/member-install-mesh-endpoint-split
charles:feat/overview-dns-flow-card
charles:fix/dns-push-per-zone-cache-convergence
charles:feat/kea-decline-probation-1h
charles:fix/oci-drift-null-served-false-positive
charles:ui/dnsdist-flow-trunk-fork
charles:ui/dnsdist-flow-merge-pool-bands
charles:ui/dnsdist-flow-sankey-ribbons
charles:fix/dns-traffic-links-fit-node-box
charles:feat/dns-traffic-flow-no-overlap
charles:feat/dns-traffic-ribbon-split
charles:feat/dns-traffic-sankey-view
charles:fix/sidecar-report-desired-digest
charles:fix/dns-traffic-peer-label-prefer-clean
charles:feat/recursor-dnssec-validate
charles:feat/dns-traffic-label-known-peers
charles:feat/dns-traffic-node-boxes
charles:feat/dns-traffic-group-by-node
charles:feat/dns-traffic-dist-lb-policy
charles:feat/dns-traffic-source-cidr-label
charles:feat/dns-traffic-source-subnets
charles:fix/ddns-trigger-idempotent-no-view-lock
charles:fix/ddns-local-first-server-order
charles:fix/dns-traffic-legend-idle-vs-down
charles:fix/dns-traffic-state-gauge-ui
charles:feat/member-detail-traffic-panel
charles:fix/remote-host-metrics-mesh-url
charles:fix/apex-ns-push-on-member-change
charles:fix/reconcile-transient-flood
charles:fix/log-server-report-real-ip
charles:fix/shared-log-read-token
charles:fix/sidecar-member-routes-hostname
charles:fix/ui-remote-member-ip-labels
charles:fix/remote-log-candidate-ip
charles:fix/remote-member-vpn-endpoint
charles:fix/host-install-live-cert
charles:fix/reconcile-adopt-unlabeled-orphan
charles:fix/agent-baseline-and-host-install-ca
charles:fix/dns-pdns-schema-apply-deadlock
charles:fix/dns-pgbouncer-backend-recycle
charles:fix/host-tile-24h-bucket-resolution
charles:fix/dns-pg-backend-recycle
charles:fix/ui-heartbeat-container-mem
charles:fix/whole-mem-ui-and-coordinator
charles:fix/heartbeat-schema-container-mem
charles:feat/host-tile-whole-service-memory
charles:fix/dns-postgres-malloc-arena
charles:fix/discovery-ui-response-schemas
charles:fix/discovery-sidebar-nav-privilege
charles:fix/discovery-neighbor-read-after-ping
charles:fix/discovery-test-fetch-mock-type
charles:fix/discovery-coordinator-base-fallback
charles:feature/auto-discovery
charles:docs/auto-discovery-spec
charles:fix/reconcile-transient-ddns-autoresolve
charles:fix/kea-ddns-apply-contention
charles:fix/chart-tooltip-timestamps
charles:feat/host-metrics
charles:feat/blocklist-entries-browser
charles:fix/ns-list-reconcile-idempotent
charles:fix/oci-drift-placeholder-soa
charles:fix/reconcile-dynamic-ns-blindspot
charles:fix/errors-tile-full-width-text
charles:fix/sparkline-range-tooltip
charles:feat/dashboard-row-shuffle
charles:fix/reconcile-envelope-and-auth-noise
charles:fix/noc-tiles-live-data
charles:fix/topology-gutter-routing
charles:feature/topology-circuit-map
charles:fix/ddns-zone-push-coalesce
charles:fix/vector-clock-loose-index-scan
charles:fix/event-log-retention-ddns-batching
charles:fix/update-image-reprobe-and-ddns-timeout
charles:fix/noc-board-height-tune
charles:fix/noc-board-style-match
charles:fix/stale-push-error-noise
charles:fix/ci-image-digest-churn
charles:worktree-dashboard-noc-overhaul
charles:fix/sidecar-self-swap-helper-auth
charles:fix/bootstrap-slim-image-bun-paths
charles:feat/mesh-hosts-container-errors
charles:fix/octet-log-exc-info
charles:ci/add-metrics-logs-images
charles:fix/sidecar-swap-latch-reset
charles:fix/registry-poller-namespace
charles:ci/fix-build-images-singlejob-main
charles:fix/dhcp-config-reconcile-false-positives
charles:ci/build-images-singlejob
charles:feature/ui-closest-coordinator-api
charles:fix/dhcp-kea-socket-retry-and-lattice-ip-race
charles:feat/tls-multi-cert-reconcile
charles:feat/tls-cert-management
charles:fix/sidecar-logs-streaming-no-timeout
charles:fix/entity-table-and-permanent-env
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
charles/projectoctet!194
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/agent-reconcile-stale-bridge-32"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of the durable fix set for the 2026-09-30 power-outage cold-boot incident (the grid did not self-recover; four chained failures required manual intervention).
Root cause (failure #3): octet-agent applied the coordinator-echoed
docker_bridge_ipas a/32secondary but never removed stale ones. When the echoed value was stale (failure #2) or a member renumbered, the stale /32 belonged to another container's current IP → duplicate IPs on the bridge → ARP blackholes (dhcp claimed the coordinator's .6; metrics claimed dist-1's .10).Fix: turn the apply into a reconcile — enumerate addrs on the bridge iface, delete every in-range (172.16.0.0/12)
/32that isn't the intended IP, add the intended one if missing. Never touches the primary /24 or out-of-range addresses; idempotent; error-tolerant (retries next tick). Adds an injectable exec seam for unit testing.Tests: 10/10 on the file, full agent suite green (329), tsc + biome clean, binary builds.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.