fix: second-host stabilization — DDNS deadlock/conflict flood + VM metrics co-location #74
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "worktree-fix-agent-baseline-hostinstall-ca"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This branch carries two independent fixes for second-host (10.2.0.9) stabilization.
1. DDNS deadlock churn + member_only conflict flood (
1b131cb6)After dns-3 joined on the second host, reconciliation "conflicts pending review" started growing again. Both causes trace to one root: DDNS bursts (Kea D2 at lease renewal) racing the coordinator's config apply on the member's local Postgres.
agent/local-dns-sync— the apply had a single deadlock retry; under heavy churn both attempts could lose the lock race, so the apply failed and rolled back, leaving records inconsistent (member-only drift). Now up to 3 retries with growing backoff[250, 500, 1000]ms. The apply transaction is idempotent (delete+upsert), so retrying is safe.api/dns-records-adapter— thecoordinator_only/toRemovebranches already skip transient (ddns) records; thetoAdd/member_onlybranch did not. Extended the transient-source skip there so DDNS records the member picked up from Kea D2 no longer flag as pending-review conflicts.2. VictoriaMetrics write-URL co-location keyed off
mesh_host_id(f59b071f)Remote members (dist-2, dns-3/4, recursor-2) got an unreachable VM write URL — the Docker mesh DNS alias of the metrics member on the coordinator host — so their resolver-sidecar failed to import metrics (~117 errors on dist-2).
Root cause:
victoriametricsEffectiveConsumerComputeIdfell back to the metrics member'scompute_instance_idwhen the consumer's was blank. Butcompute_instance_idis only set by member heartbeat and is empty for every member off the coordinator host — so that fallback turned "unknown" into a false co-location match, handing a remote consumer the co-located Docker alias instead of the bridge IP.Fix: decide co-location the same way dnsdist backends already do — prefer
mesh_host_id(assigned at mesh adoption, reliably populated for remotes), thencompute_instance_id, and only assume single-host when both signals are unknown on both sides. Dropped the compute-id fallback and threaded the consumer'smesh_host_idthrough all four config builders (dhcp/dist/dns/recursor).Verified against the live grid: metrics-1
mesh=926b0ab6(host .7), dist-2mesh=9e384c0f(host .9) → now resolves to the bridge IP172.16.1.7, reachable over WireGuard.Tests
local-dns-sync-deadlock-retry.test.ts: N-retry behavior (retries through 3 deadlocks then commits; propagates after 4 consecutive).dns-records-adapter.test.ts: addedmember_onlyDDNS-skip case.victoriametrics-sidecar.test.ts: rewrote for themesh_host_iddecision incl. the dist-2 regression case; removed the deleted fallback helper's tests.packages/api+packages/agenttsc --noEmitclean; related config-builder suites green (67 tests).🤖 Generated with Claude Code
fix(dns/reconcile): survive DDNS-heavy deadlock churn; stop member_only DDNS flooding reviewto fix: second-host stabilization — DDNS deadlock/conflict flood + VM metrics co-location