fix: second-host stabilization — DDNS deadlock/conflict flood + VM metrics co-location #74

Merged
charles merged 2 commits from worktree-fix-agent-baseline-hostinstall-ca into main 2026-07-31 09:36:52 +00:00
Owner

This branch carries two independent fixes for second-host (10.2.0.9) stabilization.


1. DDNS deadlock churn + member_only conflict flood (1b131cb6)

After dns-3 joined on the second host, reconciliation "conflicts pending review" started growing again. Both causes trace to one root: DDNS bursts (Kea D2 at lease renewal) racing the coordinator's config apply on the member's local Postgres.

  • agent/local-dns-sync — the apply had a single deadlock retry; under heavy churn both attempts could lose the lock race, so the apply failed and rolled back, leaving records inconsistent (member-only drift). Now up to 3 retries with growing backoff [250, 500, 1000]ms. The apply transaction is idempotent (delete+upsert), so retrying is safe.
  • api/dns-records-adapter — the coordinator_only/toRemove branches already skip transient (ddns) records; the toAdd/member_only branch did not. Extended the transient-source skip there so DDNS records the member picked up from Kea D2 no longer flag as pending-review conflicts.

2. VictoriaMetrics write-URL co-location keyed off mesh_host_id (f59b071f)

Remote members (dist-2, dns-3/4, recursor-2) got an unreachable VM write URL — the Docker mesh DNS alias of the metrics member on the coordinator host — so their resolver-sidecar failed to import metrics (~117 errors on dist-2).

Root cause: victoriametricsEffectiveConsumerComputeId fell back to the metrics member's compute_instance_id when the consumer's was blank. But compute_instance_id is only set by member heartbeat and is empty for every member off the coordinator host — so that fallback turned "unknown" into a false co-location match, handing a remote consumer the co-located Docker alias instead of the bridge IP.

Fix: decide co-location the same way dnsdist backends already do — prefer mesh_host_id (assigned at mesh adoption, reliably populated for remotes), then compute_instance_id, and only assume single-host when both signals are unknown on both sides. Dropped the compute-id fallback and threaded the consumer's mesh_host_id through all four config builders (dhcp/dist/dns/recursor).

Verified against the live grid: metrics-1 mesh=926b0ab6 (host .7), dist-2 mesh=9e384c0f (host .9) → now resolves to the bridge IP 172.16.1.7, reachable over WireGuard.


Tests

  • local-dns-sync-deadlock-retry.test.ts: N-retry behavior (retries through 3 deadlocks then commits; propagates after 4 consecutive).
  • dns-records-adapter.test.ts: added member_only DDNS-skip case.
  • victoriametrics-sidecar.test.ts: rewrote for the mesh_host_id decision incl. the dist-2 regression case; removed the deleted fallback helper's tests.
  • packages/api + packages/agent tsc --noEmit clean; related config-builder suites green (67 tests).

🤖 Generated with Claude Code

This branch carries two independent fixes for second-host (10.2.0.9) stabilization. --- ## 1. DDNS deadlock churn + member_only conflict flood (`1b131cb6`) After dns-3 joined on the second host, reconciliation "conflicts pending review" started growing again. Both causes trace to one root: DDNS bursts (Kea D2 at lease renewal) racing the coordinator's config apply on the member's local Postgres. - **`agent/local-dns-sync`** — the apply had a single deadlock retry; under heavy churn both attempts could lose the lock race, so the apply failed and rolled back, leaving records inconsistent (member-only drift). Now up to 3 retries with growing backoff `[250, 500, 1000]ms`. The apply transaction is idempotent (delete+upsert), so retrying is safe. - **`api/dns-records-adapter`** — the `coordinator_only`/`toRemove` branches already skip transient (`ddns`) records; the `toAdd`/`member_only` branch did not. Extended the transient-source skip there so DDNS records the member picked up from Kea D2 no longer flag as pending-review conflicts. ## 2. VictoriaMetrics write-URL co-location keyed off `mesh_host_id` (`f59b071f`) Remote members (dist-2, dns-3/4, recursor-2) got an **unreachable** VM write URL — the Docker mesh DNS alias of the metrics member on the coordinator host — so their resolver-sidecar failed to import metrics (~117 errors on dist-2). **Root cause:** `victoriametricsEffectiveConsumerComputeId` fell back to the metrics member's `compute_instance_id` when the consumer's was blank. But `compute_instance_id` is only set by member heartbeat and is empty for every member off the coordinator host — so that fallback turned "unknown" into a **false co-location match**, handing a remote consumer the co-located Docker alias instead of the bridge IP. **Fix:** decide co-location the same way dnsdist backends already do — prefer `mesh_host_id` (assigned at mesh adoption, reliably populated for remotes), then `compute_instance_id`, and only assume single-host when **both** signals are unknown on **both** sides. Dropped the compute-id fallback and threaded the consumer's `mesh_host_id` through all four config builders (dhcp/dist/dns/recursor). Verified against the live grid: metrics-1 `mesh=926b0ab6` (host .7), dist-2 `mesh=9e384c0f` (host .9) → now resolves to the bridge IP `172.16.1.7`, reachable over WireGuard. --- ## Tests - `local-dns-sync-deadlock-retry.test.ts`: N-retry behavior (retries through 3 deadlocks then commits; propagates after 4 consecutive). - `dns-records-adapter.test.ts`: added `member_only` DDNS-skip case. - `victoriametrics-sidecar.test.ts`: rewrote for the `mesh_host_id` decision incl. the dist-2 regression case; removed the deleted fallback helper's tests. - `packages/api` + `packages/agent` `tsc --noEmit` clean; related config-builder suites green (67 tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Two fixes for the same root cause — DDNS bursts (Kea D2 at lease renewal)
racing the coordinator's config apply — that grew reconciliation conflicts
after a second host (dns-3 on 10.2.0.9) joined:

1. agent/local-dns-sync: replace the single deadlock retry with up to 3
   retries and growing backoff [250,500,1000]ms. One retry could still lose
   the race under heavy churn, failing the apply and leaving records
   inconsistent (member-only drift). The apply transaction is idempotent
   (delete+upsert), so retrying is safe.

2. api/dns-records-adapter: extend the transient-source (ddns) skip to the
   toAdd/member_only branch. DDNS records the member picked up from Kea D2
   that the coordinator doesn't mirror converge on their own and are no
   longer flagged as pending-review conflicts. Records the source in detail.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remote members (dist-2, dns-3/4, recursor-2 on the second host) got an
unreachable VictoriaMetrics write URL — the Docker mesh DNS alias of the
metrics member on the coordinator host — so their resolver-sidecar failed
to import metrics (~117 errors on dist-2).

Root cause: victoriametricsEffectiveConsumerComputeId fell back to the
metrics member's compute_instance_id when the consumer's was blank. But
compute_instance_id is only set by member heartbeat and is empty for every
member off the coordinator host. That fallback turned "unknown" into a
false co-location match, so a remote consumer was handed the co-located
Docker alias instead of the metrics bridge IP.

Fix: decide co-location the same way dnsdist backends already do — prefer
mesh_host_id (assigned at mesh adoption, reliably populated for remotes),
then compute_instance_id, and only assume single-host when BOTH signals are
unknown on BOTH sides. Drop the compute-id fallback. Thread the consumer's
mesh_host_id through all four callers (dhcp/dist/dns/recursor configs);
they already load it (MemberRow.mesh_host_id / BuildDistConfigInput.distMeshHostId).

Verified against the live grid: metrics-1 mesh=926b0ab6 (host .7),
dist-2 mesh=9e384c0f (host .9) → now resolves to the bridge IP, reachable
over WireGuard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
charles changed title from fix(dns/reconcile): survive DDNS-heavy deadlock churn; stop member_only DDNS flooding review to fix: second-host stabilization — DDNS deadlock/conflict flood + VM metrics co-location 2026-07-31 09:09:33 +00:00
charles deleted branch worktree-fix-agent-baseline-hostinstall-ca 2026-07-31 09:36:52 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!74
No description provided.