fix(dns): idempotent DDNS trigger attach — stop config applies deadlocking live DDNS #88

Merged
charles merged 1 commit from fix/ddns-trigger-idempotent-no-view-lock into main 2026-08-02 23:39:19 +00:00
Owner

Fixes the recurring DDNS timeouts + octet-dns-3/postgres deadlocks.

Root cause (confirmed from dns-3's deadlock log): PowerDNS's records is a VIEW — RFC 2136 DDNS writes flow through an INSTEAD OF trigger into dns_records. ensureDdnsOutboxOnClient ran DROP TRIGGER + CREATE TRIGGER on that view on every config apply, and CREATE/DROP TRIGGER takes an AccessExclusiveLock on the view. Because the apply takes that lock after its dns_records upserts, it forms a deadlock cycle with a concurrent PowerDNS DDNS insert (which holds an AccessShareLock on the view + a conflicting row). Postgres kills one; when PowerDNS is the victim, its update fails → Kea logs SERVFAIL (RCODE 2) or timed out.

Deadlock detail (dns-3): proc A insert into records (…) for a PTR; proc B waiting for AccessExclusiveLock on relation 458803 (the records view). dns-3 was hit hardest: first in the Kea D2 list (all DDNS load) + constant zone churn (frequent applies).

Fix: the trigger references ddns_outbox_instead_of_fn by name and that function is CREATE OR REPLACE'd every apply, so an existing trigger is always current — it only vanishes when create_pdns_schema_for_view's CREATE OR REPLACE VIEW drops it (a genuine regeneration). So (re)create it only when absent. A steady-state apply now takes no lock on the records view, eliminating the deadlock window; a regeneration still re-attaches it in the same transaction.

Relation to earlier PRs: #74 added a deadlock retry on the sync side, but PowerDNS (a separate binary) can't retry, so it kept losing. #87 (local-first D2 order) reduced dns-3's share of the load. This removes the collision itself.

Tests: mock-client unit tests (skip-when-present / create-when-absent / always-probes); existing ddns-outbox + local-dns-sync deadlock-retry tests still green; agent typecheck clean.

Takes effect on the next member (dns) image deploy.

🤖 Generated with Claude Code

**Fixes the recurring DDNS timeouts + `octet-dns-3/postgres` deadlocks.** **Root cause (confirmed from dns-3's deadlock log):** PowerDNS's `records` is a VIEW — RFC 2136 DDNS writes flow through an INSTEAD OF trigger into `dns_records`. `ensureDdnsOutboxOnClient` ran `DROP TRIGGER` + `CREATE TRIGGER` on that view on **every** config apply, and CREATE/DROP TRIGGER takes an **AccessExclusiveLock** on the view. Because the apply takes that lock *after* its `dns_records` upserts, it forms a deadlock cycle with a concurrent PowerDNS DDNS insert (which holds an AccessShareLock on the view + a conflicting row). Postgres kills one; when PowerDNS is the victim, its update fails → Kea logs SERVFAIL (RCODE 2) or `timed out`. Deadlock detail (dns-3): proc A `insert into records (…)` for a PTR; proc B waiting for `AccessExclusiveLock on relation 458803` (the `records` view). dns-3 was hit hardest: first in the Kea D2 list (all DDNS load) + constant zone churn (frequent applies). **Fix:** the trigger references `ddns_outbox_instead_of_fn` by name and that function is `CREATE OR REPLACE`'d every apply, so an existing trigger is always current — it only vanishes when `create_pdns_schema_for_view`'s `CREATE OR REPLACE VIEW` drops it (a genuine regeneration). So (re)create it **only when absent**. A steady-state apply now takes no lock on the `records` view, eliminating the deadlock window; a regeneration still re-attaches it in the same transaction. **Relation to earlier PRs:** #74 added a deadlock *retry* on the sync side, but PowerDNS (a separate binary) can't retry, so it kept losing. #87 (local-first D2 order) reduced dns-3's share of the load. This removes the collision itself. **Tests:** mock-client unit tests (skip-when-present / create-when-absent / always-probes); existing ddns-outbox + local-dns-sync deadlock-retry tests still green; agent typecheck clean. Takes effect on the next member (dns) image deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Root cause of the recurring `octet-dhcp-1/kea` "timed out waiting for response
to ... DNS server 172.16.0.3" + `octet-dns-3/postgres` "deadlock detected":
PowerDNS's `records` is a VIEW (RFC 2136 writes flow through an INSTEAD OF
trigger into dns_records). `ensureDdnsOutboxOnClient` ran DROP TRIGGER +
CREATE TRIGGER on that view on EVERY config apply, and CREATE/DROP TRIGGER takes
an AccessExclusiveLock on the view. Since the apply takes that exclusive lock
AFTER its dns_records row upserts, it forms a deadlock cycle with a concurrent
PowerDNS DDNS insert (which holds an AccessShareLock on the view and a
conflicting row). Postgres kills one; when PowerDNS is the victim its update
fails → Kea logs SERVFAIL (RCODE 2) or times out. dns-3 was hit hardest because
it was first in the Kea D2 server list (all DDNS load) and its zones churn
constantly (frequent applies).

The trigger references `ddns_outbox_instead_of_fn` by name and that function is
CREATE OR REPLACE'd every apply, so an existing trigger is always current — it
only vanishes when create_pdns_schema_for_view's CREATE OR REPLACE VIEW drops it
(a real regeneration). So (re)create it only when absent. A steady-state apply
now takes no lock on the records view at all, removing the deadlock window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!88
No description provided.