fix(reconcile): stop dhcp_config false-positive flag storm #3

Merged
charles merged 1 commit from fix/dhcp-config-reconcile-false-positives into main 2026-06-30 18:37:00 +00:00
Owner

Problem

The live grid accumulated 31,703 unresolved reconciliation_events of type flagged_discrepancy, all for dhcp_config, growing ~42/hour and never converging — surfacing as a flood of "conflicts pending review".

Root cause

ReconciliationEngine.runAdapterCycle skips a domain when a member reports no fingerprints in metadata.reconcile_fingerprints — except it special-cased dhcp_config to flag anyway. But the member agent never computes dhcp_config fingerprints (packages/agent/src/services/reconcile-fingerprints.ts only emits dns_records and dhcp_leases), so every DHCP member always reports an empty dhcp_config set. The engine therefore flagged every coordinator scope as coordinator_only drift on every 10-minute tick — comparing against data that structurally cannot exist. These never cleared: createBatch has no dedup and pruneOldEvents only removes resolved flagged events.

Fix

Remove the exception so dhcp_config behaves like every other domain: no member data => skip. Guard extracted into hasMemberDataToReconcile with a regression test. The adapter stays wired so it activates cleanly once member-side dhcp_config fingerprinting is implemented.

Verification

  • New regression test (6 pass), adjacent reconcile-adapter tests pass (15 pass), typecheck + biome clean.
  • Live grid: the 31,703 existing false-positive rows were deleted (pending count -> 0). They reaccrue ~7/tick until this deploys.

Follow-ups (not in this PR)

  • Member-side dhcp_config fingerprinting is unimplemented (domain is a no-op until it lands).
  • Idempotency/prune gap: unresolved flagged_discrepancy rows are re-inserted every tick and never pruned; any real persistent drift in dns_records/dhcp_leases would flood the same way.
## Problem The live grid accumulated 31,703 unresolved `reconciliation_events` of type `flagged_discrepancy`, all for `dhcp_config`, growing ~42/hour and never converging — surfacing as a flood of "conflicts pending review". ## Root cause `ReconciliationEngine.runAdapterCycle` skips a domain when a member reports no fingerprints in `metadata.reconcile_fingerprints` — except it special-cased `dhcp_config` to flag anyway. But the member agent never computes dhcp_config fingerprints (`packages/agent/src/services/reconcile-fingerprints.ts` only emits `dns_records` and `dhcp_leases`), so every DHCP member always reports an empty dhcp_config set. The engine therefore flagged every coordinator scope as `coordinator_only` drift on every 10-minute tick — comparing against data that structurally cannot exist. These never cleared: `createBatch` has no dedup and `pruneOldEvents` only removes resolved flagged events. ## Fix Remove the exception so dhcp_config behaves like every other domain: no member data => skip. Guard extracted into `hasMemberDataToReconcile` with a regression test. The adapter stays wired so it activates cleanly once member-side dhcp_config fingerprinting is implemented. ## Verification - New regression test (6 pass), adjacent reconcile-adapter tests pass (15 pass), typecheck + biome clean. - Live grid: the 31,703 existing false-positive rows were deleted (pending count -> 0). They reaccrue ~7/tick until this deploys. ## Follow-ups (not in this PR) - Member-side dhcp_config fingerprinting is unimplemented (domain is a no-op until it lands). - Idempotency/prune gap: unresolved flagged_discrepancy rows are re-inserted every tick and never pruned; any real persistent drift in dns_records/dhcp_leases would flood the same way.
The reconciliation engine skips a domain when a member reports no
fingerprints in metadata.reconcile_fingerprints — except it special-cased
`dhcp_config` to flag anyway. But the member agent never computes
dhcp_config fingerprints (reconcile-fingerprints.ts only emits dns_records
and dhcp_leases), so every DHCP member always reports an empty dhcp_config
set. The engine therefore flagged every coordinator scope as
`coordinator_only` drift on every 10-minute tick, with no dedup and no
prune path (pruneOldEvents only removes *resolved* flagged events). On the
live grid this produced 31k+ permanent false "conflicts pending review",
growing ~42/hour.

Remove the exception: treat dhcp_config like every other domain — no
member data to compare against means skip, not flag. Extract the guard
into `hasMemberDataToReconcile` with a regression test. The adapter stays
wired (domainsForRole + interval) so it activates cleanly once member-side
dhcp_config fingerprinting is implemented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!3
No description provided.