fix(metrics/ui): dnsdist backend state as a gauge + cleaner config tables #84

Merged
charles merged 1 commit from fix/dns-traffic-state-gauge-ui into main 2026-07-31 18:01:58 +00:00
Owner

Fixes the "random up/down" (item #1)

Root cause: I'd made backend state a metric label on octet_dnsdist_backend_queries. Every state flip changed the series identity, so VictoriaMetrics kept both an up and a down series (+ stale/duplicate variants), and the instant query returned all of them — the same backend showing up and down, and rate() resetting on each flip.

Fix:

  • agent: emit a dedicated octet_dnsdist_backend_up gauge (1=up, 0=down), no state label; drop octet_state from the counter so it's stable (clean QPS rate) and state is one clean series per backend.
  • api: query the counter rate for QPS + the up-gauge for current health, joined by (dist, backend, pool, addr).
  • ui: redesigned Configuration section — one compact table per dnsdist with a Pool column, colored status dot (theme --chart tokens, no raw palette), dimmed down-rows, and an "N/M up" summary.

Rendered against live data — the page now shows the accurate current state (.7 backends up, .9 backends down) instead of a confusing flicker.

tsc + check:ui clean; agent/api tests updated for the gauge split.

Note on the other two items

  • Tolerant health checks (#2): I observed the .9 backends are stably down (9/9 samples), not flapping — so loosening checkTimeout/maxCheckFailures wouldn't help. It's a hard cross-host failure, not jitter.
  • Root-cause (#3): still blocked — TCP works, the UDP proxy-protocol'd health check fails, proxy-protocol-from already lists both dists, so it's not that ACL. Need a real DNS/proxy-protocol probe tool on a .7 box or shell on .9.

🤖 Generated with Claude Code

## Fixes the "random up/down" (item #1) Root cause: I'd made backend `state` a **metric label** on `octet_dnsdist_backend_queries`. Every state flip changed the series identity, so VictoriaMetrics kept **both** an `up` and a `down` series (+ stale/duplicate variants), and the instant query returned all of them — the same backend showing up **and** down, and `rate()` resetting on each flip. **Fix:** - **agent**: emit a dedicated `octet_dnsdist_backend_up` gauge (1=up, 0=down), no state label; drop `octet_state` from the counter so it's stable (clean QPS rate) and state is one clean series per backend. - **api**: query the counter rate for QPS + the up-gauge for current health, joined by (dist, backend, pool, addr). - **ui**: redesigned Configuration section — one compact table per dnsdist with a Pool column, colored status dot (theme `--chart` tokens, no raw palette), dimmed down-rows, and an "N/M up" summary. Rendered against live data — the page now shows the **accurate** current state (`.7` backends up, `.9` backends down) instead of a confusing flicker. `tsc` + `check:ui` clean; agent/api tests updated for the gauge split. ## Note on the other two items - **Tolerant health checks (#2):** I observed the `.9` backends are **stably down** (9/9 samples), not flapping — so loosening `checkTimeout`/`maxCheckFailures` wouldn't help. It's a hard cross-host failure, not jitter. - **Root-cause (#3):** still blocked — TCP works, the UDP proxy-protocol'd health check fails, `proxy-protocol-from` already lists both dists, so it's not that ACL. Need a real DNS/proxy-protocol probe tool on a .7 box or shell on .9. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The "random up/down" on the DNS traffic page was a metric-design bug: backend
`state` was a LABEL on the octet_dnsdist_backend_queries counter. Every state
flip changed the series identity, so VictoriaMetrics kept BOTH an up and a down
series (plus stale/duplicate variants), and the instant query returned all of
them — the same backend showing up AND down, and rate() resetting on each flip.

- agent: emit a dedicated `octet_dnsdist_backend_up` gauge (1=up, 0=down) with
  no state label; drop `octet_state` from the counter so it's stable (clean
  rate) and state is one series per backend.
- api: query the counter rate for QPS and the up gauge for current health,
  joined by (dist, backend, pool, addr). One clean state per backend.
- ui: redesign the Configuration section — one compact table per dnsdist with a
  Pool column, a colored status dot (theme --chart tokens, no raw palette),
  dimmed down-rows, and an "N/M up" summary.

tsc + check:ui clean; agent/api tests updated for the gauge split. Rendered
against live data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
charles deleted branch fix/dns-traffic-state-gauge-ui 2026-07-31 18:01:58 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!84
No description provided.