fix(coordinator): self-CPU watchdog — auto-capture profile + self-heal on sustained peg #195
Open
charles
wants to merge 1 commit from
fix/coordinator-cpu-self-watchdog into main
pull from: fix/coordinator-cpu-self-watchdog
merge into: charles:main
charles:main
charles:fix/lattice-aux-address-reserve-sidecar-ip
charles:fix/ip-refresh-two-phase-converge
charles:fix/agent-reconcile-stale-bridge-32
charles:fix/coordinator-phantom-updating-badge
charles:fix/soa-serial-content-drift
charles:diag/debug-profile-token-file
charles:diag/coordinator-cpu-heap-profile
charles:fix/coordinator-cloud-credentials-key-seed
charles:fix/coordinator-process-stats-comm
charles:fix/lattice-dot2-aux-reservation
charles:fix/docker-bridge-ip-permutation-livelock
charles:design/member-mesh-dns-wiring
charles:design/mesh-dns-sidecar
charles:design/control-plane-resolver-adoption
charles:fix/reconcile-epoch-recreate-and-discovery-volume
charles:feat/mesh-resolver-enablement-wiring
charles:fix/mesh-resolver-coordinator-discovery
charles:fix/mesh-resolver-bind-loopback
charles:fix/mesh-resolver-publish-shadow-and-loopback
charles:design/sidecar-mesh-name-resolution
charles:design/split-fresh-install-provisioning
charles:design/sidecar-managed-coordinators
charles:feat/no-backwards-updates-coordinator
charles:feat/no-backwards-image-updates
charles:hotfix/liveness-flush-cadence
charles:hotfix/leader-election-uuid-and-holder-endpoint-repair
charles:feat/cheap-liveness
charles:design/edge-homing-routing
charles:design/core-edge-state-partitioning
charles:feat/discovery-vlan-attachments-stage3-ui
charles:feat/discovery-vlan-attachments-stage2
charles:fix/discovery-lattice-network
charles:feat/ci-closure-fingerprint
charles:feat/discovery-probe-routing
charles:feat/discovery-probe-role
charles:docs/discovery-probe-role-plan
charles:feat/discovery-synced-inventory
charles:fix/dns-record-ip-address-fk-safe
charles:fix/ddns-dedup-unique-index
charles:fix/discovery-addresses-sync
charles:fix/blocklist-fetch-fields-node-local
charles:fix/oci-operational-tags-node-local
charles:fix/secondary-database-replica-mode
charles:fix/database-member-id-env-persists
charles:fix/member-install-use-configured-registry
charles:fix/wireguard-keygen-nonavx2
charles:fix/reconciler-preserve-database-joiner
charles:fix/member-install-mesh-endpoint-split
charles:feat/overview-dns-flow-card
charles:fix/dns-push-per-zone-cache-convergence
charles:feat/kea-decline-probation-1h
charles:fix/oci-drift-null-served-false-positive
charles:ui/dnsdist-flow-trunk-fork
charles:ui/dnsdist-flow-merge-pool-bands
charles:ui/dnsdist-flow-sankey-ribbons
charles:fix/dns-traffic-links-fit-node-box
charles:feat/dns-traffic-flow-no-overlap
charles:feat/dns-traffic-ribbon-split
charles:feat/dns-traffic-sankey-view
charles:fix/sidecar-report-desired-digest
charles:fix/dns-traffic-peer-label-prefer-clean
charles:feat/recursor-dnssec-validate
charles:feat/dns-traffic-label-known-peers
charles:feat/dns-traffic-node-boxes
charles:feat/dns-traffic-group-by-node
charles:feat/dns-traffic-dist-lb-policy
charles:feat/dns-traffic-source-cidr-label
charles:feat/dns-traffic-source-subnets
charles:fix/ddns-trigger-idempotent-no-view-lock
charles:fix/ddns-local-first-server-order
charles:fix/dns-traffic-legend-idle-vs-down
charles:fix/dns-traffic-state-gauge-ui
charles:feat/member-detail-traffic-panel
charles:fix/remote-host-metrics-mesh-url
charles:fix/apex-ns-push-on-member-change
charles:fix/reconcile-transient-flood
charles:fix/log-server-report-real-ip
charles:fix/shared-log-read-token
charles:fix/sidecar-member-routes-hostname
charles:fix/ui-remote-member-ip-labels
charles:fix/remote-log-candidate-ip
charles:fix/remote-member-vpn-endpoint
charles:fix/host-install-live-cert
charles:fix/reconcile-adopt-unlabeled-orphan
charles:fix/agent-baseline-and-host-install-ca
charles:fix/dns-pdns-schema-apply-deadlock
charles:fix/dns-pgbouncer-backend-recycle
charles:fix/host-tile-24h-bucket-resolution
charles:fix/dns-pg-backend-recycle
charles:fix/ui-heartbeat-container-mem
charles:fix/whole-mem-ui-and-coordinator
charles:fix/heartbeat-schema-container-mem
charles:feat/host-tile-whole-service-memory
charles:fix/dns-postgres-malloc-arena
charles:fix/discovery-ui-response-schemas
charles:fix/discovery-sidebar-nav-privilege
charles:fix/discovery-neighbor-read-after-ping
charles:fix/discovery-test-fetch-mock-type
charles:fix/discovery-coordinator-base-fallback
charles:feature/auto-discovery
charles:docs/auto-discovery-spec
charles:fix/reconcile-transient-ddns-autoresolve
charles:fix/kea-ddns-apply-contention
charles:fix/chart-tooltip-timestamps
charles:feat/host-metrics
charles:feat/blocklist-entries-browser
charles:fix/ns-list-reconcile-idempotent
charles:fix/oci-drift-placeholder-soa
charles:fix/reconcile-dynamic-ns-blindspot
charles:fix/errors-tile-full-width-text
charles:fix/sparkline-range-tooltip
charles:feat/dashboard-row-shuffle
charles:fix/reconcile-envelope-and-auth-noise
charles:fix/noc-tiles-live-data
charles:fix/topology-gutter-routing
charles:feature/topology-circuit-map
charles:fix/ddns-zone-push-coalesce
charles:fix/vector-clock-loose-index-scan
charles:fix/event-log-retention-ddns-batching
charles:fix/update-image-reprobe-and-ddns-timeout
charles:fix/noc-board-height-tune
charles:fix/noc-board-style-match
charles:fix/stale-push-error-noise
charles:fix/ci-image-digest-churn
charles:worktree-dashboard-noc-overhaul
charles:fix/sidecar-self-swap-helper-auth
charles:fix/bootstrap-slim-image-bun-paths
charles:feat/mesh-hosts-container-errors
charles:fix/octet-log-exc-info
charles:ci/add-metrics-logs-images
charles:fix/sidecar-swap-latch-reset
charles:fix/registry-poller-namespace
charles:ci/fix-build-images-singlejob-main
charles:fix/dhcp-config-reconcile-false-positives
charles:ci/build-images-singlejob
charles:feature/ui-closest-coordinator-api
charles:fix/dhcp-kea-socket-retry-and-lattice-ip-race
charles:feat/tls-multi-cert-reconcile
charles:feat/tls-cert-management
charles:fix/sidecar-logs-streaming-no-timeout
charles:fix/entity-table-and-permanent-env
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
charles/projectoctet!195
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/coordinator-cpu-self-watchdog"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Per-instance self-CPU watchdog baked into octet-api: on a sustained main-thread peg it auto-captures an in-process JSC CPU profile + heap summary to the logs (component
cpu-watchdog) to finally name the long-unidentified hot loop, then process.exit()s for a cleanrestart: unless-stoppedrecovery so the UI never stays degraded. Pure reducerevaluateWatchdogunit-tested (8 pass); tsc clean. Env-tunable OCTET_CPU_WATCHDOG_*. Already deployed to live coordinator-1 (image src-d28c5c3d, digest 2b399fa4): healthy, leader acquired, watchdog armed, heartbeats ~50ms, CPU ~12%.The coordinator octet-api process slowly climbs to a pegged core over weeks (restart-curable; a JS-level main-thread hot loop we've never caught because the peg takes weeks to build and a restart wipes the state). When it pegs, every DB-backed request queues behind the saturated event loop (2-11s), so the UI's client timeouts fire and panels render "can't connect" even though the API is up. Add a per-instance, ungated self-CPU watchdog. Each octet-api process samples its own main-thread CPU every 15s; on a sustained peg it: 1. auto-captures an in-process JSC CPU profile + heap object-type summary to the logs (names the hot loop on the stripped compiled binary, unattended), then 2. if the peg persists past a longer window, process.exit()s for a clean `restart: unless-stopped` recovery so the UI never stays degraded. The decision core (evaluateWatchdog) is a pure reducer — capture-then-restart ordering, cooldowns, and the min-uptime crash-loop guard are unit tested without timers/process.exit/the profiler. Reuses the existing debug-cpu-profile service. Tunable via OCTET_CPU_WATCHDOG* env; opt out of restart with OCTET_CPU_WATCHDOG_RESTART=0, or the whole thing with OCTET_CPU_WATCHDOG=0. This is a durable mitigation + an automatic root-cause capture: the next climb will name the exact JS frame so the underlying loop can be fixed for good. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.