fix(coordinator): self-CPU watchdog — auto-capture profile + self-heal on sustained peg #195

Open
charles wants to merge 1 commit from fix/coordinator-cpu-self-watchdog into main
Owner

Per-instance self-CPU watchdog baked into octet-api: on a sustained main-thread peg it auto-captures an in-process JSC CPU profile + heap summary to the logs (component cpu-watchdog) to finally name the long-unidentified hot loop, then process.exit()s for a clean restart: unless-stopped recovery so the UI never stays degraded. Pure reducer evaluateWatchdog unit-tested (8 pass); tsc clean. Env-tunable OCTET_CPU_WATCHDOG_*. Already deployed to live coordinator-1 (image src-d28c5c3d, digest 2b399fa4): healthy, leader acquired, watchdog armed, heartbeats ~50ms, CPU ~12%.

Per-instance self-CPU watchdog baked into octet-api: on a sustained main-thread peg it auto-captures an in-process JSC CPU profile + heap summary to the logs (component `cpu-watchdog`) to finally name the long-unidentified hot loop, then process.exit()s for a clean `restart: unless-stopped` recovery so the UI never stays degraded. Pure reducer `evaluateWatchdog` unit-tested (8 pass); tsc clean. Env-tunable OCTET_CPU_WATCHDOG_*. Already deployed to live coordinator-1 (image src-d28c5c3d, digest 2b399fa4): healthy, leader acquired, watchdog armed, heartbeats ~50ms, CPU ~12%.
The coordinator octet-api process slowly climbs to a pegged core over weeks
(restart-curable; a JS-level main-thread hot loop we've never caught because the
peg takes weeks to build and a restart wipes the state). When it pegs, every
DB-backed request queues behind the saturated event loop (2-11s), so the UI's
client timeouts fire and panels render "can't connect" even though the API is up.

Add a per-instance, ungated self-CPU watchdog. Each octet-api process samples its
own main-thread CPU every 15s; on a sustained peg it:
  1. auto-captures an in-process JSC CPU profile + heap object-type summary to the
     logs (names the hot loop on the stripped compiled binary, unattended), then
  2. if the peg persists past a longer window, process.exit()s for a clean
     `restart: unless-stopped` recovery so the UI never stays degraded.

The decision core (evaluateWatchdog) is a pure reducer — capture-then-restart
ordering, cooldowns, and the min-uptime crash-loop guard are unit tested without
timers/process.exit/the profiler. Reuses the existing debug-cpu-profile service.

Tunable via OCTET_CPU_WATCHDOG* env; opt out of restart with
OCTET_CPU_WATCHDOG_RESTART=0, or the whole thing with OCTET_CPU_WATCHDOG=0.

This is a durable mitigation + an automatic root-cause capture: the next climb
will name the exact JS frame so the underlying loop can be fixed for good.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/coordinator-cpu-self-watchdog:fix/coordinator-cpu-self-watchdog
git switch fix/coordinator-cpu-self-watchdog

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/coordinator-cpu-self-watchdog
git switch fix/coordinator-cpu-self-watchdog
git rebase main
git switch main
git merge --ff-only fix/coordinator-cpu-self-watchdog
git switch fix/coordinator-cpu-self-watchdog
git rebase main
git switch main
git merge --no-ff fix/coordinator-cpu-self-watchdog
git switch main
git merge --squash fix/coordinator-cpu-self-watchdog
git switch main
git merge --ff-only fix/coordinator-cpu-self-watchdog
git switch main
git merge fix/coordinator-cpu-self-watchdog
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/projectoctet!195
No description provided.