CI: Linux runner pod evicted for low disk on k8s2-build-01 #306

Open
opened 2026-11-12 14:22:52 +00:00 by charles · 0 comments
Owner

Linux CI runs 212, 213, 216 and 217 died without publishing a log. kubectl get events (read-only) shows why: the runner pod forgejo-runner-build-* on k8s2-build-01 was evicted for low ephemeral storage ("The node was low on resource: ephemeral-storage. Threshold quantity: 15820199745, available: 15100336Ki"). The node has about 100 GB of ephemeral storage, shared by every build on it. There are also dozens of Evicted runner pods left over from the last 13 days in namespace git.

On our side (4eec2aa on textchat) the Linux job now builds with CARGO_INCREMENTAL=0 and CARGO_PROFILE_DEV_DEBUG=line-tables-only, as the Windows job does, which should roughly halve its target dir. But the node itself is your infrastructure, so I have not touched it. Options, if it keeps happening:

  • more disk for k8s2-build-01, or an ephemeral-storage request/limit on the runner pod so the scheduler accounts for it;
  • a periodic docker system prune inside the runner's dind (the docker-gc sidecar may not be catching build caches);
  • deleting the old Evicted pods (cosmetic).

🤖 Generated with Claude Code

Linux CI runs 212, 213, 216 and 217 died without publishing a log. `kubectl get events` (read-only) shows why: the runner pod `forgejo-runner-build-*` on **k8s2-build-01** was **evicted for low ephemeral storage** ("The node was low on resource: ephemeral-storage. Threshold quantity: 15820199745, available: 15100336Ki"). The node has about 100 GB of ephemeral storage, shared by every build on it. There are also dozens of `Evicted` runner pods left over from the last 13 days in namespace `git`. On our side (4eec2aa on `textchat`) the Linux job now builds with `CARGO_INCREMENTAL=0` and `CARGO_PROFILE_DEV_DEBUG=line-tables-only`, as the Windows job does, which should roughly halve its target dir. But the node itself is your infrastructure, so I have not touched it. Options, if it keeps happening: - more disk for k8s2-build-01, or an `ephemeral-storage` request/limit on the runner pod so the scheduler accounts for it; - a periodic `docker system prune` inside the runner's dind (the `docker-gc` sidecar may not be catching build caches); - deleting the old `Evicted` pods (cosmetic). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#306
No description provided.