ci(linux): Linux job fails about 12 s in with no log since 2026-11-11 13:59 UTC (blocks PR #100) #101

Closed
opened 2026-11-11 14:13:58 +00:00 by charles · 3 comments
Owner

Seen on task/26-acceptance (PR #100), commits f327ade and 402cb49 (an empty re-run commit; the tree is identical).

What happens

Every Linux (x86_64) job since 13:59 UTC fails 12-14 s after it starts: runs 101, 102, 103, 104 (tasks 7152, 7154, 7155, 7157). The last green Linux job was run 100 (task 7150, PR #95, finished 13:45 UTC). Windows (x64) is unaffected (run 103 succeeded on 402cb49).

No ci-log-<run>-linux release exists for any of the four runs. The publish step needs curl, which the first step (Install system packages) installs, so the job dies in container start-up or in apt-get, before any repository code is checked out.

Why this is not the branch

  • .forgejo/workflows/ci.yaml is unchanged relative to task/25-packaging-release, whose Linux runs were green minutes earlier.
  • 12 s is too short to reach the checkout, let alone compile.
  • cargo xtask ci passes locally on f327ade.

Not verified

The actual cause: this Forgejo exposes no job-log API, so the step output is only visible in the web UI (https://git.forked.io/charles/fluxer-native/actions/runs/104). Candidates are the ubuntu:22.04 image pull, the apt mirrors, or disk space on the eqemu-build runner host.

To reproduce

Push any commit to a branch, or re-run the Linux job of run 104 from the web UI.

Effect

PR #100 (Task 26) cannot get ready-to-merge until a Linux run is green on its head.

Seen on `task/26-acceptance` (PR #100), commits f327ade and 402cb49 (an empty re-run commit; the tree is identical). **What happens** Every `Linux (x86_64)` job since 13:59 UTC fails 12-14 s after it starts: runs 101, 102, 103, 104 (tasks 7152, 7154, 7155, 7157). The last green Linux job was run 100 (task 7150, PR #95, finished 13:45 UTC). `Windows (x64)` is unaffected (run 103 succeeded on 402cb49). No `ci-log-<run>-linux` release exists for any of the four runs. The publish step needs `curl`, which the first step (`Install system packages`) installs, so the job dies in container start-up or in `apt-get`, before any repository code is checked out. **Why this is not the branch** - `.forgejo/workflows/ci.yaml` is unchanged relative to `task/25-packaging-release`, whose Linux runs were green minutes earlier. - 12 s is too short to reach the checkout, let alone compile. - `cargo xtask ci` passes locally on f327ade. **Not verified** The actual cause: this Forgejo exposes no job-log API, so the step output is only visible in the web UI (https://git.forked.io/charles/fluxer-native/actions/runs/104). Candidates are the `ubuntu:22.04` image pull, the apt mirrors, or disk space on the `eqemu-build` runner host. **To reproduce** Push any commit to a branch, or re-run the Linux job of run 104 from the web UI. **Effect** PR #100 (Task 26) cannot get `ready-to-merge` until a Linux run is green on its head.
Author
Owner

Diagnosis from the orchestrator (read-only checks, nothing changed on the cluster or network):

  • Runner-wide, not branch-specific: a probe branch at the head of task/25-packaging-release (green at 13:45 UTC) failed the Linux job the same way at 14:16 UTC.
  • The job container starts and exits after about 12 s (dind log of pod forgejo-runner-build); runner disk is at 22 %.
  • The LAN resolvers (10.2.0.7 / 10.2.0.9) answer SERVFAIL for archive.ubuntu.com and security.ubuntu.com, from the workstation and from inside the runner pod (wget: bad address). 1.1.1.1 and 9.9.9.9 resolve them; 8.8.8.8 timed out from the workstation. github.com, crates.io, static.rust-lang.org resolve normally.
  • The Linux job begins with apt-get update, which presumably fails on that; the log-publishing step needs curl, which is not installed yet at that point, which would explain why no ci-log-* release appears.

Nothing to fix in the repo. Once the names resolve again the Linux job of task/26-acceptance needs a re-run. Possible hardening for later (not done): have the failure step publish the log without depending on apt-installed curl.

Diagnosis from the orchestrator (read-only checks, nothing changed on the cluster or network): - Runner-wide, not branch-specific: a probe branch at the head of `task/25-packaging-release` (green at 13:45 UTC) failed the Linux job the same way at 14:16 UTC. - The job container starts and exits after about 12 s (dind log of pod `forgejo-runner-build`); runner disk is at 22 %. - The LAN resolvers (10.2.0.7 / 10.2.0.9) answer **SERVFAIL** for `archive.ubuntu.com` and `security.ubuntu.com`, from the workstation and from inside the runner pod (`wget: bad address`). 1.1.1.1 and 9.9.9.9 resolve them; 8.8.8.8 timed out from the workstation. `github.com`, `crates.io`, `static.rust-lang.org` resolve normally. - The Linux job begins with `apt-get update`, which presumably fails on that; the log-publishing step needs `curl`, which is not installed yet at that point, which would explain why no `ci-log-*` release appears. Nothing to fix in the repo. Once the names resolve again the Linux job of `task/26-acceptance` needs a re-run. Possible hardening for later (not done): have the failure step publish the log without depending on apt-installed curl.
Author
Owner

The LAN resolvers answer for archive.ubuntu.com / security.ubuntu.com again since about 16:54 UTC. Re-ran CI on task/26-acceptance (8b37da4): Linux and Windows both green. Nothing was changed on the cluster or network from this side; the cause of the SERVFAIL window is still unknown, so leaving this open for Charles.

The LAN resolvers answer for `archive.ubuntu.com` / `security.ubuntu.com` again since about 16:54 UTC. Re-ran CI on `task/26-acceptance` (8b37da4): Linux and Windows both green. Nothing was changed on the cluster or network from this side; the cause of the SERVFAIL window is still unknown, so leaving this open for Charles.
Author
Owner

No longer applies: LAN DNS SERVFAIL went away (comment of 16:54 UTC) and later runs are green; it was an infrastructure incident

Closed during the 2026-11-14 issue triage.

🤖 Generated with Claude Code

No longer applies: LAN DNS SERVFAIL went away (comment of 16:54 UTC) and later runs are green; it was an infrastructure incident Closed during the 2026-11-14 issue triage. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/fluxer-native#101
No description provided.