ci(linux): cache cargo home and target dir (#27) #534

Merged
charles merged 2 commits from ci/27-linux-cache into textchat 2026-11-14 21:15:30 +00:00
Owner

What changed

Workflow files only (plus one sentence in crates/fluxer-voice/README.md); no Rust code.

  • ci.yaml, job linux: container.volumes mounts fluxer-native-cargo:/cache/cargo and fluxer-native-target:/cache/target. CARGO_HOME=/cache/cargo/home, RUSTUP_HOME=/cache/cargo/rustup, CARGO_TARGET_DIR=/cache/target/ci, FLUXER_LIVEKIT_CACHE_DIR=/cache/cargo/downloads/livekit-server (tools/livekit-test-server.sh honours it). The pinned FFmpeg already lands in the target dir (xtask ffmpeg fetch → <target>/ffmpeg), so it's cached along with it.
    • New step Prepare the build cache: creates the dirs, adds $CARGO_HOME/bin to GITHUB_PATH, and logs whether each path is a mounted volume or a plain dir.
    • Install Rust: skipped when the pinned toolchain with rustfmt and clippy is present. A half-installed toolchain is uninstalled and reinstalled.
    • Install cargo-deny: skipped when cargo-deny --version is the pinned version. Otherwise it's unpacked to .partial and renamed into place.
    • Fetch livekit-server: a cached binary that doesn't run --version is dropped and fetched again.
    • New step Prune the build cache (disk guard, modelled on Windows): it prints df and the target dir's size, then deletes /cache/target/ci only (never the registry, the toolchain or the release dir) when free space is below MIN_FREE_GB=30 or the dir is above MAX_TARGET_GB=40. It records cold/warm.
    • New step Report disk use (always()): logs the sizes of target/debug/release/ffmpeg, CARGO_HOME, registry, git and RUSTUP_HOME, plus df. Log only, no release asset.
    • CARGO_INCREMENTAL=0 and the rest of the env are unchanged. The checkout is still a fresh git fetch.
  • release.yaml, job linux: same volumes, same CARGO_HOME/RUSTUP_HOME, its own CARGO_TARGET_DIR=/cache/target/release-build, the same idempotent Install Rust, a disk guard (MIN_FREE_GB=30, MAX_TARGET_GB=15) and Report disk use. Adds CARGO_INCREMENTAL=0, because the dir now persists and xtask's own dev build would otherwise keep incremental artifacts. The volume names are written as "fluxer\x2Dnative-…", which YAML reads as -. The literal would fail no_old_product_name_in_shipped_files (xtask/tests/package.rs), which bans the old name in release.yaml. The alternative is adding the two names to that test's ALLOWED_OLD, a Rust change this PR was not allowed to make.
  • Header comments of both files (the "not cached yet (#27)" note) are updated.

Locking (why nothing can wedge)

  • /cache/cargo/.install.lock (flock) serialises toolchain, cargo-deny and livekit installs between concurrent jobs, because rustup doesn't lock.
  • /cache/target/ci.lock (or release-build.lock) is held around prune, the FFmpeg fetch and the build. A second CI job waits instead of having its dir pruned under it. Cargo would serialise the two builds on its own target lock anyway.
  • While ci.lock is held, no other job can be running cargo deny, so any *.lock left in $CARGO_HOME/advisory-dbs must come from a killed run and is deleted.
  • All of these locks are flock(2), which the kernel releases when the holder dies. Cargo's registry and target locks work the same way, and cargo re-extracts crates that lack .cargo-ok.

What release and CI can share

The release build is the same cargo build --release --locked -p fluxer --no-default-features that CI's packaging test runs, so the release/ artifacts could in principle be shared (all but the app crate, which bakes in the version and channel). They are kept in separate dirs so that a concurrent CI job and release job don't wait on each other for the whole build. CI's debug/test artifacts are of no use to the release build. Shared: toolchain, registry, git checkouts.

Runner-side requirement

The runner ignores the mounts until both names are listed. Add them to the runner config and restart forgejo-runner:

container:
  valid_volumes:
    - fluxer-native-cargo
    - fluxer-native-target

(Keep any existing entries.) Docker creates the volumes on first use.

Without it

The jobs work exactly as before, uncached: /cache/* are plain container directories, every step creates what it needs, and the log shows cache: /cache/cargo NOT mounted (not in the runner's valid_volumes): cold build. Tested locally in ubuntu:22.04 both without volumes and with two throwaway named volumes: the second run printed rust: 1.98.1 from the cache, cargo-deny: 0.20.2 from the cache, disk: warm build.

How to measure

  1. After the runner change, dispatch ci.yaml (workflow_dispatch) on this branch. The first run is cold: expect ~50 min, and disk: cold build in the log.
  2. Dispatch it again on the same commit. It should be warm: rust: … from the cache, disk: warm build, with only the workspace's own crates recompiled where needed (fingerprints match), and no toolchain, crate, livekit or FFmpeg download.
  3. Compare the two job durations and the Report disk use lines (/cache/target/ci size → retune MIN_FREE_GB/MAX_TARGET_GB).
  4. Same for release.yaml (/cache/target/release-build).

Verified locally

  • Both files parse (PyYAML) and every run: block passes bash -n.
  • no_old_product_name_in_shipped_files, release_workflow_never_names_a_package_file_in_the_feed_steps, no_ci_workflow_writes_the_stable_feed, promote_stable (14) and package::libs::tests::declared_everywhere (3) all pass.
  • Not verified without the real runner: the actual mounts under forgejo-runner, warm build times, disk figures.

Refs #27

🤖 Generated with Claude Code

## What changed Workflow files only (plus one sentence in `crates/fluxer-voice/README.md`); no Rust code. - **`ci.yaml`, job `linux`**: `container.volumes` mounts `fluxer-native-cargo:/cache/cargo` and `fluxer-native-target:/cache/target`. `CARGO_HOME=/cache/cargo/home`, `RUSTUP_HOME=/cache/cargo/rustup`, `CARGO_TARGET_DIR=/cache/target/ci`, `FLUXER_LIVEKIT_CACHE_DIR=/cache/cargo/downloads/livekit-server` (`tools/livekit-test-server.sh` honours it). The pinned FFmpeg already lands in the target dir (`xtask ffmpeg fetch` → `<target>/ffmpeg`), so it's cached along with it. - New step **Prepare the build cache**: creates the dirs, adds `$CARGO_HOME/bin` to `GITHUB_PATH`, and logs whether each path is a mounted volume or a plain dir. - **Install Rust**: skipped when the pinned toolchain with rustfmt and clippy is present. A half-installed toolchain is uninstalled and reinstalled. - **Install cargo-deny**: skipped when `cargo-deny --version` is the pinned version. Otherwise it's unpacked to `.partial` and renamed into place. - **Fetch livekit-server**: a cached binary that doesn't run `--version` is dropped and fetched again. - New step **Prune the build cache** (disk guard, modelled on Windows): it prints `df` and the target dir's size, then deletes `/cache/target/ci` only (never the registry, the toolchain or the release dir) when free space is below `MIN_FREE_GB=30` or the dir is above `MAX_TARGET_GB=40`. It records `cold`/`warm`. - New step **Report disk use** (`always()`): logs the sizes of target/debug/release/ffmpeg, CARGO_HOME, registry, git and RUSTUP_HOME, plus `df`. Log only, no release asset. - `CARGO_INCREMENTAL=0` and the rest of the env are unchanged. The checkout is still a fresh `git fetch`. - **`release.yaml`, job `linux`**: same volumes, same CARGO_HOME/RUSTUP_HOME, its own `CARGO_TARGET_DIR=/cache/target/release-build`, the same idempotent Install Rust, a disk guard (`MIN_FREE_GB=30`, `MAX_TARGET_GB=15`) and Report disk use. Adds `CARGO_INCREMENTAL=0`, because the dir now persists and xtask's own dev build would otherwise keep incremental artifacts. The volume names are written as `"fluxer\x2Dnative-…"`, which YAML reads as `-`. The literal would fail `no_old_product_name_in_shipped_files` (xtask/tests/package.rs), which bans the old name in release.yaml. The alternative is adding the two names to that test's `ALLOWED_OLD`, a Rust change this PR was not allowed to make. - Header comments of both files (the "not cached yet (#27)" note) are updated. ### Locking (why nothing can wedge) - `/cache/cargo/.install.lock` (flock) serialises toolchain, cargo-deny and livekit installs between concurrent jobs, because rustup doesn't lock. - `/cache/target/ci.lock` (or `release-build.lock`) is held around prune, the FFmpeg fetch and the build. A second CI job waits instead of having its dir pruned under it. Cargo would serialise the two builds on its own target lock anyway. - While `ci.lock` is held, no other job can be running `cargo deny`, so any `*.lock` left in `$CARGO_HOME/advisory-dbs` must come from a killed run and is deleted. - All of these locks are flock(2), which the kernel releases when the holder dies. Cargo's registry and target locks work the same way, and cargo re-extracts crates that lack `.cargo-ok`. ### What release and CI can share The release build is the same `cargo build --release --locked -p fluxer --no-default-features` that CI's packaging test runs, so the `release/` artifacts could in principle be shared (all but the app crate, which bakes in the version and channel). They are kept in separate dirs so that a concurrent CI job and release job don't wait on each other for the whole build. CI's debug/test artifacts are of no use to the release build. Shared: toolchain, registry, git checkouts. ## Runner-side requirement The runner ignores the mounts until both names are listed. Add them to the runner config and restart forgejo-runner: ```yaml container: valid_volumes: - fluxer-native-cargo - fluxer-native-target ``` (Keep any existing entries.) Docker creates the volumes on first use. ## Without it The jobs work exactly as before, uncached: `/cache/*` are plain container directories, every step creates what it needs, and the log shows `cache: /cache/cargo NOT mounted (not in the runner's valid_volumes): cold build`. Tested locally in `ubuntu:22.04` both without volumes and with two throwaway named volumes: the second run printed `rust: 1.98.1 from the cache`, `cargo-deny: 0.20.2 from the cache`, `disk: warm build`. ## How to measure 1. After the runner change, dispatch `ci.yaml` (workflow_dispatch) on this branch. The first run is cold: expect ~50 min, and `disk: cold build` in the log. 2. Dispatch it again on the same commit. It should be warm: `rust: … from the cache`, `disk: warm build`, with only the workspace's own crates recompiled where needed (fingerprints match), and no toolchain, crate, livekit or FFmpeg download. 3. Compare the two job durations and the `Report disk use` lines (`/cache/target/ci` size → retune `MIN_FREE_GB`/`MAX_TARGET_GB`). 4. Same for `release.yaml` (`/cache/target/release-build`). ## Verified locally - Both files parse (PyYAML) and every `run:` block passes `bash -n`. - `no_old_product_name_in_shipped_files`, `release_workflow_never_names_a_package_file_in_the_feed_steps`, `no_ci_workflow_writes_the_stable_feed`, `promote_stable` (14) and `package::libs::tests::declared_everywhere` (3) all pass. - Not verified without the real runner: the actual mounts under forgejo-runner, warm build times, disk figures. Refs #27 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The Linux jobs of ci.yaml and release.yaml mount two named docker volumes,
fluxer-native-cargo at /cache/cargo (CARGO_HOME, RUSTUP_HOME, the
livekit-server download) and fluxer-native-target at /cache/target
(CARGO_TARGET_DIR /cache/target/ci for CI, /cache/target/release-build for
the release build, so the two never wait on each other's cargo lock).

Every install step is idempotent and skipped when cached (toolchain with
rustfmt/clippy, cargo-deny at the pinned version, livekit-server checked by
running it). Installs are serialised with flock on /cache/cargo/.install.lock,
builds per target dir on /cache/target/<dir>.lock; a disk guard modelled on
the Windows job drops the job's own target dir below MIN_FREE_GB free or above
MAX_TARGET_GB, and a "Report disk use" step logs the sizes on every run.

The runner mounts the volumes only once both names are in its
container.valid_volumes; until then /cache/* are plain container dirs and
the jobs build cold as before.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After pruning, the Linux jobs of ci.yaml and release.yaml now fail the
"Prune the build cache" step at once when /cache/target still has less
than DISK_FLOOR_GB (25) free ("runner low on disk: N GB free, need 25"),
instead of starting a build that runs out of disk halfway.

release.yaml writes its volume names plainly (no more `\x2D` trick); the
two names are in xtask/tests/package.rs's ALLOWED_OLD as Docker volume
names on the CI runner, not product text.

The runner's docker storage is now a 193 GB disk, so the target dir caps
go up: MAX_TARGET_GB 40 -> 60 for CI, 15 -> 25 for release.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
charles merged commit 0d7ce9ada7 into textchat 2026-11-14 21:15:30 +00:00
Sign in to join this conversation.
No description provided.