Skip to content

perf(ci): reuse caches across runs instead of rebuilding from scratch - #85

Merged
raymondk merged 3 commits into
mainfrom
perf/ci-cache-reuse
Aug 13, 2026
Merged

perf(ci): reuse caches across runs instead of rebuilding from scratch#85
raymondk merged 3 commits into
mainfrom
perf/ci-cache-reuse

Conversation

@lwshang

@lwshang lwshang commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

release.yml never ran on main, so no PR could restore a cargo cache — every PR's first push built the whole dependency tree (3.3 min on x86_64-linux, 9.9 min on macos-15-intel). Meanwhile every PR and tag run wrote 4 × ~800MB into a 10GB repo budget that was already full, evicting ci.yml's caches too. The Docker layer cache was refreshed only on tag, so it went stale the moment a dependency change merged: #82 changed one shell script and still paid the 189-second cargo chef cook layer, because nix had landed the day before and no release had refreshed the cache since.

One commit each:

  • cancel superseded PR runs; tag and main runs are never cancelled
  • build main, and save the cargo cache from main only
  • refresh the Docker layer cache on every merge instead of weekly on tag

Tag runs now restore rather than build cold — the cache-assisted release build we chose deliberately.

Note: ef2ab2b's message overstates the Docker miss as hitting "every pull request". It misses on dependency-changing PRs (unavoidable — the recipe changed) and on every PR between such a change landing on main and the next release tag. This fixes the second case.

🤖 Generated with Claude Code

lwshang and others added 3 commits August 13, 2026 14:10
Neither workflow had a concurrency group, so pushing twice in quick
succession left both runs going: eleven jobs across the two workflows,
building a commit nobody will merge, competing for runners with the push
that replaced it.

Cancel only pull_request runs. A tag run publishes and a main run populates
the caches every other run restores from, so cancelling either would cost
more than it saves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…scratch

The release build has always been cached — setup-rust-toolchain runs
rust-cache internally and its `cache` input defaults to true — but the cache
went almost entirely unused, because this workflow never ran on main. A run
can restore only from its own ref or the default branch, so every PR's first
push started from empty: 3.3 min on x86_64-linux and 9.9 min on
macos-15-intel, against 0.2 min once a second push could reuse the PR's own
cache.

Two changes. Build main, so there is a cache to restore from at all. And run
rust-cache explicitly with `save-if` limited to main, because the writes were
the other half of the problem: each PR run and each tag run saved 4 × ~800MB,
a tag under a per-tag ref nothing can ever read, against a 10GB repo budget
that was full (10,005MB across 11 entries) and therefore evicting ci.yml's
caches too. Restricting writes to main leaves roughly 3.2GB for this
workflow and 2.6GB for ci.yml, with room to spare.

Tag runs now restore the cache rather than building cold, which is the
"cache-assisted release builds" trade we chose deliberately: the dependency
graph is pinned by Cargo.lock and fetched from immutable sources, and the
cache key folds in the toolchain, that lockfile and RUSTFLAGS. It also makes
the native path consistent with the container images, which have always been
assembled from a registry layer cache on tag runs.

The trade-off: a bump PR that pushes twice no longer gets an exact-key hit on
its second push, only main's partial restore. Those PRs are automated and
push once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cargo chef cook --release` — 189 seconds, the bulk of each of the four
docker jobs — missed on every pull request, including ones that touch no
dependency at all. Two causes compounding: the cache was written by tag runs
only, so it was up to a week stale and matched exactly one tree, and
`rust:1.97.1-slim-trixie` is a mutable tag that moved underneath it
(sha256:5c6f46a6 on 2026-08-03, sha256:3b287904 by 2026-08-11), which
invalidates every layer below the base image.

Write the cache from main pushes instead. A merge is frequent enough to keep
it close to what PRs build, and any base-image drift costs one main run
rather than every PR until the next release. PRs still never write it — a
release must not be assembled from layers a pull request produced — and the
tag write is dropped as redundant: a tag always points at a main commit that
has already cached it. Registry caches are not ref-scoped like Actions
caches, so a cache written from main is one every PR and tag run can read.

Also drops the `BUILDKIT_INLINE_CACHE=1` build-arg, which does nothing here
(no matching ARG in either Dockerfile, and it is a legacy-builder mechanism
that `cache-to: type=registry` replaces), and the `platform` attribute on
cache-to, which is not a registry cache exporter option — each matrix entry
already writes to its own per-arch cache ref.

Left alone deliberately: pinning the builder base image by digest. Once main
refreshes the cache on every merge, the drift window is one run, so pinning
would buy reproducibility rather than speed — and reproducibility is what we
decided not to prioritise here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lwshang
lwshang marked this pull request as ready for review August 13, 2026 18:43
@lwshang
lwshang requested a review from a team as a code owner August 13, 2026 18:43
Copilot AI lite review requested due to automatic review settings August 13, 2026 18:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the GitHub Actions workflows to make CI and release builds reuse caches across runs (instead of rebuilding dependencies and Docker layers from scratch), while avoiding cache churn from PR and tag runs.

Changes:

  • Add workflow-level concurrency to cancel superseded PR runs (without canceling main or tag runs).
  • Run release.yml on main pushes so it can populate reusable Cargo caches, and explicitly control when caches are saved.
  • Refresh/persist the Docker layer cache on merges to main (not only on tags), while keeping PR runs build-only.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.github/workflows/release.yml Runs on main to populate Cargo cache; adds PR-only concurrency cancelation; writes Docker layer cache from main only and keeps tag runs publishing-only.
.github/workflows/ci.yml Adds PR-only concurrency cancelation to reduce wasted CI capacity on superseded PR pushes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@raymondk
raymondk merged commit ab72ed5 into main Aug 13, 2026
22 checks passed
@raymondk
raymondk deleted the perf/ci-cache-reuse branch August 13, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants