[CI] Retry the baseline MLIR cache restore once - #1080
Merged
Conversation
The baseline install is ~1.4 GB and these runners sit in different
regions, so the transfer is slow enough to be interrupted mid-flight:
Received 1010827264 of 1396566653 (72.4%), 1.6 MBs/sec
##[warning]Failed to restore: Server failed to authenticate the
request ... Authorization header ... signature.
Cache not found for input keys: mlir-install-...-4087a417...
The entry existed and matched - 1396566653 bytes is its exact size - but
the blob SAS auth failed partway through, and actions/cache reports that
as a plain miss. The run then built the baseline wheel against the PR's
own MLIR, where it cannot compile, and lost the vs-main comparison to
what was really a network fault. Seen on PR #1051, run 33117631657; the
same signature is on record for these runners from an earlier
investigation.
Retry the restore once. It costs nothing when the first attempt works,
and both attempts carry a timeout so a hung transfer cannot eat the job
budget instead.
coderfeli
approved these changes
Aug 31, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1071. A benchmark baseline was lost to a network fault, not to a missing cache entry.
What happened
On #1051 (run 33117631657),
Restore baseline MLIR cache:1396566653is the exact size of the entry sitting atrefs/heads/main:refs/heads/mainSo the key was right, the scope was right, the entry was found and 72% downloaded — then the blob SAS auth failed and
actions/cachereported it as a plain miss. The run fell back to building the baseline wheel against the PR MLIR, where a pin bump cannot compile, and the vs-main comparison was gone.Same run, the PR-scoped shared entry (1.42 GB) downloaded fine, so this is intermittent rather than a systematic refusal. The signature is already on record for these runners: they are spread across regions, and Azure blob SAS is sensitive to clock skew and to proxies on the path.
The change
One retry on the baseline restore, gated on the first attempt missing. No cost when the first works. Both attempts get
timeout-minutes: 25so a stalled transfer cannot eat the job budget instead — at 1.6 MB/s a 1.4 GB restore already takes ~15 minutes.What this does not fix
Restore shared MLIR cachehas the same exposure, and failing there is more expensive: a 45-minute cold LLVM rebuild. That is pre-existing behaviour, out of scope here, and the durable fix is the one an earlier investigation landed for the test jobs — move off the cache auth path onto artifacts.