arm64 CVEs surface weekly instead of at the release tag (WOR-873) - #490
Conversation
…WOR-873) The PR scan builds whatever the runner is — amd64. arm64 was gated in exactly one place: the release job, which fires only on a v* tag. Debian slim ships different package variants per architecture, so one fixable Medium present in arm64 and absent in amd64 turned every PR green and then failed a release with the tag already pushed and no image behind it. Adds an arm64 build + gate + informational scan to the weekly Monday run, scoped with `if: github.event_name == 'schedule'` so pull requests are not taxed ~8-12min of QEMU emulation to catch drift that originates upstream. Does NOT catch an arm64-only finding introduced AND released inside one week; the release gate remains the backstop. Written into the workflow. Two existing tests asserted exactly ONE gated and ONE informational step; they now assert the property holds for EVERY such step, which is what was meant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 48 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Docker security workflow builds and scans ChangesARM64 security scanning
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant QEMU
participant Buildx
participant Docker
participant Grype
participant Dockle
Scheduler->>QEMU: Set up ARM64 emulation
Scheduler->>Buildx: Set up ARM64 builder
Buildx->>Docker: Build linux/arm64 image
Docker->>Scheduler: Export ARM64 tarball
Scheduler->>Grype: Run gated and informational scans
Scheduler->>Dockle: Run lint step with always()
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_grype_ignore_expiry.py`:
- Around line 405-420: Extend the workflow assertions in the test around the
arm64 gate and tarball producers to require schedule-only execution for every
ARM64_TAR producer. Also identify the QEMU and Buildx setup steps and assert
each has if: github.event_name == 'schedule', preserving the existing platform
and producer validations.
- Around line 214-217: Update the scan-step validation loops in
tests/test_grype_ignore_expiry.py:214-217 and
tests/test_grype_ignore_expiry.py:230-233 so every selected step is checked. In
the loop over gated, assert each step’s only-fixed value is True; in the loop
over informational, assert each step’s output-format is "table" rather than
validating only one step.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 78d19f23-d3be-41d2-8859-1e6f4d78ba4f
📒 Files selected for processing (2)
.github/workflows/docker-security.ymltests/test_grype_ignore_expiry.py
oblangatas
changed the title
…Rs (WOR-873)
Two findings from CodeRabbit, both the same shape: I converted assertions to
loops but left two of them outside, so they still checked a single step.
- `only-fixed` was asserted on gated[0] only, and `output-format` on
whichever step the loop happened to end on. A second scan step could
violate either with the suite green. Both now inside their loops.
- the arm64 test checked the SCAN's schedule scoping but not the build's.
A change could run QEMU and an emulated arm64 build on every pull
request — the ~8-12min tax this design exists to avoid — while the
assertion stayed green. Now every QEMU, Buildx and tarball-producing
step must carry the schedule condition. Verified red by unscoping QEMU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An 11-gate review of this PR found seven defects in a change I had already
called ready. Five are fixed here; each now has a regression test verified to
fail when the fix is reverted. Previously only one of the seven was pinned —
I had been fixing instances and leaving the invariant unguarded.
- a push to main SILENTLY CANCELLED the weekly cron: both keyed on
`refs/heads/main` with cancel-in-progress. Cancelled runs render grey,
not red, so arm64 coverage could sit at zero for weeks looking fine.
`github.event_name` now in the concurrency key.
- the scan job had no `timeout-minutes` — 6h default, and this PR adds an
emulated arm64 build to it. Now 35, matching publish-docker.yml.
- deleting the cron left all 23 tests green with arm64 scanned nowhere.
The guards were green and the thing they guarded never ran.
- `workflow_dispatch` skipped arm64 entirely, so the break-glass this
workflow gained in WOR-852 answered "is main red?" for amd64 only.
Guards now fire on schedule OR dispatch — which also makes this PR
provable today instead of next Monday.
- one arm64 CVE aborted the job before Dockle, silently costing an
unrelated best-practice signal. Dockle now runs with `always()`.
Also closes WOR-873 AC 3: .grype.yaml now states that all 7 reachability
arguments were measured on amd64, why they should port to arm64 (no version
pins), and that nothing keeps that true.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_grype_ignore_expiry.py`:
- Around line 446-455: Update test_the_weekly_cron_still_exists to also assert
that "workflow_dispatch" is present in the parsed triggers, alongside the
existing schedule assertions, so manual workflow execution remains required.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: be84bbca-7252-4ef1-8f5f-4bf5f5d10463
📒 Files selected for processing (3)
.github/workflows/docker-security.yml.grype.yamltests/test_grype_ignore_expiry.py
oblangatas
changed the title
Two review findings. AC 1's second clause was unmet: the residual risk — arm64 checked weekly, so a mid-week tag can still fail at release — lived only in a YAML comment. scripts/tag-release.sh had zero mentions of arm64, scan or grype, and it is the thing a releaser actually runs. The note now prints at tag-push time and gives the recovery, including that a re-run does NOT help. And the arm64 tarball comment claimed the approach was 'the same shape the release job uses' as if that were precedent. publish-docker.yml's arm64 scan landed in WOR-871 on 2026-08-07 and last ran on 2026-07-28 — it had never executed either. Two unproven pipelines citing each other. Now stated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Feature status
The feature in one sentenceA maintainer can now find out whether the ARM image carries a vulnerability on a Monday morning — or on demand, any time — instead of discovering it mid-release with the tag already pushed. Real before/after# BEFORE — the pull-request scan builds whatever the runner is: Intel.
$ grep -c arm64 .github/workflows/docker-security.yml
0
# ARM's only check lived in the release job, which runs on a version tag.
# An ARM-only vulnerability meant: every PR green, then the release fails
# with the tag public, no image behind it, and the documented pull 404ing.Why this is harder than it sounds
Where we areDeviation measured against WOR-873 live. The ticket proposed weekly-only. Review found the workflow's existing manual trigger silently excluded ARM, so the guards now cover schedule and dispatch — closing a hole and making the change provable today rather than next Monday.
User stories
Risk dial
Where this fits in the productThird and last of the container-scanning fixes in this sprint, after the pull-request gate and the release gate. Together they mean: a new vulnerability fails a pull request, a release refuses to publish one, and the architecture nobody was watching is now watched. What remains is proving the image we scanned is byte-for-byte the image we shipped — tracked separately. What's left — and who's holding it
Step 1 gates the rest: ARM will never fail a pull request, only a weekly run or a release. If that trade is wrong, the fix is scanning ARM on every PR and paying ~8-12 minutes of emulation per contributor. Out of scope: re-measuring the seven accepted CVEs against ARM. They are now labelled as Intel-measured rather than silently universal. What I need from youApproval to merge, plus the branch decision in step 3. One deviation to confirm: the on-demand trigger was not in the ticket — it was added because review found the existing manual trigger silently skipped ARM. WOR-873 · PR #490 · dispatched run 31247125905 |
Proof of FunctionClaim under proofThe ARM build of the published container is checked for fixable Medium-or-higher vulnerabilities before a release, not during one. Threat defeated: an ARM-only CVE passing every pull request, then failing the release job after the version tag is already public — leaving a tag with no image behind it and the documented Provenance
Proof summary
Strongest layer — live CIBefore this run, no ARM scan had ever executed in this repository, on either workflow. The release job's ARM scan landed a day earlier and its most recent run predates it. So "the same shape the release job uses" was two unproven pipelines citing each other; this run is the first execution of either. The discriminating detail is the doubling. Why it can't be faked greenClaim 6 is proven by absence: this PR's own 41 checks contain no ARM steps, because the guards scope them off the pull-request path. So the same commit demonstrates both that ARM runs (on dispatch) and that it does not run on a PR (here) — the two halves of the design, from one artifact each. What this does NOT prove
Independent reproductiongh workflow run docker-security.yml --ref fix/wor-873-scan-arm64-before-releasegh run view 31247125905 --log | grep -c "ignoring 182 matches" # expect 2 — one per architecture
gh run view 31247125905 --log | grep -c "CVE-2026-0864" # expect 2 — sameuv run pytest tests/test_grype_ignore_expiry.py -q # 28 passedProof of Function — bae0339 · PR #490 · generated from live |
ARM64_WHEN references schedule AND workflow_dispatch, but only the cron was pinned. Removing workflow_dispatch left every ARM64_WHEN assertion passing while the only way to check arm64 between Mondays disappeared — the same hole this ticket exists to close, one trigger over. Caught by CodeRabbit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/tag-release.sh`:
- Around line 137-143: The recovery message in the tag-release failure guidance
should not claim reruns never help. Update the output around the “Recovery”
instructions in the release script to distinguish deterministic vulnerability
failures, which require remediation, from transient signing or runner failures,
where rerunning may succeed; preserve the existing remediation steps for the
deterministic case.
- Around line 138-139: Update the release instructions near the tag-push
guidance to dispatch docker-security.yml using the release tag variable, such as
"$tag", instead of the current main branch. Ensure the workflow is instructed to
check out and scan that tagged release ref.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9adeb983-28c8-4a1f-92e7-c8074f42556a
📒 Files selected for processing (3)
.github/workflows/docker-security.ymlscripts/tag-release.shtests/test_grype_ignore_expiry.py
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/workflows/docker-security.yml
- tests/test_grype_ignore_expiry.py
…ged ref (WOR-873)
Two CodeRabbit findings on the releaser note, both mine.
- it said a re-run 'will NOT help'. True for a CVE finding, false for a
transient failure (cosign 5xx, runner error) — which is exactly what
re-run exists for. I had learned the CVE case earlier today and wrote it
as universal. The workflow comment already drew the distinction; the
script flattened it.
- it pointed the recovery scan at --ref main, which dispatches against the
TIP of main, not the tagged commit. A releaser debugging a failed tag
would have scanned a different tree than the one that failed. Now uses
--ref $tag.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
GAP CLOSED — the arm64 gate provably blocks. The proof-of-function on this PR listed one claim as unproven: "no ARM CVE has ever actually failed a build. The gate exits 0 because everything is suppressed." That is now demonstrated. Branch Two changes isolate arm64 as the only possible cause:
Result: arm64 was the only gate able to fail the job, and it failed. Compare run 31247125905 — same workflow, all suppressions present, arm64 gate success. One variable differs. Note the same 90-day log-retention caveat as PR #476: this output is transcribed here because the run's logs expire around 2026-11-08. The proof branch is kept — the commit it cites is reachable only from it. |
1 participant
TL;DR
Features
Before / After
Summary
WOR-871 set the release gate to fail on fixable Medium-or-higher for both architectures, but the PR-path scan builds only amd64. ARM therefore met that bar for the first time during a release, where a failure leaves a pushed tag with no image behind it. WOR-873 adds an ARM build, gate and informational scan to the existing Monday cron and to the manual trigger, scoped so no pull request pays the emulation cost.
Setup
Debian slim ships different package variants per architecture. One fixable Medium present in ARM and absent in Intel was enough for every PR to pass and the release to fail — at the worst possible moment, with the recovery being "delete the tag and re-push" under time pressure. That pressure is exactly how an argued suppression turns into a rubber stamp.
Every reachability argument in
.grype.yamlwas also measured against amd64. The two architectures happened to agree when last checked; nothing kept them agreeing and nothing would have noticed when they stopped.The workflow already had a manual trigger, added so someone could answer "is main red right now?" between Mondays. It covered Intel only.
What
Why
How
Five steps in
docker-security.yml'sscanjob — QEMU, Buildx, an ARM build to a tarball, then the gate and an ignore-free informational scan reading it viadocker-archive:— each gated onscheduleorworkflow_dispatch.An 11-gate review found seven defects in the first version of this change, five of them fixed here and each now pinned by a test verified to fail when its fix is reverted: the cron shared a cancellation lane with pushes to main (a Monday push killed it mid-build, and cancelled runs render grey rather than red); the job had no
timeout-minuteswhile gaining a second build; deleting the cron left every test green with ARM scanned nowhere; the manual trigger skipped ARM entirely; and one ARM CVE aborted the job before Dockle, costing an unrelated signal..grype.yamlnow states that its seven reachability arguments were measured on amd64, why they should port to ARM (the rules carry no version pins), and that nothing keeps that true.Follow-ups
What this does NOT do
.grype.yaml's reachability arguments against ARM — those remain amd64 measurements applied to both, now labelled as such.publish-docker.yml's ARM scan still has never executed. Its last run predates the change that added it, so the release path's ARM gate remains unproven.Tests
tests/test_grype_ignore_expiry.py→ 28 passedpre-commiton changed files → passes, includingactionlint,zizmorandshellcheck🤖 Generated with Claude Code
Summary by CodeRabbit
Security
Documentation