fix(images): add cc/c++ symlinks so cargo bootstrap can link - #2101
Conversation
The commit annotator runs `bazel query` against the IC repo. IC commit
3622b58e ("chore(bazel): bump rules_rust to bazelbuild/rules_rust@5f04c8c1",
dfinity/ic#10900) switched the rules_rust archive_override from the
official 0.71.3 release tarball to a raw source archive of an upstream
git commit. Release tarballs let crate_universe use a prebuilt
cargo-bazel binary; a git source archive has none, so rules_rust falls
back to the cargo_bazel_bootstrap repo rule and compiles cargo-bazel
from source with cargo. Cargo defaults to `cc` as its linker driver, so
the query started failing with:
error: linker `cc` not found
error: could not compile `serde` (build script)
leaving the annotator in a permanent retry loop and blocking release
notes generation.
`build-essential` and gcc are installed in the image, but /usr/bin/cc
and /usr/bin/c++ are update-alternatives symlinks created by the gcc
and g++ postinst scripts. rules_distroless only unpacks each .deb's
data archive and never runs maintainer scripts, so neither symlink has
ever existed -- it just was not exercised until `bazel query` had to
compile something. Add them explicitly, mirroring the existing
clang-symlinks workaround for the same root cause.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5700ed2 to
4e0c3b5
Compare
|
The red
Root cause: Fixed separately in #2102. Once that lands, this branch needs a rebase and a re-run. |
Problem
The commit annotator in the DRE cluster (
release-controller-564977cfbc-9v2n7, nsrelease-controllerondm1-dre1) is stuck in a permanent retry loop. The pod reports2/2 Runningwith 0 restarts, so it looks healthy — the failure is internal and silent:The
bazel querydies during the loading phase while bootstrapping the rules_rustcargo_bazel_bootstraprepo:Impact: master has 5 unannotated GuestOS and 5 unannotated HostOS commits, and the annotation API returns 404 for
guestos-changed, which blocks release notes generation.Trigger
The annotator bisected this for us. It annotated cleanly up to IC commit
7f6db4852b(2026-07-27 09:35 UTC) and dies on its direct child3622b58e71(merged 2026-07-27 10:36 UTC) — which is dfinity/ic#10900,chore(bazel): bump rules_rust to bazelbuild/rules_rust@5f04c8c1:IC switched the rules_rust
archive_overridefrom the official 0.71.3 release tarball to a raw source archive of an upstream git commit. Release tarballs let crate_universe use a prebuiltcargo-bazelbinary; a git source archive has none, so rules_rust falls back to thecargo_bazel_bootstraprepo rule and compiles cargo-bazel from source with cargo. Cargo defaults toccas its linker driver.Root cause
/usr/bin/ccdoes not exist in thecommit-annotatorimage. Confirmed in the running pod:build-essentialandgccare installed (dpkg -lconfirms), but/etc/alternatives/is empty./usr/bin/ccand/usr/bin/c++areupdate-alternativessymlinks created by the gcc/g++postinstscripts, andrules_distrolessonly unpacks each.deb's data archive without ever running maintainer scripts — so they have never existed in this image. Nothing changed on the DRE side; the gap was simply never exercised untilbazel queryhad to compile something.Note that IC's rules_rust pin looks temporary (#10900 also drops a local
rules_rust_dep_env.patch, suggesting they are tracking an unreleased upstream commit). When IC returns to a release tarball the bootstrap requirement goes away on its own, but the image should not depend on that.Fix
Add a
gcc-symlinkstar providingcc → gccandc++ → g++, and wire it into//images:ubuntu_24_04. This mirrors the existingclang-symlinkstarget, which is the same workaround for the same root cause.Verification
Rebuilt and ran the actual
commit-annotatorimage:In the live pod, adding only a
cc → gccsymlink toPATHmakes compile+link+run succeed — the exact step cargo was failing on.bazel queryagainst the IC repo was not re-run, so a second missing tool behind this one can't be ruled out. Notablylld/ld.lldare also absent from the image if the IC build ever asks for them.🤖 Generated with Claude Code