Skip to content

fix(images): add cc/c++ symlinks so cargo bootstrap can link - #2101

Merged
pietrodimarco-dfinity merged 1 commit into
mainfrom
pmarco/fix-annotator-cc-symlink
Jul 27, 2026
Merged

fix(images): add cc/c++ symlinks so cargo bootstrap can link#2101
pietrodimarco-dfinity merged 1 commit into
mainfrom
pmarco/fix-annotator-cc-symlink

Conversation

@pietrodimarco-dfinity

@pietrodimarco-dfinity pietrodimarco-dfinity commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Problem

The commit annotator in the DRE cluster (release-controller-564977cfbc-9v2n7, ns release-controller on dm1-dre1) is stuck in a permanent retry loop. The pod reports 2/2 Running with 0 restarts, so it looks healthy — the failure is internal and silent:

EE:annotate_object.3622b58e71bd74430a858b762a9e589de5449f5a.GuestOS — Annotation failed.  Aborting.
EE:annotator — Failed to annotate.  Retrying in 30 seconds.
subprocess.CalledProcessError: Command '[... 'bazel', 'query', ... 'deps(//ic-os/guestos/envs/prod:update-img.tar.zst)']' returned non-zero exit status 7

The bazel query dies during the loading phase while bootstrapping the rules_rust cargo_bazel_bootstrap repo:

error: linker `cc` not found
error: could not compile `serde` (build script) due to 1 previous error

Impact: master has 5 unannotated GuestOS and 5 unannotated HostOS commits, and the annotation API returns 404 for guestos-changed, which blocks release notes generation.

Trigger

The annotator bisected this for us. It annotated cleanly up to IC commit 7f6db4852b (2026-07-27 09:35 UTC) and dies on its direct child 3622b58e71 (merged 2026-07-27 10:36 UTC) — which is dfinity/ic#10900, chore(bazel): bump rules_rust to bazelbuild/rules_rust@5f04c8c1:

-    urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.71.3/rules_rust-0.71.3.tar.gz"],
+    strip_prefix = "rules_rust-5f04c8c14c2ffacfe10ad14e55298be21454be04",
+    urls = ["https://codeload.github.com/bazelbuild/rules_rust/tar.gz/5f04c8c14c2ffacfe10ad14e55298be21454be04"],

IC switched the rules_rust archive_override from the official 0.71.3 release tarball to a raw source archive of an upstream git commit. Release tarballs let crate_universe use a prebuilt cargo-bazel binary; a git source archive has none, so rules_rust falls back to the cargo_bazel_bootstrap repo rule and compiles cargo-bazel from source with cargo. Cargo defaults to cc as its linker driver.

Root cause

/usr/bin/cc does not exist in the commit-annotator image. Confirmed in the running pod:

cc: MISSING        gcc: /usr/bin/gcc        clang: /usr/bin/clang        ld: /usr/bin/ld

build-essential and gcc are installed (dpkg -l confirms), but /etc/alternatives/ is empty. /usr/bin/cc and /usr/bin/c++ are update-alternatives symlinks created by the gcc/g++ postinst scripts, and rules_distroless only unpacks each .deb's data archive without ever running maintainer scripts — so they have never existed in this image. Nothing changed on the DRE side; the gap was simply never exercised until bazel query had to compile something.

Note that IC's rules_rust pin looks temporary (#10900 also drops a local rules_rust_dep_env.patch, suggesting they are tracking an unreleased upstream commit). When IC returns to a release tarball the bootstrap requirement goes away on its own, but the image should not depend on that.

Fix

Add a gcc-symlinks tar providing cc → gcc and c++ → g++, and wire it into //images:ubuntu_24_04. This mirrors the existing clang-symlinks target, which is the same workaround for the same root cause.

Verification

Rebuilt and ran the actual commit-annotator image:

cc   /usr/bin/cc          c++  /usr/bin/c++
gcc  /usr/bin/gcc         g++  /usr/bin/g++
COMPILE_LINK_RUN_OK

In the live pod, adding only a cc → gcc symlink to PATH makes compile+link+run succeed — the exact step cargo was failing on.

⚠️ Not fully verified end-to-end: I confirmed the reported error is resolved, but the full bazel query against the IC repo was not re-run, so a second missing tool behind this one can't be ruled out. Notably lld/ld.lld are also absent from the image if the IC build ever asks for them.

🤖 Generated with Claude Code

@pietrodimarco-dfinity
pietrodimarco-dfinity requested a review from a team as a code owner July 27, 2026 14:28
The commit annotator runs `bazel query` against the IC repo. IC commit
3622b58e ("chore(bazel): bump rules_rust to bazelbuild/rules_rust@5f04c8c1",
dfinity/ic#10900) switched the rules_rust archive_override from the
official 0.71.3 release tarball to a raw source archive of an upstream
git commit. Release tarballs let crate_universe use a prebuilt
cargo-bazel binary; a git source archive has none, so rules_rust falls
back to the cargo_bazel_bootstrap repo rule and compiles cargo-bazel
from source with cargo. Cargo defaults to `cc` as its linker driver, so
the query started failing with:

    error: linker `cc` not found
    error: could not compile `serde` (build script)

leaving the annotator in a permanent retry loop and blocking release
notes generation.

`build-essential` and gcc are installed in the image, but /usr/bin/cc
and /usr/bin/c++ are update-alternatives symlinks created by the gcc
and g++ postinst scripts. rules_distroless only unpacks each .deb's
data archive and never runs maintainer scripts, so neither symlink has
ever existed -- it just was not exercised until `bazel query` had to
compile something. Add them explicitly, mirroring the existing
clang-symlinks workaround for the same root cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pietrodimarco-dfinity
pietrodimarco-dfinity force-pushed the pmarco/fix-annotator-cc-symlink branch from 5700ed2 to 4e0c3b5 Compare July 27, 2026 14:33
@pietrodimarco-dfinity
pietrodimarco-dfinity merged commit c44d1d8 into main Jul 27, 2026
6 of 9 checks passed
@pietrodimarco-dfinity
pietrodimarco-dfinity deleted the pmarco/fix-annotator-cc-symlink branch July 27, 2026 15:07
@pietrodimarco-dfinity

Copy link
Copy Markdown
Contributor Author

The red Release controller pipeline here is not caused by this change.

//release-controller:integration_tests failed on test_reconciler_reconciles_without_error_already_submitted_proposals, which asserts nothing this PR touches. Two pushes on this branch have byte-identical trees (images/BUILD.bazel blob 55124ab4 at both 5700ed25 and 4e0c3b5a; only the commit message differs) and the pipeline passed on the first at 14:29 UTC and failed on the second at 14:33 UTC. The same test also failed on pmarco/guestos-launch-measurements at 10:50 UTC today.

Root cause: MockDashboard._fake_proposal always writes hostos_version_to_elect, whatever os_type it is handed, so the GuestOS side of the fixture never registers and falls through to the real dre proposals filter -t ic-os-version-election against the live governance canister. That query defaults to --limit 100, and the test hardcodes proposal 138708 (GuestOS for 206b61a8, executed 2025-09-29) — so it started failing when that proposal aged out of the window.

Fixed separately in #2102. Once that lands, this branch needs a rebase and a re-run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants