fix(langmgr,provenance): tolerate broken upstream go.mod with 'go mod tidy -e'

[omercnet]

Fixes #1601.

Prior art: refines #1089, which introduced the post-go get go mod tidy cleanup step.

Problem

Copa's app-level Go patcher runs go get <patched-versions> followed by go mod tidy to clean up the module graph before rebuilding. Strict go mod tidy is too aggressive a cleanup gate — for real-world upstream Go projects whose go.mod is brittle under a modern Go toolchain, a fresh tidy discovers transitive imports that no longer resolve and exits non-zero, killing the entire patch even though the CVE-relevant go get calls succeeded.

Concrete failure

Patching quay.io/prometheus-operator/prometheus-config-reloader:v0.81.0:

#14 4.934 go: github.com/prometheus-operator/prometheus-operator/pkg/namespacelabeler imports
#14 4.934   github.com/prometheus-community/prom-label-proxy/injectproxy imports
#14 4.934   github.com/go-openapi/runtime/client imports
#14 4.934   github.com/go-openapi/runtime/yamlpc tested by
#14 4.934   github.com/go-openapi/runtime/yamlpc.test imports
#14 4.934   github.com/go-openapi/testify/enable/yaml/v2 imports
#14 4.934   github.com/go-openapi/testify/v2/assert/yaml: module
            github.com/go-openapi/testify/v2@latest found (v2.5.0), but
            does not contain package github.com/go-openapi/testify/v2/assert/yaml
#14 ERROR: process "/usr/local/go/bin/go mod tidy" did not complete successfully: exit code: 1

Error: Patch Failed
  failed to solve: process "/usr/local/go/bin/go mod tidy" did not complete
  successfully: exit code: 1

The go.mod problem is upstream-project hygiene (github.com/go-openapi/testify/v2/assert/yaml does not exist in the resolved version of testify/v2); it has nothing to do with the CVE being patched.

Affected images

The same failure mode has been observed in production patching workflows against:

• prometheus-operator/prometheus-config-reloader
• grafana/promtail
• cilium/cilium-envoy (partially Go-related)
• kyverno/background-controller, kyverno/cleanup-controller, kyverno/kyvernopre, kyverno/reports-controller, kyverno/readiness-checker
• rabbitmqoperator/messaging-topology-operator

Fix

Switch all three go mod tidy invocations to go mod tidy -e:

1. pkg/langmgr/golang.go — primary in-image path (factored into a small buildGoUpdateCmd helper for testability)
2. pkg/langmgr/golang.go — tooling-container fallback path (uses the same helper)
3. pkg/provenance/rebuilder.go — binary-rebuild path. This is the call site that produces the literal error in the log above — go version -m runs the rebuilder first and its bare go mod tidy is what exits 1.

From go help mod tidy:

The -e flag causes tidy to attempt to proceed despite errors encountered while loading packages.

tidy -e will still:

• Update go.mod/go.sum based on what it can resolve
• Honor the go get version pins applied just before tidy (the actual CVE fix)
• Produce a working module graph for the subsequent go build

It will not:

• Mask errors in our patched package versions (go get runs before tidy and still fails loudly)
• Hide CVE regressions (the post-patch Trivy rescan is a second backstop)
• Change behavior for projects whose go.mod was already healthy (tidy -e is a strict superset of tidy's success path)

Smoke test

Built copa from this branch and ran end-to-end against the canonical reproducer.

COPA_EXPERIMENTAL=1 ./dist/linux_amd64/release/copa patch \
  -i quay.io/prometheus-operator/prometheus-config-reloader:v0.81.0 \
  -r trivy-before.json \
  -t v0.81.0-patched \
  --pkg-types library --toolchain-patch-level minor \
  --debug --timeout 30m

Result:

level=debug msg="Running go mod tidy -e..."
#10 /usr/local/go/bin/go mod tidy -e
…
Patch Summary: 47 total, 39 patched, 8 skipped
Patched image (linux/amd64): quay.io/prometheus-operator/prometheus-config-reloader:v0.81.0-patched

Patched binary --version

$ docker run --rm --entrypoint /bin/prometheus-config-reloader \
    quay.io/prometheus-operator/prometheus-config-reloader:v0.81.0-patched --version
prometheus-config-reloader, version 0.81.0 (branch: , revision: 240b303)
  go version:       go1.26.3
  platform:         linux/amd64

(Original image was on go1.23.7; the rebuilt binary now ships on go1.26.3 — the actual fix carrier for the stdlib CVEs.)

Trivy before/after delta

	Total	CRITICAL	HIGH	MEDIUM	LOW	UNKNOWN
Before	47	1	12	25	1	8
After	8	0	3	5	0	0
Fixed	39	1	9	20	1	8
New regressions	0	0	0	0	0	0

All 19 stdlib CVEs (1 CRITICAL — CVE-2025-68121; 9 HIGH including CVE-2025-47907, CVE-2025-58183, CVE-2025-61726…) resolved by the toolchain upgrade. The 8 remaining are minor-version bumps of prometheus/prometheus, golang.org/x/crypto, golang.org/x/net, golang.org/x/oauth2 filtered out by the default --library-patch-level patch policy — unrelated to this PR.

What this PR does

• Replaces the three go mod tidy invocations with go mod tidy -e (pkg/langmgr/golang.go ×2, pkg/provenance/rebuilder.go ×1).
• Adds TestBuildGoUpdateCmd (pkg/langmgr/golang_test.go) — table-driven, asserts both langmgr call sites.
• Adds TestRebuilderUsesGoModTidyDashE (pkg/provenance/rebuilder_test.go) — regression guard for the rebuilder path.
• Updates website/docs/app-level-patching.md with a "Tolerance for broken upstream go.mod" subsection under the Go limitations.
• make test green locally; golangci-lint run ./pkg/langmgr/... ./pkg/provenance/... clean for changed code.
• Smoke-tested end-to-end against the reproducer image (see above).
• DCO sign-off present on the commit.

Out of scope (intentionally)

• No replace directive injection (per-image overrides are a separate, larger proposal)
• No fallback rebuild architecture (e.g. Wolfi base swap)
• No changes to Python, Node, Java, or .NET patchers
• No behavior change for healthy upstream go.mod files

[Copilot]

Pull request overview

This PR makes Go app-level patching more resilient by switching the post-go get cleanup step from go mod tidy to go mod tidy -e, so upstream module-graph hygiene issues don’t block otherwise-successful CVE patching flows.

Changes:

• Replace strict go mod tidy with tolerant go mod tidy -e in both Go lang manager update paths and the provenance rebuilder binary rebuild path.
• Factor the Go lang manager update command into a small helper (buildGoUpdateCmd) and add regression tests to enforce -e usage.
• Document the new behavior in the app-level patching docs under Go limitations.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
website/docs/app-level-patching.md	Documents that Go binary patching uses go mod tidy -e to tolerate broken upstream module graphs.
pkg/provenance/rebuilder.go	Updates the rebuild path to run go mod tidy -e and logs the new command.
pkg/provenance/rebuilder_test.go	Adds a regression test guarding against reintroducing bare go mod tidy.
pkg/langmgr/golang.go	Uses a shared helper to build the go get … && go mod tidy -e update command for both update sites.
pkg/langmgr/golang_test.go	Adds a table-driven test validating the helper emits go mod tidy -e.

[robert-cronin]

lgtm