fix(ci): retry transient sandbox:none E2E shard failures once (#10566) - #10572
fix(ci): retry transient sandbox:none E2E shard failures once (#10566)#10572qwen-code-dev-bot wants to merge 3 commits into
Conversation
E2E report — issue #10566 (Main CI failed: E2E Tests on 3aa1b14)BackgroundWorkflow run 33317457036 failed on InvestigationWhich job failed. The public jobs API shows exactly one failing job: The failure is a recurring class, not this commit. The two previous main E2E runs failed the exact same leg at the exact same step on two other hosts and two other commits ( Local reproduction. A full surrogate of the failing leg at Root-cause analysis of the signature. "Exit 1 with zero vitest Change madeOne bounded retry for the Honest limits: without authenticated log access the exact internal vitest path of each death cannot be named, and the retry is a mitigation for a transient environment class, not a patch of vitest internals — the same shape of mitigation this file already carries for the image build. The final gate is the E2E lane itself turning green on main. Verification
中文说明E2E 报告 — issue #10566（Main CI failed: E2E Tests on 3aa1b14）背景Workflow 运行 33317457036 在 调查哪个任务失败。 公开的 jobs API 显示恰好一个失败任务：池运行器 失败是复发的失败类别，而非本提交引入。 此前两次 main E2E 运行在另外两台主机、另外两个提交上以完全相同的通道、完全相同的步骤失败（ 本地复现。 在 签名层面的根因分析。 "退出码 1 且零条 vitest 所做变更在 诚实的边界：在没有认证日志访问的情况下，无法点名每次死亡所处的 vitest 内部具体路径；该重试是对瞬时环境类别的缓解，而非对 vitest 内部的修补——这与该文件中镜像构建重试的缓解形态相同。最终门槛是 main 上的 E2E 通道本身变绿。 验证
🧠 Handled by Qwen Code · model/模型 |
|
Thanks for the PR! Template looks good ✓ Problem: observed, with evidence — workflow run 33317457036 died at the Direction: aligned — this is the same shape of mitigation the workflow already carries for the sandbox image build (#10355), and consistent with the just-merged #10552. CI stability is squarely in-repo remit; no product-surface concerns. Size: not applicable — no core paths touched. The whole diff is 17 production lines in the workflow file plus 32 lines of witness tests. Approach: the scope feels right. One bounded retry, scoped to the leg that has actually been failing; a deterministic failure still fails both attempts and keeps the job red; the docker leg is left alone because two ~30-minute attempts would outrun the job timeout. Witness tests pin both the retry and the docker-no-retry invariants. Risk: no elevated risk signals (no high-risk paths matched). Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到，且有证据——运行 33317457036 在 方向：对齐——与本 workflow 中沙箱镜像构建已有的重试（#10355）是同一形态的缓解，也与刚合并的 #10552 一致。CI 稳定性完全属于仓库职责范围，无产品面顾虑。 规模：不适用——未触及核心路径。整个 diff 是 workflow 文件 17 行生产改动加 32 行见证测试。 方案：范围合理。一次有界重试，且限定在实际失败的支路；确定性失败在两次尝试中都会失败、任务保持红色；docker 支路不动，因为两次约 30 分钟的尝试会超出任务超时。见证测试同时固定了"重试"与"docker 不重试"两个不变量。 风险：无升级风险信号（未命中高风险路径）。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
github-actions
Bot
added
the
review/self-reported
Code reviewReviewed the diff at the commit below. No blockers.
CI test evidenceFrom the PR's own CI on the reviewed commit (fetched via API; nothing PR-derived was executed here):
The three skips are by design, not caused by this PR: One honest limit: the retry shape is pinned by the witness tests and the exit-status semantics are statically verifiable, but the live absorption effect only manifests the next time a real transient shard death hits the post-merge lane. That is environmental by definition for any retry mitigation — same profile as the already-merged #10355 build retry — and no sandboxed lane ( Real-scenario testing: N/A — CI-harness change with nothing user-visible, and this is an unattended CI run. 中文说明代码审查在上文记录的提交上审阅了 diff，无阻塞问题。
CI 测试证据来自该提交上 PR 自身的 CI（经 API 获取；此处未执行任何 PR 派生代码）。表中三个跳过项均为设计使然、与本 PR 无关： 诚实的边界：重试形态由见证测试固定、退出状态语义可静态验证，但实际吸收效果要等下一次真实的瞬时死亡发生在合并后通道上才会显现。对任何重试缓解而言这都是环境使然——与已合并的 #10355 构建重试同一形态——且没有任何沙箱通道（ 真实场景测试：N/A——CI 基础设施变更、无用户可见行为，且本次为无人值守 CI 运行。 — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 5/5 — minimal, well-evidenced mitigation that matches the repo's existing retry precedent; the only open item is the CI run still in flight. My independent proposal for this problem — wrap the failing shard command and retry it once, warn between attempts, let the final attempt's exit status decide, pin the shape with witness tests, leave the docker leg alone — is essentially what this PR does, and I did not find a simpler path it missed. A step-level retry in the #10355 style would have duplicated the whole env/TMPDIR block for more YAML, and a marketplace retry action would trade an in-file pattern for an external dependency in privileged CI. On the bigger picture: this is the twentieth "Main CI failed" open issue of this shape in ~36 hours, three consecutive main runs died at the same leg and step on three different hosts, and the shard passes on re-run — the problem is real and bleeding maintainer attention, and a bounded retry is the same mitigation already accepted for the sandbox build in this exact file. Nothing in the diff weakens a real failure signal: two attempts both fail → the job stays red. Every line in the diff serves the stated goal; the comments carry the run IDs a future maintainer will want. Approval is deferred only because 中文说明我针对此问题的独立方案——包住失败的分片命令并重试一次、两次尝试之间告警、以最后一次尝试的退出状态为准、用见证测试固定形态、不动 docker 支路——与本 PR 基本一致，且我没有找到它遗漏的更简路径。采用 #10355 的步骤级重试会把整个 env/TMPDIR 块复制一遍、产生更多 YAML；用市场重试 action 则是以特权 CI 中的外部依赖换掉文件内模式。 全局来看：约 36 小时内已有第 20 个同形态的 "Main CI failed" 开放 issue，连续三次 main 运行在三台不同主机上死于同一支路与步骤，而该分片重跑即通过——问题真实存在且持续消耗维护者精力，有界重试正是同一文件中沙箱构建已被接受的缓解形态。diff 没有削弱任何真实失败信号：两次尝试都失败时任务依旧红色。每一行改动都服务于既定目标；注释中保留了未来维护者需要的运行 ID。 暂缓批准仅因 — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "agent 3c": executing scripts/tests/e2e-workflow.test.js — the review worktree has no node_modules (no vitest, no yaml package) and a full install exceeds the tool bu….
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未探索到全部深度（达到工具调用预算）："agent 3c"：executing scripts/tests/e2e-workflow.test.js — the review worktree has no node_modules (no vitest, no yaml package) and a full install exceeds the tool bu…。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
|
🤖 Addressed the latest review feedback (round 1/10). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/10 轮）。改动内容与我反驳保留之处如下： Review round summary — PR #10572 (issue #10566)All 6 findings from the automated review (1 Critical, 5 Suggestions) are resolved in code in commit The review body Findings and dispositionsR1-1 [Critical] — unconditional retry can outrun
|
| Probe | Mutation | Tests that went red |
|---|---|---|
| P1 | Budget gate removed (pre-round unconditional retry restored) | fails fast when the remaining job budget cannot fit a retried shard (bash witness), gates the retry on the remaining job budget |
| P2 | Record job start epoch step removed |
records the job start epoch before the expensive setup steps |
| P3 | Retry run_shard moved after the ` |
|
| P4 | || true appended to the brace group |
keeps the step red when the shard fails both attempts (bash witness), retries the sandbox:none shard exactly once |
| P5 | Shard/exclude arguments stripped from run_shard() body |
retries the full shard command, shard and excludes included |
| P6 | Docker command wrapped in a function called twice | does not retry the docker leg |
| P7 | continue-on-error: true added to Run E2E tests |
keeps the run step red when the shard stays red |
All seven mutants were caught, and the suite returned to 19/19 green after restoring the fix. P1 also doubles as proof that the Critical's witness tests fail against the pre-round code.
Failed checks noted in the feedback
Test (ubuntu-latest Node 22.x): CANCELLED— a cancellation, not a test failure; consistent with concurrency cancelling the previous push's run on this branch. Re-runs with the new push; nothing to fix in code.Post Coverage Comment: FAILURE— downstream of the cancelled Test lane; same treatment.
Verification
npm run build— passed (exit 0)npm run typecheck— passed (exit 0). A first attempt exited 2 withTS6305"output file has not been built" errors while a concurrentnpm run buildwas still writingdist/; re-run sequentially after the build completed and passed cleanly — that failure was a concurrency artifact, not a code defect.npm run lint— passed (exit 0, zero errors/warnings from both eslint invocations)- Focused Vitest (touched files):
npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/e2e-workflow.test.js scripts/tests/e2e-shard-retry.test.js— 19/19 passed (re-run green after the probes and after the stash round-trip) npm run test:scripts(full scripts suite) — 1971/1974 runnable tests pass in the final sequential run; the 3 remaining failures are all outside this PR's footprint and each is evidenced as not caused by this change:no-config-object-create-config.test.jsandqwen-autofix-workflow.test.jsfail only under full-suite load (30s timeouts on shared-host contention) and pass when run in isolation (isolated re-run: 252/253, the one failure below).verify-capture.test.js > renders 256-colour and truecolor via the default-grey fallbackfails deterministically in isolation, and reproduces identically with this change stashed (base treed8e833b1aa) — a pre-existing librsvg/rendering environment issue on this pool runner, not introduced by this commit.
- yamllint — the
yamllintbinary is not installed on this runner and installing it is out of scope for allowed commands; the YAML edits follow the file's existing quoting/indentation conventions and CI'snode scripts/lint.js --yamllintstep remains the authority. - Integration tests after
npm run bundle— not applicable: the changed behavior is CI workflow YAML and its script-test witnesses, which are not exercised through the bundled CLI or the integration harness.
中文说明
评审轮次总结 — PR #10572（issue #10566）
自动评审的全部 6 条发现（1 条 Critical、5 条 Suggestion）已在提交 4140912b34 中以代码解决。本轮未报告冲突（--conflict false），也未执行任何合并。
评审主体 [rv:5061654349] 本身不含可执行的发现——它报告的是评审者自身的工具预算耗尽（在没有 node_modules 的 worktree 中无法执行 scripts/tests/e2e-workflow.test.js），以及一项无法运行的 actionlint 内嵌 shell 检查。其可执行内容就是下面六条内联发现，均已处理。
发现与处置
R1-1 [Critical] — 无条件重试可能超出 timeout-minutes: 60 — 已解决（rc:3890261023）
该论断与 workflow 相符：timeout-minutes: 60 是整个任务共享的预算，而重试会无条件从零重跑整个分片——在三次动机运行中的两次（33293739505、33302550436）里，分片死亡时任务只剩约 10–12 分钟，不足一个分片时长（实测 17–21 分钟），重试只会在进行中被任务超时取消。
修复采用该发现建议的任务级耗时方案（步骤级方案不可行：安装耗时在约 10–29 分钟之间浮动，没有任何静态的步骤级阈值能区分这三次运行）：
- 新增首个步骤
Record job start epoch，向GITHUB_ENV导出E2E_JOB_START_EPOCH=$(date +%s)，使门控能看到整个任务的消耗，而不仅是测试步骤的消耗。 - 在
|| { … }组内，重试现在位于elapsed=$(( $(date +%s) - ${E2E_JOB_START_EPOCH:-0} ))/if (( elapsed > 2100 ))之后：任务耗时超过 2100 秒（3600 秒预算减去 25 分钟预留——实测最差分片约 21 分钟加上余量）时，步骤以::error::注解快速失败，而不是启动一次不可能完成的重试。若 epoch 缺失，算术上失败即关闭（elapsed ≈ 当前时间戳 > 2100 → 不重试）。 - 按该发现的约束，未上调
timeout-minutes: 60。
对三次动机运行逐一核对：33302550436（重试将在任务第 ~49.7 分钟开始）→ 拒绝；33293739505（~47.8）→ 拒绝；33317457036（~19.5）→ 允许，且 19.5 + 25 分钟预留 ≤ 60，放得下。
R1-2 [Suggestion] — 未固定"重试是有条件的"（末尾锚定）— 已解决（rc:3890261025）
在 retries the sandbox:none shard exactly once 中加入末尾锚定断言 expect(runStep.run).toMatch(/run_shard\s*\n\s*\}\s*\n\s*fi\s*$/)，把被重试的 run_shard 固定为组的最后一条命令、且组是脚本的最后一条语句。该模式能容忍 ${RUNNER_NAME:-this runner} 内的字面 }，因为它锚定在末尾的 run_shard 上，而非按作用域匹配。探针 P3（把重试移到组外）使其变红。
R1-3 [Suggestion] — 没有断言固定"第二次尝试失败会让步骤失败" — 已解决（rc:3890261026）
在新文件 scripts/tests/e2e-shard-retry.test.js 中加入 bash 执行见证：提取 runStep.run，把 ${{ matrix.sandbox }} 替换为 sandbox:none、${{ matrix.shard }} 替换为 1/3，用可按调用次序控制退出码的桩替换 npm，并在与运行器完全相同的 bash -e -o pipefail 下执行脚本。断言：(a) 失败一次后成功的桩以退出码 0 结束、npm 被调用两次；(b) 总是失败的桩以非零退出、npm 被调用两次。探针 P4（给大括号组追加 || true）使 (b) 变红。新文件已加入 scripts/tests/vitest.config.ts 的 win32 排除列表（与其他 bash 驱动套件并列），test_windows 与 windows-runner-smoke.yml 通道保持绿色；纯 YAML 解析的 e2e-workflow.test.js 仍继续在 Windows 上运行。
R1-4 [Suggestion] — 未固定被重试命令的参数列表 — 已解决（rc:3890261031）
新增 retries the full shard command, shard and excludes included，逐字固定完整调用，包括两个 --exclude 参数与 --shard='${{ matrix.shard }}'。探针 P5（把 run_shard() 函数体简化为裸脚本名）使其变红。
R1-5 [Suggestion] — does not retry the docker leg 固定的是计数而非结构 — 已解决（rc:3890261034）
在既有测试中补充结构断言 expect(runStep.run).not.toMatch(/[A-Za-z_]+\(\)\s*\{[^}]*QWEN_SANDBOX=docker/s)，把 docker 命令固定为不位于任何函数体内。它对当前未包裹的单条命令仍然通过；探针 P6（把 docker 命令包进函数并调用两次）使其变红。
R1-6 [Suggestion] — 没有任何断言阻止给 Run E2E tests 加 continue-on-error: true — 已解决（rc:3890261039）
新增 keeps the run step red when the shard stays red，断言 expect(runStep['continue-on-error']).toBeUndefined()，与姊妹块的固定方式一致。它只针对 run 步骤——sandbox 镜像构建步骤上有意保留的 continue-on-error: true 不受影响。探针 P7（加上该键）使其变红。
变异探针（逐一应用、跑红后还原）
| 探针 | 变异 | 变红的测试 |
|---|---|---|
| P1 | 移除预算门控（恢复轮次前的无条件重试） | fails fast when the remaining job budget cannot fit a retried shard（bash 见证）、gates the retry on the remaining job budget |
| P2 | 移除 Record job start epoch 步骤 |
records the job start epoch before the expensive setup steps |
| P3 | 把重试的 run_shard 移到 || { … } 组之后 |
retries the sandbox:none shard exactly once（末尾锚定）、gates the retry on the remaining job budget |
| P4 | 给大括号组追加 || true |
keeps the step red when the shard fails both attempts（bash 见证）、retries the sandbox:none shard exactly once |
| P5 | 从 run_shard() 函数体中移除分片/排除参数 |
retries the full shard command, shard and excludes included |
| P6 | 把 docker 命令包进函数并调用两次 | does not retry the docker leg |
| P7 | 给 Run E2E tests 加 continue-on-error: true |
keeps the run step red when the shard stays red |
七个变异体全部被捕获；还原修复后套件恢复 19/19 全绿。P1 同时证明 Critical 的见证测试在轮次前代码上会失败。
反馈中提到的失败检查
Test (ubuntu-latest Node 22.x): CANCELLED— 这是取消而非测试失败；与本分支上新一次推送触发的并发取消一致。新推送后会重新运行，代码层面无需修复。Post Coverage Comment: FAILURE— 是被取消的 Test 通道的下游结果，同样处理。
验证
npm run build— 通过（退出码 0）npm run typecheck— 通过（退出码 0）。首次尝试以退出码 2 失败，报TS6305"output file has not been built" 错误，原因是并发的npm run build当时仍在写dist/；在构建完成后按顺序重跑即干净通过——该失败是并发伪影，不是代码缺陷。npm run lint— 通过（退出码 0，两次 eslint 调用均零错误零警告）- 聚焦 Vitest（触及文件）：
npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/e2e-workflow.test.js scripts/tests/e2e-shard-retry.test.js— 19/19 通过（探针之后与 stash 往返之后均复跑为绿） npm run test:scripts（完整脚本套件）— 最终按序运行 1971/1974 个可运行测试通过；其余 3 个失败都在本 PR 足迹之外，且各有证据证明与本变更无关：no-config-object-create-config.test.js与qwen-autofix-workflow.test.js只在完整套件负载下失败（共享主机争用下的 30 秒超时），单独运行时通过（隔离复跑：252/253，唯一失败见下条）。verify-capture.test.js > renders 256-colour and truecolor via the default-grey fallback在隔离运行时确定性失败，且在 stash 掉本变更后（基线树d8e833b1aa）同样复现——是该池运行器上既有的 librsvg/渲染环境问题，非本提交引入。
- yamllint — 本运行器未安装
yamllint二进制，安装不属于允许的命令范围；YAML 改动遵循该文件既有的引号/缩进惯例，以 CI 的node scripts/lint.js --yamllint步骤为准。 npm run bundle后的集成测试 — 不适用：本变更的行为是 CI workflow YAML 及其脚本测试见证，不经过打包后的 CLI 或集成测试框架执行。
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。
🧵 Resolved all 6 selected review thread(s). · 已关闭全部选中的 6 条评审线程。
Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审；第 10 轮后本 bot 停止并将 PR 交给人工。
🧠 Handled by Qwen Code · model/模型 qwen3.8-max
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
Not explored to full depth (tool budget reached): "agent 3b": could not execute scripts/tests/e2e-shard-retry.test.js / scripts/tests/e2e-workflow.test.js — no node_modules exists in the review worktree or the main c….
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
已审查。 建议见行内评论。
未探索到全部深度（达到工具调用预算）："agent 3b"：could not execute scripts/tests/e2e-shard-retry.test.js / scripts/tests/e2e-workflow.test.js — no node_modules exists in the review worktree or the main c…。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
…10566) Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
|
🤖 Addressed the latest review feedback (round 2/10). What changed, and what I pushed back on: · 已处理最新评审反馈（第 2/10 轮）。改动内容与我反驳保留之处如下： Autofix review round — PR #10572 (issue #10566)Round outcome: all 4 inline findings addressed in code (commit Feedback points and decisions
Mutation probes (every new witness verified against its own mutant)
Each probe failed exactly one test — the intended witness — with the rest of the suite green; Failed checks triage
Verification
中文说明Autofix 审查轮次 — PR #10572（issue #10566）本轮结果：4 条行内发现全部在代码中处理完毕（提交 反馈点与处理决定
变异探测（每个新见证都针对其自身变异体验证）
每个探测恰好只有一个测试失败——正是预期的见证——套件其余部分保持绿色；每次探测后 失败检查的分诊
验证（Verification）
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧵 Resolved all 4 selected review thread(s). · 已关闭全部选中的 4 条评审线程。 Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审；第 10 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed. Suggestions are inline.
Not reviewed: reverse audit — stopped before round 1 by the review time budget.
Deferred under the convergence posture (round 3, not a blocker) — recorded, not requested in this round:
scripts/tests/e2e-shard-retry.test.js:94 — [probe] No execution witness for the green first-attempt path — an unconditional pre-gate side effect (e.g. a spurious ::warning:: before run_shard || { ) ships with every witness greenscripts/tests/e2e-workflow.test.js:180 — [probe] Docker-leg anti-retry pin [^}]* cannot see through an inner } — a function-wrapped docker double invocation (folding the e2e.yml:85 preflight into it) ships with every witness green
Convergence: round 3 posted 1 inline comment(s), 1 of them reported for the first time; the previous round posted 4 (4 new). Findings keep coming back to the same files: scripts/tests/e2e-shard-retry.test.js (findings in round 1; 1 more now). A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. (Observation only — nothing was withheld from this review because of this observation.)
中文说明
仅完成部分审查，审查缺口已披露。 建议见行内评论。
未审查：反向审计——评审时间预算不足，未能开始第 1 轮。
收敛姿态下延后（第 3 轮，非阻断）——已记录，本轮不要求修改：共 2 条（原文未翻译，列表见上方英文部分）。
收敛情况：第 3 轮发布了 1 条行内评论，其中 1 条是首次提出；上一轮发布了 4 条（其中 4 条首次提出）。发现反复回到同一批文件：scripts/tests/e2e-shard-retry.test.js（第 1 轮已出过发现，本轮又有 1 条）。一个不断再生兄弟发现的簇，通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因，或把独立的簇拆成单独的 PR，通常比逐条修复更快结束循环。（仅为观察——本轮评审未因此扣留任何内容。）
— qwen3.8-max via Qwen Code /review (v0.22.3)
| import { parse } from 'yaml'; | ||
|
|
||
| // Executes the e2e workflow's 'Run E2E tests' script under GitHub Actions' | ||
| // default Linux step shell — `bash -e {0}`: the step has no `shell:` override |
There was a problem hiding this comment.
[Suggestion] R1-3: (fix-induced) The round-2 fix for R1-3 moved this harness to bash -e and documented the shell-fidelity premise in the header comment — the step has no shell: override and e2e.yml no defaults: block — but that premise is load-bearing and pinned by no assertion, while the sibling shape suite pins strictly weaker premises (the epoch step's if:, continue-on-error at both levels, timeout-minutes: 60). If a later change adds defaults: run: shell: bash — explicit bash expands to bash --noprofile --norc -e -o pipefail {0}; .github/workflows/qwen-autofix.yml:71-73 does exactly this — or a step-level shell: override to e2e.yml, the production 'Run E2E tests' block switches shell semantics while this harness keeps executing bare bash -e, so every execution witness stays green while silently blessing a shell contract the lane no longer runs. Since e2e.yml is post-merge-only, this unit suite is the only pre-merge signal for the step. The divergence is latent today — the step script contains no pipelines, so both shell modes agree — but once a pipeline exists, or the shell becomes sh (where [[ ]] is a syntax error), the lane changes semantics while the suite reports green.
Witness:
MUTATED (shell: bash added to the step), no pin: Tests 22 passed (22)
MUTATED + pin: × keeps the default step shell the execution harness assumes → expected 'bash' to be undefined
CLEAN + pin: Tests 23 passed (23)
Fix — pin the premise in the sandbox:none shard retry describe of scripts/tests/e2e-workflow.test.js:
expect(runStep.shell).toBeUndefined();
expect(yml.defaults).toBeUndefined();The pin must assert absence only and never sweep jobs that deliberately set a shell — shell: bash expands to bash --noprofile --norc -eo pipefail, as pinned at scripts/tests/qwen-autofix-workflow.test.js:5139-5143 and exercised by .github/workflows/qwen-autofix.yml:71-73. Please also add the mutation proof: introducing shell: bash or defaults: run: shell: bash into e2e.yml must turn the new assertion red while the rest of the suite stays green.
中文说明
[Suggestion] R1-3：（fix-induced，由上一轮 R1-3 的修复引入）上一轮对 R1-3 的修复把本 harness 改为以 bash -e 执行，并在文件头注释中记录了 shell 保真前提——该步骤没有 shell: 覆盖、e2e.yml 也没有 defaults: 块——但这个前提是承重的，却没有任何断言固定它；而姊妹形态测试套件连更弱的前提都固定了（epoch 步骤的 if:、步骤级与任务级的 continue-on-error、timeout-minutes: 60）。若未来有人给 e2e.yml 加上 defaults: run: shell: bash（显式 bash 会展开为 bash --noprofile --norc -e -o pipefail {0}；.github/workflows/qwen-autofix.yml:71-73 正是这样做的）或步骤级 shell: 覆盖，生产环境的 'Run E2E tests' 脚本就切换了 shell 语义，而本 harness 仍以裸 bash -e 执行——所有执行见证依旧全绿，却悄悄为一个通道已不再运行的 shell 契约作证。由于 e2e.yml 只在合并后运行，这套单元测试是该步骤唯一的合并前信号。目前该分歧是潜在的——步骤脚本里没有任何管道，两种 shell 模式行为一致——但一旦出现管道，或 shell 变成 sh（[[ ]] 在 sh 下是语法错误），通道语义已变而套件依旧报绿。
见证：变异（给步骤加 shell: bash）且无固定时 22/22 全绿；加上固定后该断言变红（expected 'bash' to be undefined）；干净代码加固定 23/23 全绿。
修复——在 scripts/tests/e2e-workflow.test.js 的 sandbox:none shard retry 块中加入 expect(runStep.shell).toBeUndefined(); expect(yml.defaults).toBeUndefined();。该固定只应断言缺席，绝不能横扫有意设置 shell 的任务（见 scripts/tests/qwen-autofix-workflow.test.js:5139-5143 与 .github/workflows/qwen-autofix.yml:71-73）。请同时补上变异证明：给 e2e.yml 引入 shell: bash 或 defaults: run: shell: bash 时新断言必须变红，其余套件保持绿色。
— qwen3.8-max via Qwen Code /review (v0.22.3)
2 participants
What this PR does
Gives the
sandbox:noneleg of the post-merge Linux E2E lane one bounded retry at its test step. The shard command is wrapped in a small shell function and, if the first attempt exits non-zero, the step emits a::warning::naming the runner and re-runs the same shard once; the second attempt's exit status is the step's, so a deterministic test failure fails both attempts and keeps the job red. The retry is deliberately scoped to thesandbox:noneleg — two ~30-minutesandbox:dockerattempts would outrun the job'stimeout-minutes— and is pinned by new witness tests next to the existing sandbox-image-build-retry block, including one asserting the docker leg is not retried.Why it's needed
Run 33317457036 failed
E2E Test (Linux) - sandbox:none - shard 2/3at theRun E2E testsstep with exit code 1 and zero vitestFAILlines in the log, which is the class this repo documents in the integration test config: under shared self-hosted pool pressure, vitest's fixed 60-second worker→main RPC budget turns a stalled-but-otherwise-green run red via unhandled-error paths, and the deaths that survive the lane's existing self-hosted exemption sit inside vitest itself, unreachable from repository code. The two previous main E2E runs failed the identical leg and step on two different hosts and commits, while the run between them was fully green and a local reproduction of the exact shard passes — the same transient, pressure-correlated shape the sandbox-image build retry in this same workflow was added for (#10355). Without a retry, one host under pressure exits a whole shard red and files a per-commit "Main CI failed" issue (this one is the twentieth such open issue in ~36 hours); with it, a transient death costs one extra shard attempt and a deterministic failure still reports red.Reviewer Test Plan
How to verify
This is CI-harness behavior plus its witness tests; the retry logic is fully inspectable from the workflow file and was verified three ways. First, the witness block in the scripts suite:
npx vitest run --config ./scripts/tests/vitest.config.ts e2e-workflowpasses 12/12, and removing the retry block from the workflow (or duplicating the docker leg's command) makes the corresponding witness test fail. Second, the bash semantics were simulated underbash -eo pipefail(the Actions default shell) with a stubbed shard command: one failure retries and exits 0, two failures exit 1, and a passing first attempt never retries. Third, the real gate is the lane itself: the next main push should either pass both attempts or, if a genuine test failure exists, report itsFAILlines on the second attempt instead of dying silently.Evidence (Before & After)
N/A — CI-harness change, not user-visible. Before: run 33317457036 (and runs 33302550436, 33293739505) red at
Run E2E testswith zeroFAILlines. After: one bounded retry absorbs a single transient death; deterministic failures stay red on both attempts.Tested on
Environment (optional)
Self-hosted pool machine (the same pool the failing lane runs on): witness suite with mutation probes, functional retry simulation under
bash -eo pipefail,bash -non the generated step script for both matrix legs,npm run build,npm run typecheck,npm run lint,npm run test:scripts, and a full surrogate of the failing shard (test:integration:sandbox:none, shard 2/3, real model credentials) which passes at the failing commit.Risk & Scope
sandbox:noneshard now takes up to one extra shard attempt (~15–20 minutes, still inside the job's 60-minute timeout) before reporting red; transient pressure deaths report green with a::warning::in the log instead of filing another per-commit issue. No test assertion, timeout, or exclusion is weakened anywhere.sandbox:dockerleg keeps no retry because two attempts would not fit the job timeout; the poisoned-host and disk-pressure classes tracked by other issues/PRs (ci: gate heavy jobs on a disk floor and persist pressure samples #10394's disk floor, fix(cli): don't crash startup when the output-language file is unwritable (#10453) #10455's unwritable-home crash sites) are separate and untouched.Linked Issues
Fixes #10566
中文说明
本 PR 做了什么
为合并后 Linux E2E 通道的
sandbox:none支路在其测试步骤上增加一次有界重试。分片命令被包进一个小的 shell 函数，首次尝试非零退出时，步骤输出一条点名运行器的::warning::并将同一分片重跑一次；第二次尝试的退出状态即步骤的状态，因此确定性测试失败会在两次尝试中都失败、保持任务红色。重试刻意限定在sandbox:none支路——两次约 30 分钟的sandbox:docker尝试会超出任务的timeout-minutes——并由紧邻既有"沙箱镜像构建重试"块的新见证测试固定，其中包括一条断言 docker 支路不被重试。为什么需要
运行 33317457036 在
Run E2E tests步骤以退出码 1 失败，且日志中没有任何一条 vitestFAIL行，这正是本仓库在集成测试配置中记录过的类别：在共享自托管池压力下，vitest 固定的 60 秒 worker→main RPC 预算会把一个停滞但本来全绿的运行经由未处理错误路径变红；而在该通道已有的自托管豁免之后仍会发生的那些死亡，位于 vitest 内部，仓库代码无法触及。此前两次 main E2E 运行在不同的主机、不同的提交上以完全相同的支路与步骤失败，夹在中间的一次运行完全绿色，且对同一分片的本地复现通过——与同一 workflow 中沙箱镜像构建重试（#10355）所针对的瞬时、压力相关形态完全一致。没有重试时，一台承压主机就会让整个分片变红并归档一个按提交的 "Main CI failed" issue（本 issue 是约 36 小时内的第 20 个同类开放 issue）；有了重试，一次瞬时死亡只多花一次分片尝试，而确定性失败依旧报红。审阅者测试计划
如何验证
这是 CI 基础设施行为及其见证测试；重试逻辑可以直接从 workflow 文件中审阅，并已用三种方式验证。第一，scripts 套件中的见证块：
npx vitest run --config ./scripts/tests/vitest.config.ts e2e-workflow12/12 通过；从 workflow 中移除重试块（或复制 docker 支路的命令）会使对应的见证测试失败。第二，在bash -eo pipefail（Actions 默认 shell）下用桩分片命令模拟了 bash 语义：失败一次会重试并以 0 退出，失败两次以 1 退出，首次通过则绝不重试。第三，真正的门槛是通道本身：下一次 main 推送要么两次尝试都通过，要么在存在真实测试失败时于第二次尝试上报出FAIL行，而不是无声死亡。证据（前后对比）
N/A — CI 基础设施变更，无用户可见行为。之前：运行 33317457036（以及 33302550436、33293739505）在
Run E2E tests变红且零条FAIL行。之后：一次有界重试吸收单次瞬时死亡；确定性失败在两次尝试中保持红色。测试环境
环境（可选）
自托管池机器（与失败通道相同的池）：带变异探测的见证套件、
bash -eo pipefail下的重试功能模拟、对两个矩阵支路生成步骤脚本的bash -n、npm run build、npm run typecheck、npm run lint、npm run test:scripts，以及对失败分片的完整替代复现（test:integration:sandbox:none，shard 2/3，真实模型凭证），在失败提交上通过。风险与范围
sandbox:none分片现在最多多花一次分片尝试（约 15–20 分钟，仍在任务 60 分钟超时之内）才报红；瞬时压力死亡会在日志中留下一条::warning::并报绿，而不是再归档一个按提交的 issue。任何测试断言、超时或排除都未被削弱。sandbox:docker支路不加重试，因为两次尝试放不下任务超时；由其他 issue/PR 跟踪的宿主机污染与磁盘压力类别（ci: gate heavy jobs on a disk floor and persist pressure samples #10394 的磁盘下限、fix(cli): don't crash startup when the output-language file is unwritable (#10453) #10455 的不可写主目录崩溃点）是独立问题，未被触及。关联 Issue
Fixes #10566