fix(ci): stop shared-pool hosts exiting all-green E2E runs red (#10325) - #10329
Merged
Conversation
The post-merge E2E run at 1637aa9 failed three Linux pool jobs at the test step with the whole suite passing and no vitest FAIL line in any failed job log — an exit-red-with-no-failing-test signature that drains the lane's signal (the same three jobs failed the same way in the dispatch run an hour earlier). Two demonstrated mechanisms produce it, and this closes both: - The harness saves the host's global QWEN.md before the run and writes it back in teardown. On a pool host a privileged job can leave that file readable-but-unwritable (root-owned 0644; the ownership heal step only covers the workspace, not $HOME), and the restore threw EACCES after every test passed — exit 1, 'Startup Error', zero FAIL lines. Reproduced end-to-end with a chmod-444 QWEN.md under a scratch QWEN_HOME: 6/6 tests green, run red. The restore is now best-effort and logs a warning instead, keeping the poisoned host diagnosable. - Vitest's worker->main onTaskUpdate RPC stalls past its fixed 60s budget under runner resource pressure and exits an all-green run red via an unhandled error. That class was exempted off-Linux in #10097, but #10085 moved the Linux shards from ubuntu-hosted VMs onto the shared self-hosted pool, where the same pressure lives. Extend the exemption to self-hosted runners, mapping RUNNER_ENVIRONMENT explicitly in the test step; github-hosted Linux (the nightly isolated legs) and local Linux runs keep the signal. Real test failures still fail every run on every runner. Both guards are pinned by witness tests (new globalSetup.test.ts; extended integration-vitest-config.test.ts) and each was mutation-probed.
Collaborator
Author
E2E Report — Issue #10325 (Main CI failed: E2E Tests on 1637aa9)DiagnosisRun 33092671484 (push, commit
The sibling dispatch run on the same tree minus the goal fix's PR branch (33091085019) failed the exact same three jobs. Failed-job durations match green siblings (205s vs 206s for none shard 2/3), and every job annotation is the generic Mechanism 1 (reproduced end-to-end): poisoned-host memory restore turns teardown red
This also explains the observed determinism (the same three jobs failing in two runs an hour apart = the jobs scheduled onto the poisoned hosts), macOS staying green (ephemeral VMs), and the failure class appearing only after the pool move. Mechanism 2 (house-precedent class): pressure-flake unhandled errorsVitest's worker→main Not reproducible locally here (single shard and three-shard-contended runs both stayed green on this machine), but the lane-wide signature matches and the mitigation is the established house one. Fix
Both behaviors are pinned by witness tests, and each guard was mutation-probed (remove guard → witness fails → restore → green):
Verification
中文说明E2E 报告 — Issue #10325（主分支 CI 失败：E2E Tests @ 1637aa9）诊断运行 33092671484（push，提交
在同一棵代码树（未含 goal 修复的 PR 分支）上的手动 dispatch 运行 33091085019 失败的恰好也是同样三个作业。失败作业的耗时与绿色兄弟作业相当（none 分片 2/3 为 205 秒对 206 秒），且每个作业的注解都是通用的 机制 1（已端到端复现）：被污染的宿主机内存文件还原使 teardown 变红
这也解释了观察到的确定性（相隔一小时的两次运行失败的是同样三个作业 = 被调度到中毒宿主机的作业）、macOS 保持绿色（一次性虚机），以及该失败类别只在迁移到池之后才出现。 机制 2（既有先例类别）：资源压力导致的未处理错误vitest 的 worker→main 在本地无法复现（单分片与三分片并发运行在本机均保持绿色），但整个通道的签名吻合，且所用缓解措施正是既有的仓库惯例。 修复
两个行为都由见证测试钉住，且每个守卫都做了变异探测（移除守卫 → 见证失败 → 恢复 → 变绿）：
验证
🧠 Handled by Qwen Code · model/模型 |
github-actions
Bot
added
the
review/self-reported
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed, with run-level evidence. I pulled run 33092671484 from #10325: exactly the three Linux jobs the PR names ( Direction: aligned. A red E2E lane with no failing test to bisect is worse than no lane — it hides real regressions and trains people to ignore the signal. This keeps every real test-failure signal and only demotes the two infra failure classes. The pressure-flake exemption extends the exact mechanism #10112 added for the off-Linux lanes, whose scoping assumption (Linux shards on ubuntu-hosted VMs) the pool move invalidated — consistent lineage, not a new invention. Size: not applicable — no core module paths touched. ~36 production lines (harness + workflow env mapping) plus ~138 test lines. Approach: the scope feels right. The two mechanisms are independent and each maps to one way the lane dies; I checked for a single simpler fix and there isn't one that preserves the signal. Alternatives like exempting unhandled errors everywhere would also silence the nightly github-hosted isolated legs and local Linux runs, which this deliberately keeps. Both fixes carry witness tests. One scoping note, non-blocking: the pool lane in Risk: no elevated risk signals — none of the changed files match the revert-correlated paths. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到，且有运行级证据。我从 #10325 调出了运行 33092671484：恰好是本 PR 点名的三个 Linux 作业（ 方向：对齐。一条没有可定位失败测试的红色 E2E 通道比没有通道更糟——既掩盖真实回归，又让人习惯性忽略信号。本 PR 保留了全部真实测试失败信号，只降级两类脚手架失败。压力抖动豁免沿用的正是 #10112 为非 Linux 通道加的机制，其适用范围假设（Linux 分片跑在 ubuntu 托管虚机上）已被迁池推翻——谱系一致，不是新发明。 规模：不适用——未触及核心模块路径。约 36 行生产改动（脚手架 + 工作流环境变量映射），约 138 行测试。 方案：范围合理。两个机制相互独立，各自对应通道失败的一种方式；我找过是否存在单一更简方案，结论是没有——既要修复又要保留信号就绕不开这两处。像"处处豁免未处理错误"这类替代方案会同时让夜间 github 托管隔离腿与本地 Linux 运行失去信号，而本 PR 刻意保留了它们。两个修复都带了见证测试。一条非阻塞的范围提示： 风险：无升级风险信号——改动文件均未命中与 revert 相关的高风险路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
Code reviewApproach matches what I'd have proposed: the restore becomes best-effort with a visible warning, and the existing unhandled-error exemption gains a What I verified by reading the code:
Testing evidence carried: CI check results fetched via the GitHub API for the reviewed commit (unattended run — PR code is never built or executed here). The witness for the teardown half is not exercised by this PR's pre-merge CI: the full integration suite only runs in the post-merge E2E workflow, and the "Integration Tests (CLI, No Sandbox)" check is skipped on this PR's CI profile — so below the line names a lane that would settle it before merge. Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 No failures so far; the ubuntu test job (which runs the config witness via the scripts suite) is still in flight, and the table above updates when CI settles. The E2E lane itself doesn't run pre-merge by design, so the final proof — pool runs actually going green — is the next post-merge push run, as the PR's own plan says. Sandboxed verification would settle the remaining pre-merge gap: 中文说明代码审查方案与我独立想到的做法一致：还原改为尽力而为并保留可见警告；既有的未处理错误豁免新增 读代码核实过的内容：
测试证据：通过 GitHub API 获取的该提交 CI 检查结果（无人值守运行——此处从不构建或执行 PR 代码）。teardown 一侧的见证不在本 PR 合并前 CI 中运行：完整集成套件只在合并后的 E2E 工作流中跑，而本 PR 的 CI 画像跳过了 "Integration Tests (CLI, No Sandbox)"——因此下面一行点名了可在合并前补足验证的通道。 （CI 表格见上，合并后由 finalize 工作流更新。） 目前无失败；ubuntu 测试作业（经由 scripts 套件运行配置见证）仍在进行中。E2E 通道按设计不在合并前运行，因此最终证明——池上的运行真正变绿——要看合并后的下一次 push 运行，与本 PR 自述计划一致。 沙箱验证可以补上合并前的最后一个缺口： — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 4/5 — clean, minimal fix for a verified lane failure; the two nits below are non-blocking. Stepping back: this one holds up. I verified the problem independently rather than taking the PR's word for it — run 33092671484 really does show exactly the three Linux jobs dying at the test step with their siblings green, so despite the Non-blocking nits:
Verdict: approve. CI is still running, so approval is deferred until CI lands green on 中文说明置信度：4/5 —— 针对已核实的通道故障的干净、最小修复；以下两点均非阻塞。 退一步看：这个 PR 站得住。我没有只听 PR 自述，而是独立核实了问题——运行 33092671484 确实呈现所述签名：恰好三个 Linux 作业在测试步骤死掉、同组其余作业全绿。所以尽管带着 非阻塞的小点：
结论：批准。CI 仍在运行，因此批准推迟到 CI 在 — Qwen Code · qwen3.8-max Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
approved these changes
Aug 27, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Partially reviewed — gaps disclosed. Suggestions are inline.
Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。 建议见行内评论。
未审查：build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.2)
Comment on lines
+81
to
+83
| await mkdir(join(qwenHome, 'QWEN.md')); | ||
|
|
||
| await expect(teardown()).resolves.toBeUndefined(); |
Collaborator
There was a problem hiding this comment.
[Suggestion] The failure-path witness test pins only that teardown() resolves — it never asserts the warning this diff's own comment deliberately keeps ("Keep the warning visible so the poisoned host is still diagnosable"). I mutation-tested this at the reviewed commit: deleting the console.error line in globalSetup.ts leaves Tests 2 passed (2) — the mutation survives. If a later cleanup removes or debug-levels that warning, every test stays green and a poisoned pool host becomes undiagnosable: the next #10325-flavoured incident has no warning in the log and a green suite, reproducing the exact "red run with no signal" diagnosis problem this PR set out to keep visible. Spy on the warning and pin it in the failure-path case:
Suggested change
| await mkdir(join(qwenHome, 'QWEN.md')); | |
| await expect(teardown()).resolves.toBeUndefined(); | |
| await mkdir(join(qwenHome, 'QWEN.md')); | |
| const errSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); | |
| await expect(teardown()).resolves.toBeUndefined(); | |
| expect(errSpy).toHaveBeenCalledWith( | |
| expect.stringContaining('Warning: could not restore'), | |
| expect.anything(), | |
| ); | |
| errSpy.mockRestore(); |
Fix witness: with this assertion in place, deleting the console.error line from integration-tests/globalSetup.ts turns the failure-path case red — verified at the reviewed commit: mutant + assertion → 1 failed (AssertionError: expected "error" to be called with arguments [ StringContaining, Anything ]); un-mutated code + assertion → Tests 2 passed (2).
中文说明
失败路径的见证测试只钉住了 teardown() 正常结束，却没有断言本 diff 注释刻意保留的警告（"Keep the warning visible so the poisoned host is still diagnosable"）。已在评审提交上做了变异测试：删除 globalSetup.ts 中的 console.error 行后仍是 Tests 2 passed (2) —— 变异存活。若后续清理移除或降级该警告，所有测试依旧绿色，中毒的池宿主机将变得无法诊断：下一次 #10325 式事故在日志中没有任何警告且套件全绿，重新造成本 PR 着力保留可见性的"红色运行却无信号"诊断难题。建议在失败路径用例中监听并钉住该警告（见上方 suggestion 代码块）。
修复见证：加上该断言后，删除 integration-tests/globalSetup.ts 中的 console.error 行会使失败路径用例变红 —— 已在评审提交上验证：变异体 + 断言 → 1 failed（AssertionError: expected "error" to be called with arguments [ StringContaining, Anything ]）；未变异代码 + 断言 → Tests 2 passed (2)。
— qwen3.8-max via Qwen Code /review (v0.22.2)
Comment on lines
+122
to
+125
| try { | ||
| await mkdir(dirname(memoryFilePath), { recursive: true }); | ||
| await writeFile(memoryFilePath, originalMemoryContent, 'utf-8'); | ||
| } catch (e) { |
Collaborator
There was a problem hiding this comment.
[Suggestion] This best-effort restore guard and its only witness test (integration-tests/globalSetup.test.ts) never execute before merge: both sit outside every npm workspace (this review's scoped build/test collected zero workspaces and ran nothing), PR CI only typechecks them, and e2e.yml — the only workflow whose vitest run --root ./integration-tests collects the test — has no pull_request trigger (it runs post-merge on main, nightly, and on workflow_dispatch), while ci.yml's integration legs filter to cli / an explicit no-AK file list that does not collect the root-level test. The PR can therefore merge with every pre-merge gate green; if the guard is wrong — or a later change mis-scopes the try/catch — the first execution is a post-merge E2E run on main on a pool host carrying a poisoned QWEN.md, where a broken teardown turns every all-green E2E run red with no failing test — the exact #10325 failure class this PR fixes — forcing a revert on main instead of a pre-merge fix. Give the new test one pre-merge run, e.g. in PR CI after build + bundle:
npx vitest run --root ./integration-tests globalSetup.test.tsor have a maintainer dispatch e2e.yml on this branch before merging (it supports workflow_dispatch).
Fix witness: integration-tests/globalSetup.test.ts — "does not exit an all-green run red when the restore cannot write" goes red if this try/catch is removed.
中文说明
这个尽力而为的还原守卫及其唯一见证测试（integration-tests/globalSetup.test.ts）在合并前从不执行：两者都在所有 npm 工作区之外（本次评审按范围划定的构建/测试收集到零个工作区、没有运行任何套件），PR CI 只对其做类型检查；而 e2e.yml —— 唯一会以 vitest run --root ./integration-tests 收集该测试的工作流 —— 没有 pull_request 触发器（只在合并后的 main、夜间定时与 workflow_dispatch 时运行），ci.yml 的集成腿则用 cli / 显式 no-AK 文件列表过滤，不会收集根目录下的该测试。因此本 PR 合并时所有合并前门槛都是绿色的；若守卫有错 —— 或后续改动错误地限制了 try/catch 的范围 —— 首次执行将是合并后 main 上的 E2E 运行，且池宿主机上带着被投毒的 QWEN.md：损坏的 teardown 会让每一次全绿 E2E 运行变红且没有可定位的失败测试 —— 恰是本 PR 要修复的 #10325 失败类别 —— 只能被迫在 main 上回滚而不是在合并前修复。请给新测试一次合并前的执行，例如在 PR CI 中于 build + bundle 之后运行上方命令，或请维护者在合并前对本分支手动触发 e2e.yml（支持 workflow_dispatch）。
修复见证：integration-tests/globalSetup.test.ts —— 若移除该 try/catch，"does not exit an all-green run red when the restore cannot write" 会变红。
— qwen3.8-max via Qwen Code /review (v0.22.2)
wenshao
approved these changes
Aug 27, 2026
Collaborator
|
Released in v0.22.3. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Post-merge E2E runs on the shared self-hosted pool have been exiting red with every test green and no failing test to point at — run 33092671484 failed three Linux jobs at the test step, each with the full suite passing and zero
FAILlines in the log. This PR closes the two mechanisms that produce that signature. First, the integration test harness saves the host's globalQWEN.mdbefore the run and writes it back afterwards; on a pool host where a privileged job left that file readable-but-unwritable, the write-back threw after every test had passed and turned the whole run red. The restore is now best-effort: it logs a warning (so the poisoned host stays diagnosable) instead of failing the run. Second, the existing exemption that keeps runner-pressure unhandled errors from failing an all-green run is extended to self-hosted runners: the Linux shards moved from ubuntu-hosted VMs onto the shared pool, where the same resource pressure lives, while github-hosted Linux (the nightly isolated legs) and local Linux runs keep the unhandled-error signal. Real test failures still fail every run on every runner.Why it's needed
Since the Linux E2E shards moved to the persistent pool, every push E2E run has been red while reporting no failing test, which drains the suite's signal: a red lane nobody can bisect hides real regressions and trains people to ignore the lane. The failure was reproduced end-to-end for the restore path — a read-only
QWEN.mdunder a scratchQWEN_HOMEmakes a run with 6/6 tests passing exit 1 withStartup Error: EACCES— and the pressure-flake path is the same class already exempted off-Linux in #10097/#10112, whose scoping assumption (Linux shards on ubuntu-hosted VMs) the pool move invalidated. Both fixes preserve every real test-failure signal.Reviewer Test Plan
How to verify
mkdir -p /tmp/poison && echo x > /tmp/poison/QWEN.md && chmod 444 /tmp/poison/QWEN.md, then runQWEN_HOME=/tmp/poison QWEN_SANDBOX=false npx vitest run --root ./integration-tests test-helper.test.ts. Expected: exit 0, all tests pass, oneWarning: could not restore /tmp/poison/QWEN.mdline in the output. Without this PR the same command exits 1 with aStartup Errorsection despite all tests passing.npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/integration-vitest-config.test.tsandQWEN_SANDBOX=false npx vitest run --root ./integration-tests globalSetup.test.ts— both green.self-hostedclause inintegration-tests/vitest.config.tsand the config witness fails; drop the teardown try/catch inintegration-tests/globalSetup.tsand the globalSetup witness fails.sandbox:noneshard 1/3,sandbox:dockershards 1/3 and 3/3 in run 33092671484) should either go green or, if a genuine test failure exists, finally report itsFAILline instead of dying silently.Evidence (Before & After)
N/A (CI-harness change, not user-visible)
Before: all-green suite, exit 1 —
After: same suite, exit 0 —
Tested on
Environment (optional)
Self-hosted pool machine,
sandbox:noneintegration legs with the repository's model credentials andQWEN_SANDBOX=false; the docker legs were not runnable on this machine (no docker daemon), and the macOS/Windows lanes are covered by the unchanged platform clause.Risk & Scope
QWEN.mdstays modified after the run — it was already un-restorable before, previously at the cost of a red run.$HOMEownership heal analogous to the workspace one) is an operational follow-up, and the startup-stall interactive tests tracked in E2E: external-context mem0 tests hang at 'Connecting to MCP servers' on macOS and ecs-qwen pool (pass on ubuntu-hosted) #10272 are a separate matter — this PR only stops the all-green-exit-red failure class from masking the lane.Linked Issues
Fixes #10325
中文说明
本 PR 做了什么
合并后的 E2E 运行在共享的自托管池上一直以"所有测试全绿却以红色结束、且没有任何可定位的失败测试"的方式失败 —— 运行 33092671484 中有三个 Linux 作业在测试步骤失败，每个作业都是整套测试通过、日志中却没有一条
FAIL行。本 PR 封堵产生这一签名的两个机制。其一，集成测试脚手架会在运行前保存宿主机的全局QWEN.md，并在结束后写回；当池宿主机上的某个特权作业把该文件留成"可读但不可写"时，写回会在所有测试通过之后抛错，把整个运行染红。现在该还原改为尽力而为：记录一条警告（中毒宿主机因此仍可诊断），而不再使运行失败。其二，把"运行器资源压力导致的未处理错误不得使全绿运行失败"的既有豁免扩展到自托管运行器：Linux 分片已从 ubuntu 托管虚机迁到共享池，同样的资源压力就在那里，而 github 托管的 Linux（夜间隔离腿）与本地 Linux 运行继续保留未处理错误信号。真实测试失败在所有运行器上仍会使运行失败。为什么需要
自从 Linux E2E 分片迁到持久化池以来，每次 push 的 E2E 运行都报红却给不出任何失败测试，这耗尽了该通道的信号价值：一条无法二分的红色通道既掩盖真实回归，也会让人习惯性忽略它。还原路径已被端到端复现 —— 在临时
QWEN_HOME下放一个只读的QWEN.md，就能让 6/6 测试全通过的运行以Startup Error: EACCES退出 1；压力抖动路径则与 #10097/#10112 中已在非 Linux 平台豁免的类别相同，其适用范围假设（Linux 分片跑在 ubuntu 托管虚机上）已被迁池推翻。两个修复都保留了全部真实测试失败信号。评审者测试计划
如何验证
mkdir -p /tmp/poison && echo x > /tmp/poison/QWEN.md && chmod 444 /tmp/poison/QWEN.md，然后运行QWEN_HOME=/tmp/poison QWEN_SANDBOX=false npx vitest run --root ./integration-tests test-helper.test.ts。预期：退出码 0、所有测试通过、输出中有一行Warning: could not restore /tmp/poison/QWEN.md。不含本 PR 时，同样的命令会以Startup Error段退出 1（尽管所有测试都通过）。npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/integration-vitest-config.test.ts与QWEN_SANDBOX=false npx vitest run --root ./integration-tests globalSetup.test.ts—— 均为绿色。integration-tests/vitest.config.ts中的self-hosted子句，配置见证会失败；去掉integration-tests/globalSetup.ts中 teardown 的 try/catch，globalSetup 见证会失败。sandbox:none分片 1/3、sandbox:docker分片 1/3 与 3/3）要么变绿，要么在确有真实失败时终于报出那条FAIL行，而不是无声死掉。证据（前后对比）
N/A（CI 脚手架变更，非用户可见）
修复前：全套绿色，退出码 1 ——
修复后：同一套件，退出码 0 ——
测试环境
环境（可选）
自托管池机器，使用仓库的模型凭据以
QWEN_SANDBOX=false运行sandbox:none集成腿；本机没有 docker 守护进程，因此 docker 腿无法在本机运行；macOS/Windows 通道由未改动的平台子句覆盖。风险与范围
QWEN.md在运行后保持被改动状态 —— 在此之前它同样无法还原，只是代价是一次红色运行。$HOME所有权修复）属于运维后续事项；E2E: external-context mem0 tests hang at 'Connecting to MCP servers' on macOS and ecs-qwen pool (pass on ubuntu-hosted) #10272 跟踪的启动停滞交互测试是另一回事 —— 本 PR 只阻止"全绿却红色退出"这一失败类别继续掩盖通道。关联 Issue
Fixes #10325