Skip to content

fix(agent-runner): self-heal poisoned-resume crash loop - #2670

Closed
ddaniels wants to merge 1 commit into
nanocoai:mainfrom
ddaniels:skill/thinking-poison-selfheal
Closed

fix(agent-runner): self-heal poisoned-resume crash loop#2670
ddaniels wants to merge 1 commit into
nanocoai:mainfrom
ddaniels:skill/thinking-poison-selfheal

Conversation

@ddaniels

@ddaniels ddaniels commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #2669.

Problem. A session crash-loops forever on a corrupt resumed transcript (thinking/redacted_thinking blocks in the latest assistant message cannot be modified). The existing isSessionInvalid recovery never fires because the SDK surfaces this 400 as a result event, not a throw, and STALE_SESSION_RE wouldn't match it anyway. Manual deletion of the continuation:claude row was the only escape (twice in prod).

Fix.

  • Add optional AgentProvider.isPoisonedResume(text) (optional → out-of-tree providers unaffected); implement on ClaudeProvider via a narrow POISON_RESUME_RE.
  • In processQuery, when a result reports a poisoned resume, suppress the raw error text (it would only land as scratchpad + a pointless unwrapped nudge) and flag the result.
  • In the outer loop, clear the continuation and retry the turn once from a fresh session — bounded to one retry, gated on having actually resumed, so a fresh session that still poisons can't loop.

A permanent crash-loop becomes at worst one dropped turn that auto-recovers.

Tests. Unit coverage for isPoisonedResume (matches the real 400, rejects benign/unrelated text) + an integration test that seeds a continuation, emits the poison as a result, and asserts the continuation is cleared and the fresh retry's clean reply is delivered. Full agent-runner suite (106) + container typecheck green.

Note. The real API 400 can't be triggered on demand; the integration test simulates it via a provider that emits the poison signature on resume — the realistic end-to-end proof.

🤖 Generated with Claude Code

A session could crash-loop forever on a corrupt resumed transcript. When a
turn is interrupted mid-stream (host-sweep kills a stale container, or it
crashes) while an assistant message with thinking blocks is still streaming,
a partial message lands in the resumed .jsonl. On the next resume the API
rejects appending to those blocks:

  400 messages.N.content.M: `thinking` or `redacted_thinking` blocks in the
  latest assistant message cannot be modified.

The runner already clears a bad continuation via provider.isSessionInvalid,
but that path never fired here for two independent reasons:

1. The SDK surfaces this 400 as a *result* event, not a thrown error, so the
   catch-block recovery is unreachable.
2. STALE_SESSION_RE wouldn't have matched the text anyway.

So the poisoned continuation was never cleared and every queued task re-woke
the container into the same failure. The only exit was manually deleting the
continuation:claude row — needed twice in production.

Fix: detect the poison signature on the result path and self-heal.

- Add optional AgentProvider.isPoisonedResume(text) (optional so out-of-tree
  providers need no change); implement it on ClaudeProvider via a narrow
  POISON_RESUME_RE.
- In processQuery, when a result reports a poisoned resume, suppress the raw
  error text (it would only land as scratchpad + a pointless unwrapped nudge)
  and flag the result.
- In the outer loop, clear the continuation and retry the same turn once from
  a fresh session. Bounded to one retry and gated on having actually resumed,
  so a fresh session that still poisons can't loop.

A permanent crash-loop becomes, at worst, one dropped turn that recovers
automatically.

Tests: unit coverage for isPoisonedResume (matches the real 400, rejects
benign/unrelated text) and an integration test that seeds a continuation,
emits the poison as a result, and asserts the continuation is cleared and the
fresh retry's clean reply is delivered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sturdy4days

Copy link
Copy Markdown
Contributor

Ported this to our (heavily diverged) downstream fork today and can vouch for the approach: the result-path detection + bounded retry-once design is right, and the isPoisonedResume provider method composes cleanly even where the poll-loop has drifted a long way from main. We've hit the crash-loop this fixes twice in production (manual continuation:claude row deletion was our only recovery), so +1 to merging. The regex matched both real-world poison signatures we'd logged (thinking and redacted_thinking variants).

@ddaniels ddaniels closed this Jun 13, 2026
@ddaniels
ddaniels deleted the skill/thinking-poison-selfheal branch June 13, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent-runner: corrupt resumed transcript ("thinking blocks cannot be modified" 400) crash-loops forever instead of self-healing

2 participants