fix(agent-runner): self-heal poisoned-resume crash loop - #2670
Closed
ddaniels wants to merge 1 commit into
Closed
Conversation
A session could crash-loop forever on a corrupt resumed transcript. When a turn is interrupted mid-stream (host-sweep kills a stale container, or it crashes) while an assistant message with thinking blocks is still streaming, a partial message lands in the resumed .jsonl. On the next resume the API rejects appending to those blocks: 400 messages.N.content.M: `thinking` or `redacted_thinking` blocks in the latest assistant message cannot be modified. The runner already clears a bad continuation via provider.isSessionInvalid, but that path never fired here for two independent reasons: 1. The SDK surfaces this 400 as a *result* event, not a thrown error, so the catch-block recovery is unreachable. 2. STALE_SESSION_RE wouldn't have matched the text anyway. So the poisoned continuation was never cleared and every queued task re-woke the container into the same failure. The only exit was manually deleting the continuation:claude row — needed twice in production. Fix: detect the poison signature on the result path and self-heal. - Add optional AgentProvider.isPoisonedResume(text) (optional so out-of-tree providers need no change); implement it on ClaudeProvider via a narrow POISON_RESUME_RE. - In processQuery, when a result reports a poisoned resume, suppress the raw error text (it would only land as scratchpad + a pointless unwrapped nudge) and flag the result. - In the outer loop, clear the continuation and retry the same turn once from a fresh session. Bounded to one retry and gated on having actually resumed, so a fresh session that still poisons can't loop. A permanent crash-loop becomes, at worst, one dropped turn that recovers automatically. Tests: unit coverage for isPoisonedResume (matches the real 400, rejects benign/unrelated text) and an integration test that seeds a continuation, emits the poison as a result, and asserts the continuation is cleared and the fresh retry's clean reply is delivered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 2, 2026
Contributor
|
Ported this to our (heavily diverged) downstream fork today and can vouch for the approach: the result-path detection + bounded retry-once design is right, and the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2669.
Problem. A session crash-loops forever on a corrupt resumed transcript (
thinking/redacted_thinkingblocks in the latest assistant message cannot be modified). The existingisSessionInvalidrecovery never fires because the SDK surfaces this 400 as a result event, not a throw, andSTALE_SESSION_REwouldn't match it anyway. Manual deletion of thecontinuation:clauderow was the only escape (twice in prod).Fix.
AgentProvider.isPoisonedResume(text)(optional → out-of-tree providers unaffected); implement onClaudeProvidervia a narrowPOISON_RESUME_RE.processQuery, when a result reports a poisoned resume, suppress the raw error text (it would only land as scratchpad + a pointless unwrapped nudge) and flag the result.A permanent crash-loop becomes at worst one dropped turn that auto-recovers.
Tests. Unit coverage for
isPoisonedResume(matches the real 400, rejects benign/unrelated text) + an integration test that seeds a continuation, emits the poison as a result, and asserts the continuation is cleared and the fresh retry's clean reply is delivered. Full agent-runner suite (106) + container typecheck green.Note. The real API 400 can't be triggered on demand; the integration test simulates it via a provider that emits the poison signature on resume — the realistic end-to-end proof.
🤖 Generated with Claude Code