orchestrate: interrupt-safe restart — detect a dead cross-session resume, never re-implement landed work

[TBarregren]

Context — field experience (kntnt-extractor, 2026-07-22, orchestrate 0.16.1; and earlier larger runs)

A 3-issue merge-mode run was killed mid-run by a Claude Code session restart (field trigger, reproducible across several runs: entering the agent view with arrow-left to check progress, then leaving it → a "compact memory?" prompt appears → the background workflow is already dead; the TUI behavior itself is a harness matter outside this plugin, but the plugin must survive it). The completion notification suggested relaunching with resumeFromRunId. That was done — with byte-identical script and args — but Workflow resume is same-session-only, and nothing detected the mismatch: the "resume" silently got zero cache hits and re-ran the entire plan from scratch.

Consequences observed: the engine re-implemented issue #15 — already independently verified and landed on the default branch — from zero on a new branch; the verifier panel flagged the duplication (high severity), a fix round made the branch tree-identical to main, integrate hit a conflict, reconcile rebased, and three redundant commits landed anyway. The engine was in the middle of the same expensive dance for the also-already-landed #16 when the run was manually stopped and a hand-scoped #17-only run finished the job. In an earlier, larger field run the same failure mode compounded across restarts into 50+ agents and 50+ branches, all wasted.

Root cause

1. A dead resume degrades silently to a full re-run. resumeFromRunId is same-session-only, but neither the skill nor the engine detects a cross-session resume; the launch proceeds with an empty cache and no warning.
2. The engine has no idempotence guard. It never checks whether an issue's work already landed before dispatching an implementer. "Landed" exists only in the orchestrator session's memory — lost with the session — and in git history no agent is told to consult.
3. GitHub state cannot disambiguate. Issues are closed only at finalize, so after a mid-run crash a fully-landed issue is indistinguishable from an untouched one by its open/closed state.

Proposed fix

1. Durable per-issue landed-marker (shared mechanism with orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48). At integrate time, record the landing durably and machine-readably — recommended: a structured comment on the GitHub issue (e.g. orchestrate: landed <sha> on <default-branch>, run <runId>), posted by the integrate step (already the authorized mutator). Decision D1: additionally close the issue at integrate time instead of finalize — the inviolable floor ("close only after independent verification and integration") is already satisfied per issue at that moment, and closed-at-integrate makes GitHub state reflect durable progress; the cost is losing the single close-them-together finalize moment.
2. Engine preflight guard (idempotence). Before dispatching an implementer, the engine checks the issue for a landed-marker / closed state (a cheap mechanical-tier dispatch, or folded into the implement prompt's mandatory first step with a distinct status: "already-landed" early return). An already-landed issue is skipped with that distinct status — never parked, never re-implemented, never given a second branch. This bounds the damage of any blind restart, whatever caused it.
3. Skill restart protocol. SKILL.md documents loudly that resumeFromRunId is same-session-only, and prescribes the cross-session restart: never blind-resume — re-run orchestrate.py plan against reality (open issues minus landed-markers), launch a fresh run scoped to the remainder, and hand the new implementers the prior state as context (the recovery that worked by hand on 2026-07-22). If the engine can detect an impossible resume (e.g. an args flag carrying the original session id), it should refuse or park loudly rather than re-run — "detect that the preconditions to continue are gone, then hand over or abort; never start over from zero."

Acceptance criteria

• An engine run whose plan contains an already-landed issue (landed-marker present, or issue closed) dispatches no implementer for it and reports it with a distinct already-landed status — engine-helpers unit-testable.
• After a mid-run kill, a fresh orchestrate invocation scopes only the un-landed remainder: no duplicate implementation branch, no duplicate commits, for any previously-landed issue.
• The landed-marker is written at integrate time, machine-readable, and its format is shared with orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48's milestone reporting.
• SKILL.md states that Workflow resume is same-session-only and specifies the restart protocol; the completion-notification advice can no longer lead a cross-session operator into a silent full re-run.

Companion ticket: #48 (continuous out-of-band progress reporting — the milestone/landed-marker mechanism serves both).

---

🤖 Generated with Claude Code

[TBarregren]

This was generated by AI during triage.

Agent Brief

Category: bug (robustness / interrupt-safety)
Summary: Make an orchestrate run idempotent across a dead cross-session resume — durable per-issue landed-markers, close-at-integrate, and a preflight guard that skips already-landed work instead of blindly re-implementing it.

Current behavior:
The engine's notion of "landed" lives only in the running orchestrator session (an in-session set consulted by the prerequisite-cascade helper). Issues are closed only at the Finalize step, after the whole run. A resumeFromRunId relaunch is a Workflow-tool cache mechanism that is same-session-only; a cross-session relaunch with byte-identical script+args gets zero cache hits and silently re-runs the entire plan from scratch — re-implementing already-landed issues on fresh branches, wasting agents and landing redundant commits. Nothing detects this, and GitHub open/closed state cannot disambiguate a fully-landed issue from an untouched one after a mid-run crash. SKILL.md documents no restart protocol.

Desired behavior:

1. Durable landed-marker (shared format — orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48 reuses it).

   • Merge mode: at integrate time, a structured comment is posted on the issue in the exact canonical form orchestrate: landed <sha> on <branch>, run <runId> (fixed-prefix, positional; no embedded timestamp — the comment's own metadata carries it), and the issue is closed at integrate time (Decision D1), not at Finalize.
   • PR mode (nothing lands on the default branch): a symmetric lighter marker orchestrate: opened PR #<pr>, run <runId> is posted at PR-open; the issue stays open.
   • The canonical format has a single source of truth: a format_landed_marker(...) / parse_landed_marker(...) pair in the deterministic Python CLI (orchestrate.py), designed to be extensible for orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48's richer milestone vocabulary. A consistency test binds the SKILL-documented template to the parser (see acceptance).

2. Writer authority (narrow exception). The orchestrator is blocked on the single Workflow call for the whole run and cannot post per-issue markers mid-run, and the engine has no shell/filesystem I/O. Therefore the integrate step alone gets an explicit, tightly-scoped exception to its otherwise-forbidden outward writes: it MAY post the exact marker comment and (merge mode) close the issue — and nothing else. All other sub-agents (implement, fix, verify, teardown, preflight) remain forbidden from closing/pushing/merging. The verify-then-integrate floor is preserved: integrate runs only after the verifier cleared, so "close only after independent verification AND integration" still holds.

   • Record this contract change as new ADR-0005 ("durable landed-markers + per-issue close-at-integrate"), amending ADR-0001 §5/§7 by reference. Capture the why: the orchestrator-blocked-mid-run constraint, the narrowness of the integrate exception, and floor preservation.

3. Preflight idempotence guard. Before an implementer is dispatched for an issue, a dedicated mechanical-tier preflight sub-agent gathers state (gh issue view --comments, git merge-base --is-ancestor) and returns a structured result (e.g. { alreadyLanded, landedSha }). The engine then branches on it via a pure, unit-testable decision function in the engine-helpers module (mirrored inline in the workflow engine, per the one-top-level-export constraint):

   • Marker present and its SHA is an ancestor of the current default tip → skip the issue with a distinct already-landed status; dispatch no implementer.
   • Marker present but SHA not an ancestor of default (reverted / rebased away / foreign history) → park loudly with a distinct landed-marker-stale status for a human to reconcile. Never rebuild from zero.
   • PR mode, an open orchestrate-opened PR already exists for the issue → benign skip with a distinct already-open status (an existing PR is the expected completed PR-mode state; not a loud park).
   • No marker → dispatch normally.

4. Restart protocol in SKILL.md. Add a loud callout that Workflow resume is same-session-only, and a Cross-session restart subsection prescribing the clean path: re-run orchestrate.py plan scoped to the open-issues-minus-landed-markers remainder, then launch a fresh run over that remainder (the preflight guard makes even a naive relaunch idempotent) — never blind-resume. Correct the completion-notification advice so it can no longer steer a cross-session operator into a silent full re-run. No engine-side resume-detection is attempted: the script receives no resume signal from the harness, so the preflight guard is the safety net, not detection.

Key interfaces:

• format_landed_marker(sha, branch, run_id) / parse_landed_marker(comment_body) in the Python CLI — canonical formatter + tolerant parser (returns structured fields or a null/None on no match), reusable by orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48's status command. Support both the landed and opened PR verbs, or a shape that extends cleanly to them.
• A pure preflight-decision function in the engine-helpers module — inputs are the structured facts the preflight agent returns (marker presence, ancestry boolean, closed state, run mode); output is the guard's verdict (already-landed / landed-marker-stale / already-open / dispatch). No I/O inside it.
• The integrate step's dispatch prompt/constraints — carve the narrow marker+close exception; the marker string it posts must be exactly what parse_landed_marker accepts.
• ADR-0005 markdown under the repo's ADR directory, following the existing ADR house style.

Acceptance criteria:

• A run whose plan contains an already-landed issue (marker present, SHA is an ancestor of default) dispatches no implementer and reports it with a distinct already-landed status.
• A marker present but whose SHA is not an ancestor of default yields a distinct landed-marker-stale park — no implementer, no rebuild.
• In PR mode, an issue with an existing open orchestrate PR is skipped as already-open; no duplicate PR is opened.
• After a mid-run kill, a fresh invocation scoped to the remainder produces no duplicate implementation branch and no duplicate commits for any previously-landed issue.
• Each landed (merge-mode) issue leaves the canonical orchestrate: landed <sha> on <branch>, run <runId> comment and is closed at integrate time; each PR-mode issue leaves the opened PR marker and stays open.
• format_landed_marker/parse_landed_marker round-trip, and reject malformed inputs, under unit tests in test_orchestrate.py.
• A consistency test (in test_orchestrate_skill.py) asserts the marker template SKILL.md tells integrate to post round-trips through parse_landed_marker.
• The engine-helpers preflight-decision function is unit-tested in test_orchestrate_workflow.py across all four verdict cases.
• SKILL.md states Workflow resume is same-session-only, gives the cross-session restart protocol, and the completion-notification advice no longer leads to a silent full re-run — asserted structurally in test_orchestrate_skill.py.
• ADR-0005 exists, amends ADR-0001 §5/§7 by reference, and records the orchestrator-blocked constraint + floor-preserved rationale.
• A cross-issue defect found by the Finalize integration review does not reopen any already-closed issue (issues stay closed; the finding is handled at run level).
• Full suite green: python3 scripts/audit.py and pytest (re-run after any changelog reconcile).

Out of scope (belongs to #48):

• The orchestrate.py status command and reading journal.jsonl.
• The richer milestone vocabulary beyond landed/opened PR (implementation-green, verification-cleared, etc.) — orchestrate: continuous out-of-band progress reporting for multi-hour runs (status command + durable milestone markers) #48 extends the same helper.
• Push notifications.

Out of scope (this ticket, generally):

• Engine-side detection of a dead resume (no harness signal reaches the script; guard is the net).
• Any change to the merge-vs-PR policy, the level ladder, or the inviolable-floor thresholds themselves.
• Reopening issues on a failed integration review.

[TBarregren]

Implemented and committed to main in 411da14 ("orchestrate: interrupt-safe restart — durable landed-markers + preflight idempotence guard").

All 12 acceptance criteria met: durable landed-markers with a single source of truth (format_landed_marker/parse_landed_marker), close-at-integrate (D1), the preflight idempotence guard (preflightDecision with four verdicts), the SKILL restart protocol, and ADR-0005 amending ADR-0001 §5/§7. Full gate green (audit.py + pytest, 533 tests; ruff + mypy clean).

Note: the commit is local on main and not yet pushed to origin — a subsequent push/release will publish it.