Skip to content

fix(serenity): fail fast on terminal Semrush sub-workspace provisioning status (LLMO-7352) - #3223

Open
IrisAlexandrescu wants to merge 1 commit into
mainfrom
fix/LLMO-7352-workspace-terminal-failure
Open

fix(serenity): fail fast on terminal Semrush sub-workspace provisioning status (LLMO-7352)#3223
IrisAlexandrescu wants to merge 1 commit into
mainfrom
fix/LLMO-7352-workspace-terminal-failure

Conversation

@IrisAlexandrescu

Copy link
Copy Markdown
Contributor

Problem

A Semrush sub-workspace can settle to a terminal creation failed status that never becomes created. pollUntilCreated only recognized the literal created, so it could not distinguish a terminally-dead workspace from a merely-slow not ready one — it burned the full ~30s poll budget on a shell that was already dead on the first read, then threw a generic timeout.

On the existing-pointer branch (reached on every "Add Market" retry against an already-failed brand), this repeated on every call. This is the mechanism behind the CUHK demo incident (2026-09-03) and, verified live against the Semrush API on 2026-09-07, 4 of 6 recently-created brands (www.ifc.org, live.worldbank.org, World Bank Arabic, Kotak Neo) currently sit bound to a creation failed workspace.

Related: LLMO-7352 (epic), origin of the createReadiness: 'skip' behavior is LLMO-6569.

What this changes

This is Phase 1 of the LLMO-7352 async-provisioning redesign — the smallest, self-contained slice that stops the bleeding, in the one shared code path (ensureSubworkspace's existing-pointer branch) reached by all 5 ensureSubworkspace call sites.

  • Recognize the full status superset the gateway actually returns, ported from the already-shipped classification in mysticat-data-service (scripts/serenity_migration/semrush_write.py:104-118): created/active/ready = ready; creation_failed/creation failed/failed/error = terminal failure.
  • Fail fast on the first poll with a stable, sanitized 409 workspaceProvisioningFailed instead of a ~30s stall then a generic 504.
  • Normalize status (lower + trim) before comparison, matching the Python reference — a case/whitespace variant can no longer slip past the checks.
  • Apply the same READY superset in findAdoptableFamilyMatch, so a family candidate that settled to active/ready is adopted rather than misread as a zombie and duplicated.
  • Never leak the Semrush workspace UUID in a client-facing message (it flows through mapError/safeError, which does not redact ids) — log it for triage instead, on both the terminal-failure and the timeout paths.

creation_failed/creation failed are live-verified; failed/error are carried defensively from the reference (unobserved here, but inert if they never appear and safer if they do).

Testing

  • test/support/serenity/workspace-lifecycle.test.js: 81 passing (was 61 before this work). New cases cover all terminal strings on both the fresh-create and existing-pointer branches, all ready strings, mixed-case/padded normalization, family adoption of active/ready candidates, and that the workspace id is logged (never in the thrown message) on both failure paths.
  • Broader run: test/support/serenity/** + serenity/brands controllers — 2201 passing, 0 failing (nothing downstream depended on the old behavior).
  • eslint + tsc (base + strict) clean.

Scope / follow-ups (not in this PR)

Later phases of LLMO-7352 (tracked in LLMO-7418): the durable provisioning state machine, async worker, endpoint contract change (202 + polling), reconciliation for the currently-stuck brands, and an optional reserved-workspace pooling optimization. The shared isWorkspaceReady/isWorkspaceTerminalFailure helpers introduced here are the extraction point those phases build on. The reference implementation's creation_failed-vs-failed reaper distinction is intentionally not split here (no reaper in Phase 1).

Introduced by: N/A

🤖 Generated with Claude Code

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

This PR will trigger a patch release when merged.

…ng status (LLMO-7352)

A Semrush sub-workspace can settle to a terminal `creation failed` status that
never becomes `created`. `pollUntilCreated` only recognized the literal
`created`, so it could not tell a terminally-dead workspace apart from a
merely-slow `not ready` one: it burned the full ~30s poll budget on a shell
that was already dead on the first read, then threw a generic timeout. On the
existing-pointer branch (reached on every Add Market retry against an
already-failed brand) this repeated on every call — the CUHK/IFC/World Bank/
Kotak incidents (4 of 6 recently-created brands verified stuck in prod).

- Recognize the full status superset the gateway actually returns, ported from
  the already-shipped classification in mysticat-data-service
  (semrush_write.py): `created`/`active`/`ready` = ready,
  `creation_failed`/`creation failed`/`failed`/`error` = terminal failure.
- Fail fast on the first poll with a stable, sanitized 409
  `workspaceProvisioningFailed` instead of a ~30s stall then a generic 504.
- Normalize status (lower + trim) before comparison, matching the Python
  reference — a case/whitespace variant can no longer slip past the checks.
- Apply the same READY superset in `findAdoptableFamilyMatch` so a family
  candidate that settled to `active`/`ready` is adopted, not misread as a
  zombie and duplicated.
- Never embed the Semrush workspace UUID in a client-facing message (it flows
  through mapError/safeError, which does not redact ids); log it for triage
  instead — on both the terminal-failure and the timeout paths.

Fixes the shared existing-pointer branch, so all 5 `ensureSubworkspace` call
sites are protected. First phase of the LLMO-7352 async-provisioning redesign.

Introduced by: N/A

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@IrisAlexandrescu
IrisAlexandrescu force-pushed the fix/LLMO-7352-workspace-terminal-failure branch from f40444a to db0574f Compare September 10, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant