fix(serenity): fail fast on terminal Semrush sub-workspace provisioning status (LLMO-7352) - #3223
Open
IrisAlexandrescu wants to merge 1 commit into
Open
fix(serenity): fail fast on terminal Semrush sub-workspace provisioning status (LLMO-7352)#3223IrisAlexandrescu wants to merge 1 commit into
IrisAlexandrescu wants to merge 1 commit into
Conversation
IrisAlexandrescu
temporarily deployed
to
dev-branches
September 7, 2026 15:48 — with
GitHub Actions
Inactive
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Sep 9, 2026
IrisAlexandrescu
force-pushed
the
fix/LLMO-7352-workspace-terminal-failure
branch
from
September 9, 2026 15:45
f3d28a1 to
f40444a
Compare
|
This PR will trigger a patch release when merged. |
…ng status (LLMO-7352) A Semrush sub-workspace can settle to a terminal `creation failed` status that never becomes `created`. `pollUntilCreated` only recognized the literal `created`, so it could not tell a terminally-dead workspace apart from a merely-slow `not ready` one: it burned the full ~30s poll budget on a shell that was already dead on the first read, then threw a generic timeout. On the existing-pointer branch (reached on every Add Market retry against an already-failed brand) this repeated on every call — the CUHK/IFC/World Bank/ Kotak incidents (4 of 6 recently-created brands verified stuck in prod). - Recognize the full status superset the gateway actually returns, ported from the already-shipped classification in mysticat-data-service (semrush_write.py): `created`/`active`/`ready` = ready, `creation_failed`/`creation failed`/`failed`/`error` = terminal failure. - Fail fast on the first poll with a stable, sanitized 409 `workspaceProvisioningFailed` instead of a ~30s stall then a generic 504. - Normalize status (lower + trim) before comparison, matching the Python reference — a case/whitespace variant can no longer slip past the checks. - Apply the same READY superset in `findAdoptableFamilyMatch` so a family candidate that settled to `active`/`ready` is adopted, not misread as a zombie and duplicated. - Never embed the Semrush workspace UUID in a client-facing message (it flows through mapError/safeError, which does not redact ids); log it for triage instead — on both the terminal-failure and the timeout paths. Fixes the shared existing-pointer branch, so all 5 `ensureSubworkspace` call sites are protected. First phase of the LLMO-7352 async-provisioning redesign. Introduced by: N/A Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IrisAlexandrescu
force-pushed
the
fix/LLMO-7352-workspace-terminal-failure
branch
from
September 10, 2026 07:33
f40444a to
db0574f
Compare
IrisAlexandrescu
temporarily deployed
to
dev-branches
September 10, 2026 07:44 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Semrush sub-workspace can settle to a terminal
creation failedstatus that never becomescreated.pollUntilCreatedonly recognized the literalcreated, so it could not distinguish a terminally-dead workspace from a merely-slownot readyone — it burned the full ~30s poll budget on a shell that was already dead on the first read, then threw a generic timeout.On the existing-pointer branch (reached on every "Add Market" retry against an already-failed brand), this repeated on every call. This is the mechanism behind the CUHK demo incident (2026-09-03) and, verified live against the Semrush API on 2026-09-07, 4 of 6 recently-created brands (
www.ifc.org,live.worldbank.org,World Bank Arabic,Kotak Neo) currently sit bound to acreation failedworkspace.Related: LLMO-7352 (epic), origin of the
createReadiness: 'skip'behavior is LLMO-6569.What this changes
This is Phase 1 of the LLMO-7352 async-provisioning redesign — the smallest, self-contained slice that stops the bleeding, in the one shared code path (
ensureSubworkspace's existing-pointer branch) reached by all 5ensureSubworkspacecall sites.mysticat-data-service(scripts/serenity_migration/semrush_write.py:104-118):created/active/ready= ready;creation_failed/creation failed/failed/error= terminal failure.409 workspaceProvisioningFailedinstead of a ~30s stall then a generic 504.findAdoptableFamilyMatch, so a family candidate that settled toactive/readyis adopted rather than misread as a zombie and duplicated.mapError/safeError, which does not redact ids) — log it for triage instead, on both the terminal-failure and the timeout paths.creation_failed/creation failedare live-verified;failed/errorare carried defensively from the reference (unobserved here, but inert if they never appear and safer if they do).Testing
test/support/serenity/workspace-lifecycle.test.js: 81 passing (was 61 before this work). New cases cover all terminal strings on both the fresh-create and existing-pointer branches, all ready strings, mixed-case/padded normalization, family adoption ofactive/readycandidates, and that the workspace id is logged (never in the thrown message) on both failure paths.test/support/serenity/**+serenity/brandscontrollers — 2201 passing, 0 failing (nothing downstream depended on the old behavior).eslint+tsc(base + strict) clean.Scope / follow-ups (not in this PR)
Later phases of LLMO-7352 (tracked in LLMO-7418): the durable provisioning state machine, async worker, endpoint contract change (202 + polling), reconciliation for the currently-stuck brands, and an optional reserved-workspace pooling optimization. The shared
isWorkspaceReady/isWorkspaceTerminalFailurehelpers introduced here are the extraction point those phases build on. The reference implementation'screation_failed-vs-failedreaper distinction is intentionally not split here (no reaper in Phase 1).Introduced by: N/A
🤖 Generated with Claude Code