feat(serenity): opt-in async provisioning for createMarket, createBrandForOrg, activate (LLMO-7352/LLMO-7418) by IrisAlexandrescu · Pull Request #3246 · adobe/spacecat-api-service

IrisAlexandrescu · 2026-09-09T11:12:52Z

Summary

Third PR in the LLMO-7352/LLMO-7418 stack (after #3223 and #3233). Converts the three remaining synchronous Semrush sub-workspace/project provisioning endpoints — createMarket, createBrandForOrg's initial-market create, and activate's project-activation branch — to use the async worker built in #3233, gated by a per-request opt-in flag rather than an unconditional cutover.

Opt-in flag, not a cutover: async: true in the request body enqueues provision-workspace-job chained to the relevant follow-up job and returns 202 with a pollable jobId. Absent/false (the default) runs the exact same synchronous logic these endpoints have always run — this mirrors the existing createPrompts async flag precedent already in this codebase, so no current consumer's behavior changes until they explicitly ask for async. This flag is a migration aid, not a permanent dual-mode API surface, and should be retired once consumers move over.
Shared orchestration: business logic reachable from both the sync controller and the async job handler now lives in create-market-orchestration.js / activate-markets-orchestration.js, called directly by the controller (sync) or by a thin job-handler adapter (async).
Concurrency safety: guardAgainstConcurrentProvisioning is called from every synchronous branch so a sync request can't race a live async attempt for the same brand (and vice versa).
Fresh-brand compensation: if beginProvisioningAttempt itself throws right after a brand row is persisted in createBrandForOrg, recordFreshBrandProvisioningStartFailure marks that just-created row failed (unconditional update — safe only because the brand id was minted moments earlier in the same request) instead of leaving it silently inert.
Polling: adds a generic /serenity/jobs/:jobId route alias (job-type-agnostic, same handler as the existing prompts-jobs route).

Reviewed with a 4-perspective adversarial pass (architect / race-condition / senior-engineer-correctness / test-coverage) and a security review; the two real bugs and test gaps the code review surfaced are fixed and covered here. The security review found no high-confidence issues; one medium-confidence, same-tenant-only gap (an unredacted upstream error message can reach the new job-status polling response for the brand's own owner, in code paths outside the per-market sanitizing catch) is a good fast-follow but isn't a blocker since it discloses no capability beyond what the caller already holds.

Test plan

npx mocha test/controllers/serenity.test.js test/controllers/brands.test.js test/support/brands-storage.test.js — 924 passing, 0 failing
npm run lint — clean
npm run type-check — clean
New unit tests: malformed async value → 400, explicit async: false regression, beginProvisioningAttempt-throws compensation path, createMarket concurrency-guard wiring/409
create-market-orchestration.test.js / activate-markets-orchestration.test.js / *-job.test.js cover the extracted orchestration + job-handler adapters exhaustively

🤖 Generated with Claude Code

codecov · 2026-09-09T11:16:59Z

Codecov Report

❌ Patch coverage is 98.31522% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/support/brands-storage.js	92.00%	16 Missing ⚠️
src/controllers/brands.js	95.51%	10 Missing ⚠️
src/controllers/serenity.js	98.59%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

…de call sites (LLMO-7352/LLMO-7418 Phase 4) Extends the opt-in `async: true` contract (PR-C, #3246) to the last 3 ensureSubworkspace call sites that were still fully synchronous: createBrandForOrg's bare-create (no semrushMarket) branch, and activate's pending->active and bare-reactivation branches. These 3 sites use ensureSubworkspace's `createReadiness: 'skip'` mode (persist the workspace pointer immediately after a single create call, without polling for settle) rather than 'poll' mode, so they never had the in-request settle-poll latency/timeout problem PR-A/PR-B/PR-C fixed for the other 2 sites. This is deliberate hardening for consistency and defense against a slow/hanging single Semrush create call, not a fix for a still-open incident — the original LLMO-7352 symptom is already closed by PR-A alone, since any workspace pointer's first real use (e.g. adding a market) always runs the now-fail-fast pollUntilCreated first. - createBrandForOrg's bare-create branch mirrors its sibling hasSemrushMarket branch exactly: async:true persists the row first, then hands off to provision-workspace-job with no chained job (no project to create). - activate's two skip-mode branches each mint a provisioning attempt and chain to a new activate-brand-workspace-job, which does the one piece of business logic the generic worker doesn't: flipping (or re-affirming) the brand's status to active once the workspace is confirmed ready. One job handler covers both branches; `wasPending` in its metadata distinguishes a real pending->active transition (502 on save failure) from a no-op reactivation re-affirm (207). - Existing guardAgainstConcurrentProvisioning calls on the sync branches are unchanged (still needed to protect a sync caller against a concurrent async attempt from another endpoint). Stacked on PR-C (#3246). 30 new unit tests; 954 passing overall, 0 regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-09T16:38:45Z

This PR will trigger a minor release when merged.

…de call sites (LLMO-7352/LLMO-7418 Phase 4) Extends the opt-in `async: true` contract (PR-C, #3246) to the last 3 ensureSubworkspace call sites that were still fully synchronous: createBrandForOrg's bare-create (no semrushMarket) branch, and activate's pending->active and bare-reactivation branches. These 3 sites use ensureSubworkspace's `createReadiness: 'skip'` mode (persist the workspace pointer immediately after a single create call, without polling for settle) rather than 'poll' mode, so they never had the in-request settle-poll latency/timeout problem PR-A/PR-B/PR-C fixed for the other 2 sites. This is deliberate hardening for consistency and defense against a slow/hanging single Semrush create call, not a fix for a still-open incident — the original LLMO-7352 symptom is already closed by PR-A alone, since any workspace pointer's first real use (e.g. adding a market) always runs the now-fail-fast pollUntilCreated first. - createBrandForOrg's bare-create branch mirrors its sibling hasSemrushMarket branch exactly: async:true persists the row first, then hands off to provision-workspace-job with no chained job (no project to create). - activate's two skip-mode branches each mint a provisioning attempt and chain to a new activate-brand-workspace-job, which does the one piece of business logic the generic worker doesn't: flipping (or re-affirming) the brand's status to active once the workspace is confirmed ready. One job handler covers both branches; `wasPending` in its metadata distinguishes a real pending->active transition (502 on save failure) from a no-op reactivation re-affirm (207). - Existing guardAgainstConcurrentProvisioning calls on the sync branches are unchanged (still needed to protect a sync caller against a concurrent async attempt from another endpoint). Stacked on PR-C (#3246). 30 new unit tests; 954 passing overall, 0 regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ndForOrg, activate (LLMO-7352/LLMO-7418) Converts the three remaining synchronous Semrush sub-workspace/project provisioning endpoints to an opt-in async model, gated by a per-request `async: true` flag (mirroring the existing `createPrompts` precedent) so default behavior is unchanged for every current consumer. - createMarket, createBrandForOrg's initial market, and activate's project-activation branch now mint a provisioning attempt and enqueue provision-workspace-job -> {create-market,activate-markets}-job when `async: true` is supplied, returning 202 with a pollable jobId. - Business logic shared between the sync controller path and the async job handler is extracted into create-market-orchestration.js and activate-markets-orchestration.js. - guardAgainstConcurrentProvisioning protects every synchronous branch against racing a live async attempt for the same brand. - Adds a generic /serenity/jobs/:jobId polling alias. - Adds recordFreshBrandProvisioningStartFailure so a brand row created in the same request is marked failed (not left silently inert) if minting the provisioning attempt itself throws. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…-sites reconcile stale attempts (LLMO-7418 external-review Findings 1, 9, 11) Finding 1: guardAgainstConcurrentProvisioning's read (getBrandProvisioningState) threw a bare Error on ANY PostgREST failure, including "column does not exist" — meaning if this service ever deploys even briefly ahead of the mysticat-data-service migrations that add the provisioning columns, every synchronous createMarket/activate call 500s on an endpoint that worked a moment ago. Preserve the underlying SQLSTATE on the thrown error and degrade to a no-op specifically for 42703 (undefined_column) — no column means no attempt could possibly be in flight, so proceeding is correct, not just convenient. Any other read error still fails closed, unchanged. Finding 11 (adjacent, same function): a missing/unparseable updated_at yielded NaN, which is never < the staleness threshold, so it fell through to "reconcile as stale" — capable of tearing down a genuinely fresh, healthy attempt. Treat NaN as "assume fresh" instead (409, never silently reconciled away). Finding 9: beginProvisioningAttempt's own CAS has no staleness awareness, so a stuck pending row (crashed worker, DLQ'd message) 409s forever once a brand's callers are async-only — there is no scheduled sweep (architecturally impossible; Semrush only accepts user-token auth). Call guardAgainstConcurrentProvisioning immediately before beginProvisioningAttempt at the two async begin-sites in this branch (createMarket, activate's project-activation batch) that operate on a pre-existing brand id — reusing the guard's own tested reconcile-or-409 logic rather than inventing new logic. The brand-create async sites (createBrandForOrg) are NOT touched: they mint a brand-new, freshly-generated UUID in the same request, so no prior attempt could exist for it — adding the guard there would be a pure no-op read on every call. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…terminal hop (LLMO-7418 external-review Finding 8) The runner marks the ORIGINAL AsyncJob COMPLETED on any non-throwing handler return, including a self-requeue ({ requeuedJobId }, workspace not yet settled) or a chain hand-off ({ provisioningStatus: 'ready', chainedJobId }, the chained market-create/activate work not yet started). getPromptsJobStatus (the same handler serving the generic /serenity/jobs/:jobId polling alias) just echoed that first hop's status/result verbatim, so a client polling during a chain saw a premature "COMPLETED" the moment the FIRST hop merely handed off, with no market created and no way to tell the difference from a genuinely finished request. The frontend consumers built against this contract this session all resolve on the first COMPLETED with no chain awareness. Fix is entirely backend-scoped, so no frontend change is needed: when the polled job is COMPLETED and its result carries chainedJobId or requeuedJobId, follow that pointer (recursively, capped at 10 hops against a corrupt/cyclic chain) and report the EFFECTIVE terminal hop's status/result/ error instead. jobId/jobType in the response still reflect the ORIGINALLY- requested job, so a caller polling a fixed URL never needs to learn intermediate hop ids. Updated the two OpenAPI descriptions that previously overclaimed "result is the same body a synchronous 2xx would have returned" without qualifying that this only became true once the whole chain, not just the first hop, had run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…op job id (LLMO-7418 external-review Findings 12, 17) - provision-workspace-job.js: enqueueChainedJobIfConfigured now retries up to 3 times with backoff before giving up, since a transient enqueue failure right after ready-promotion previously stranded a ready workspace with no follow-up job and no retry. - serenity.js: createMarket and activate's async batch-market branch now call updateProvisioningJobId (best-effort) immediately after the first hop's createAndEnqueueJob, so the job id is recorded even if the worker never reaches its own self-requeue path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…path, cancel-first on deactivate (LLMO-7418 external-review N4/N5, N2, Finding 3) - N4/N5 (Option B): the shared getPromptsJobStatus handler now keys off the job family. Provisioning jobs report jobType 'provisionWorkspace' (not the classifyPrompts default) and have their requeue/chain followed to the effective terminal hop; classifyPrompts jobs are no longer chain-followed, restoring their pre-stack polling contract (a large CSV import that self-requeues again returns its hop-0 {created,skipped} result, not a later hop's shape). - N2: activate's async batch-market branch passes the brand name as the sub-workspace title. An active flat-mode brand reaches this branch with no pointer, so the worker takes its create path -- which fail-fasts without a title. createMarket deliberately does NOT pass one (subworkspace-mode invariant means the worker polls the existing pointer; the rare pointer-cleared race is an acceptable clear fail-fast, since minting a workspace mid add-market is not a valid outcome). - Finding 3 (deactivate half): cancelProvisioningAttempt now runs FIRST in deactivate, before decommission/clear/save. Once it flips provisioning_status to 'failed' a concurrent worker hop can no longer win its promote CAS and resurrect the brand, and it is no longer skipped when the pointer/status save throws. Tests: N4 label, N5 no-follow-for-classify, activate title, chain-follow fixtures marked as provisioning jobs. serenity suite 237 passing. eslint + type-check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…reporting success (LLMO-7418 external-review Finding 12) After the workspace is promoted ready, if the configured chained job (the market-create / activate work the caller actually requested) cannot be enqueued even after the bounded retries, the handler now THROWS instead of returning a plain `{ provisioningStatus: 'ready' }`. Previously that plain return made the runner mark the job COMPLETED and the client poll back a green success while no market was ever created and nothing would retry it. A new `candidatePromoted` flag makes the throw safe: once promoteProvisioningReady succeeds the candidate is the brand's canonical, live workspace, so the outer catch skips BOTH cleanup (never empty a live workspace) and failBestEffort (never mark an already-ready attempt failed). The job goes FAILED so the caller learns the chained work did not run; the brand stays ready/active and the caller can safely re-issue the request. Tests: the Finding 12 test now asserts the throw + no-deface + no-cleanup + full retry count. provision-workspace-job suite 43 passing. eslint + type-check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ling the attempt (LLMO-7418 external-review, adversarial) deactivate loads the brand BEFORE cancelProvisioningAttempt, so its in-memory semrush_sub_workspace_id is a pre-cancel snapshot. A worker hop whose promoteProvisioningReady committed in the loadBrand -> cancel window has since written a canonical pointer that object cannot see; acting on the stale null skipped the entire decommission / pointer-clear / tombstone block, leaving that workspace live (with the customer's projects intact) behind a "successful" deactivate. Re-read the pointer via the existing brandPointerReloader after the cancel and prefer the fresh value, falling back to the snapshot when the reloader cannot read (it is best-effort by contract and returns null on missing data-access). The cancel-first ordering already closed the resurrection half of this race; this closes the stale-read half. Test: a promote landing mid-flight (findById returns the stale snapshot first, the promoted brand second) is still decommissioned with the freshly-promoted workspace id. serenity suite 238 passing. eslint + type-check clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…de call sites (LLMO-7352/LLMO-7418 Phase 4) Extends the opt-in `async: true` contract (PR-C, #3246) to the last 3 ensureSubworkspace call sites that were still fully synchronous: createBrandForOrg's bare-create (no semrushMarket) branch, and activate's pending->active and bare-reactivation branches. These 3 sites use ensureSubworkspace's `createReadiness: 'skip'` mode (persist the workspace pointer immediately after a single create call, without polling for settle) rather than 'poll' mode, so they never had the in-request settle-poll latency/timeout problem PR-A/PR-B/PR-C fixed for the other 2 sites. This is deliberate hardening for consistency and defense against a slow/hanging single Semrush create call, not a fix for a still-open incident — the original LLMO-7352 symptom is already closed by PR-A alone, since any workspace pointer's first real use (e.g. adding a market) always runs the now-fail-fast pollUntilCreated first. - createBrandForOrg's bare-create branch mirrors its sibling hasSemrushMarket branch exactly: async:true persists the row first, then hands off to provision-workspace-job with no chained job (no project to create). - activate's two skip-mode branches each mint a provisioning attempt and chain to a new activate-brand-workspace-job, which does the one piece of business logic the generic worker doesn't: flipping (or re-affirming) the brand's status to active once the workspace is confirmed ready. One job handler covers both branches; `wasPending` in its metadata distinguishes a real pending->active transition (502 on save failure) from a no-op reactivation re-affirm (207). - Existing guardAgainstConcurrentProvisioning calls on the sync branches are unchanged (still needed to protect a sync caller against a concurrent async attempt from another endpoint). Stacked on PR-C (#3246). 30 new unit tests; 954 passing overall, 0 regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…of writing to the org parent workspace (LLMO-7418 external-review B1) A brand whose sub-workspace pointer is not yet written resolves to `mode: 'flat'` with `workspaceId = the ORG'S SHARED PARENT` (workspace-resolver.js: if the brand has no sub-workspace id it returns the parent). That is correct for a genuinely flat brand and WRONG for a Semrush brand that is merely mid-provisioning. Async creation makes that window reachable: the brand is persisted before Semrush confirms the workspace, so for the whole provisioning window it looks flat. Add Market in that window created and PUBLISHED a project in the shared org workspace -- consuming the org's shared allocation, bound in the DB to this brand -- and the moment the real pointer landed the brand flipped to sub-workspace mode and that market became invisible: an orphaned upstream project plus a stale mapping row. The window is the full workspace create + settle poll (tens of seconds to minutes), and the create wizard's own success screen walks the user straight into Add Market, so the race is normally lost rather than rarely. createMarket's flat branch now reads the brand's provisioning state and refuses with a 409 (`semrush_provisioning_incomplete`) when an attempt is `pending` or `failed` -- both mean this is a sub-workspace brand with no workspace of its own yet. A genuinely flat brand (non-Serenity org) has no provisioning row and is unaffected. The state read is best-effort: a read failure never turns an otherwise-valid flat create into an error. This is the server-side half, and the half that actually closes the hole -- UI gating of the Add Market affordance on semrushProvisioningStatus is the remaining (frontend) half, tracked for LLMO-7419 along with the epic's AC17. Tests: both refusal cases pinned (pending and failed), asserting the flat handler is never reached. serenity suite 240 passing. eslint + type-check clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…de call sites (LLMO-7352/LLMO-7418 Phase 4) Extends the opt-in `async: true` contract (PR-C, #3246) to the last 3 ensureSubworkspace call sites that were still fully synchronous: createBrandForOrg's bare-create (no semrushMarket) branch, and activate's pending->active and bare-reactivation branches. These 3 sites use ensureSubworkspace's `createReadiness: 'skip'` mode (persist the workspace pointer immediately after a single create call, without polling for settle) rather than 'poll' mode, so they never had the in-request settle-poll latency/timeout problem PR-A/PR-B/PR-C fixed for the other 2 sites. This is deliberate hardening for consistency and defense against a slow/hanging single Semrush create call, not a fix for a still-open incident — the original LLMO-7352 symptom is already closed by PR-A alone, since any workspace pointer's first real use (e.g. adding a market) always runs the now-fail-fast pollUntilCreated first. - createBrandForOrg's bare-create branch mirrors its sibling hasSemrushMarket branch exactly: async:true persists the row first, then hands off to provision-workspace-job with no chained job (no project to create). - activate's two skip-mode branches each mint a provisioning attempt and chain to a new activate-brand-workspace-job, which does the one piece of business logic the generic worker doesn't: flipping (or re-affirming) the brand's status to active once the workspace is confirmed ready. One job handler covers both branches; `wasPending` in its metadata distinguishes a real pending->active transition (502 on save failure) from a no-op reactivation re-affirm (207). - Existing guardAgainstConcurrentProvisioning calls on the sync branches are unchanged (still needed to protect a sync caller against a concurrent async attempt from another endpoint). Stacked on PR-C (#3246). 30 new unit tests; 954 passing overall, 0 regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This was referenced Sep 9, 2026

feat(serenity): opt-in async provisioning for the 3 remaining skip-mode call sites (LLMO-7352/LLMO-7418 Phase 4) #3249

Open

feat(serenity): expose semrush provisioning status on brand reads (LLMO-7352/LLMO-7418) #3250

Open

IrisAlexandrescu force-pushed the feat/LLMO-7418-provisioning-worker branch from 70fe12d to ae4b0a9 Compare September 9, 2026 16:00

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch from 57504cb to 2d095bd Compare September 9, 2026 16:14

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch from fafceee to 6064817 Compare September 9, 2026 17:24

IrisAlexandrescu temporarily deployed to dev-branches September 9, 2026 17:59 — with GitHub Actions Inactive

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch 2 times, most recently from 8e279ab to 553f05a Compare September 9, 2026 18:25

IrisAlexandrescu temporarily deployed to dev-branches September 9, 2026 18:57 — with GitHub Actions Inactive

IrisAlexandrescu force-pushed the feat/LLMO-7418-provisioning-worker branch from fce9e83 to 09c741f Compare September 10, 2026 07:42

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch from 553f05a to ab699df Compare September 10, 2026 08:00

IrisAlexandrescu temporarily deployed to dev-branches September 10, 2026 08:12 — with GitHub Actions Inactive

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch from ab699df to aae939f Compare September 10, 2026 10:23

IrisAlexandrescu temporarily deployed to dev-branches September 10, 2026 10:36 — with GitHub Actions Inactive

Iris Alexandrescu and others added 7 commits September 10, 2026 14:13

IrisAlexandrescu force-pushed the feat/LLMO-7418-async-provisioning-endpoints branch from aae939f to 3a1ffe9 Compare September 10, 2026 11:17

IrisAlexandrescu temporarily deployed to dev-branches September 10, 2026 11:26 — with GitHub Actions Inactive

IrisAlexandrescu deployed to dev-branches September 10, 2026 18:29 — with GitHub Actions Active

Navigation Menu

Uh oh!

feat(serenity): opt-in async provisioning for createMarket, createBrandForOrg, activate (LLMO-7352/LLMO-7418) - #3246

feat(serenity): opt-in async provisioning for createMarket, createBrandForOrg, activate (LLMO-7352/LLMO-7418)#3246
IrisAlexandrescu wants to merge 8 commits into
feat/LLMO-7418-provisioning-workerfrom
feat/LLMO-7418-async-provisioning-endpoints

IrisAlexandrescu commented Sep 9, 2026

Uh oh!

codecov Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

IrisAlexandrescu commented Sep 9, 2026

Summary

Test plan

Uh oh!

codecov Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Sep 9, 2026 •

edited

Loading