refactor(nlp): remove the python langwatch_nlp service, nlpgo is the only engine#4653
Merged
Conversation
The Python langwatch_nlp service is being removed; nlpgo becomes the sole NLP engine. Remove the uvicorn child manager, the reverse proxy, and all NLPGO_CHILD_* config from the Go service. Any non-/go/* request now returns the go-only 502 unconditionally (no proxy fall-through). Delete the uvicornchild + proxypass adapter packages and their tests; update the lifecycle tests to assert the child-free service set. Rewrite specs/nlp-go to the Go-only end state: drop the parallel-deployment, feature-flag, and front-door specs, add python-removal.feature, repoint topic-clustering at langevals unconditionally, and update the shared contract. Seed the code-block sandbox-requirements.txt (httpx, requests, pydantic, langwatch); the artifact keeps a minimal python3 only for code blocks.
With the Python langwatch_nlp service removed, nlpgo is the only NLP engine and topic clustering lives in langevals. Remove the release_nlp_go_engine_enabled feature flag and every Python-fallback branch it gated: studio execution and the playground proxy always hit nlpgo's /go/* paths; topic clustering always posts to langevals (dropping the langwatch_nlp / TOPIC_CLUSTERING_SERVICE fallback and the fetch-h2 path); execute_optimization always returns 410. Delete the flag from the registry, the isNlpGoEnabled helper, and the TOPIC_CLUSTERING_SERVICE env. Update the affected unit tests (featureFlag fixtures, post-event routing, topic-clustering staged-fetch).
…x python Replace the python:3.12-slim + uvicorn + litellm + langwatch_nlp image with a distroless/python3 image carrying only the static Go binary plus a curated code-block sandbox (requests, httpx, pydantic, langwatch from sandbox-requirements.txt). 136MB, down from ~1GB+. The Go binary is the only long-lived process; python3.11 is spawned per code-block node and exits. Verified end-to-end: /healthz 200, /go/version served, non-/go returns the go-only 502, and the sandbox python imports the curated libs (cp311 ABI match against distroless/python3-debian12). Babel's CLDR locale data is pruned to en+root (~30MB saved); each *.dist-info is kept because opentelemetry resolves its runtime context via importlib.metadata entry points.
…time Delete the Python langwatch_nlp service entirely and make the @langwatch/server npx self-host Go-only. The npx-server already ran nlpgo from the aigateway monobinary in its default 'go' mode; this removes the legacy 'python' (uvicorn) mode: delete startLangwatchNlp, the --nlp CLI flag + parseNlpMode, nlpMode from the runtime contract / env scaffold / cli, and the langwatch_nlp venv + project wiring. Drop langwatch_nlp from the python feature-parity test roots and the package files list, and update the affected packages/server tests. Keep LANGWATCH_NLP_SERVICE (still the nlpgo base URL), langevals, and the uv predep (langevals needs it).
The langwatch_nlp image is now a distroless Go binary (no litellm/dspy/uvicorn), so the deployment drops the python-tier resources to requests 250m/256Mi, limits 1cpu/1Gi, repoints the probes at the Go endpoints (/healthz liveness, /readyz readiness, /startupz startup), and pins SERVER_ADDR to the service port. LANGEVALS_ENDPOINT is already wired on the app (topic clustering routes there). Updated the example/test overlays + README resource rows.
…erfile Finish the langwatch_nlp removal across CI and local dev. CI: langwatch-nlp-ci.yml drops the python lint/test/integration jobs (the dir is gone) and keeps the Go-image build smoke check (the Go source is covered by go-ci.yaml); strip the langwatch_nlp pyproject/uv.lock path filters from the npx-server workflows and the codeql path filter. Delete Dockerfile.langwatch_nlp.lambda: broken after the dir deletion and unbuilt by monorepo CI (the saas owns its own lambda runtime image). Dev: the all-local-nlp preset runs the Go nlpgo (make service svc=nlpgo) instead of uvicorn; compose services drop NLPGO_CHILD_* env; dev scripts + CLAUDE.md structure updated.
Contributor
|
Important Review skippedToo many files! This PR contains 170 files, which is 20 over the limit of 150. To get a review, narrow the scope: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (170)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
langwatch-agent
left a comment
langwatch-agent
left a comment
Contributor
There was a problem hiding this comment.
2 findings (1 P1, 1 P2) — scenario NLP callers still hit removed legacy paths, and the Helm full-stack smoke still checks the removed /health endpoint. Details inline.
The go-only 502 fallback (this PR) breaks live callers that bypass nlpgoFetch:
scenario code/workflow agents posted to ${nlpServiceUrl}/studio/execute_sync and
the scenario model factory proxied via /proxy/v1, both now 502. The helm
full-stack smoke also still hit the removed /health. Route the adapters through
/go/studio/execute_sync, the model factory through /go/proxy/v1, and the smoke
through /healthz; update the asserting unit tests.
…scales langevals The nlpgo service is a lean Go binary, so the chart default and the prod/HA overlays request far less than the old Python service. Update the sizing & scaling tables/profiles to match (250m/1 CPU, 256Mi/1Gi), and fix the scaling guide: topic clustering runs on langevals now, not the NLP service. Regenerate llms-full.txt. Also trim a backward-looking comparison from the values.yaml resources comment (describe current state, not history).
0xdeafcafe
added a commit
that referenced
this pull request
Jun 20, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jun 23, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jun 24, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
xuzhu007
added a commit
to xuzhu007/langwatch
that referenced
this pull request
Jul 3, 2026
* fix(nlpgo): preserve evaluator result envelope so reasoning survives (#4648)
Customer report: the experiments-v3 workbench evaluator result popover
showed only Status (Passed/Failed) with no reasoning/Details, for both
langevals scorers and LLM-as-judge evaluators.
Root cause: the nlpgo Studio executor (runEvaluator) filtered the
evaluator's output map down to the node's DECLARED output names.
experiments-v3 (workflowBuilder) declares evaluator outputs as exactly
[passed, score, label] and never lists the metadata envelope, so the
filter silently dropped `details` (the reasoning) plus `status` and
`cost`. The legacy Python executor surfaced the full result dict
(end_component_event did `outputs = dict(result)`) and never filtered,
so this was a NLP->Go migration regression.
The same filtered map (ns.Outputs) feeds two consumers, both of which
lost the reasoning:
- the experiments-v3 SSE path: resultMapper reads outputs.details to
build the evaluator_result event the workbench popover renders.
- the batch eval reporter: evaluation.go reads ns.Outputs["details"].
Fix: in runEvaluator, always surface the EvaluationResultWithMetadata
envelope (status / details / cost) regardless of declarations, and apply
the declared-names filter only to the VALUE outputs (passed / score /
label). Value outputs stay author-controlled (a workflow that wires only
`passed` still gets only `passed`); the reasoning always reaches the
result panel and batch report.
Surfaced by #4642 only in the sense that it unblocked reaching a rendered
downstream meta-eval result (the boolean-output 400 previously made this
path unreachable); #4642 did not write this bug.
Tests: evaluator_envelope_test.go runs entry -> evaluator end-to-end
against a stub langevals server and asserts details survives a
[passed,score,label] declaration, and that undeclared value fields are
still filtered while the envelope is kept. Reverting the fix fails both
(details=nil). Bound to specs/studio/workflow-evaluator-output-envelope.feature.
* fix(deps): bump @vitest/browser-playwright to >=4.1.7 (closes CRITICAL #1237) (#4569)
@vitest/browser-playwright@4.1.5 pulled a transitive @vitest/browser@4.1.5,
which is < 4.1.6 and still vulnerable to GHSA-2h32-95rg-cppp even after the
direct @vitest/browser bump in #4474. Bumping the provider to ^4.1.7 pulls
@vitest/browser@4.1.7, so every resolved copy is now >= 4.1.6 and the alert closes.
Co-authored-by: Ubuntu <ubuntu@ip-10-0-3-183.eu-central-1.compute.internal>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(datasets): searchable span mapping dropdown + clearer expansion label (#4649)
Two follow-ups now that the dataset-mapping dropdowns offer every project
name from the last 30 days (#4644, #4645):
1. The key dropdown (span / metadata / evaluation / event names) is now a
searchable chakra-react-select instead of a plain native <select>.
These lists can hold hundreds of names, so a type-to-filter search box
makes finding the right one fast. Keeps the "* (any ...)" entry, the
loading + disabled state, and the same onChange semantics.
2. Fix the misleading expansion toggle label. The toggle splits a trace
into one dataset row per span (normalize / expand), but its label read
"One row per all spans", which sounds like the opposite (all spans in
one row). Relabel the all-spans expansion to "span" so it reads "One
row per span". This is display-only: the saved expansion keys and the
apply logic are unchanged, so existing automations are unaffected (the
behaviour was never reverted - it has been one-row-per-span when enabled
for years, and live automations rely on it).
A unit test locks the expansion behaviour (3 spans + expansion -> 3 rows;
no expansion -> 1 row with the spans array) so it cannot silently flip.
* fix(deps): consolidated npm security overrides across the monorepo (closes 18 alerts) (#4650)
Force-patch transitive npm advisories and one direct HIGH bump, all within the
same major so breakage risk stays low. No app migration required.
langwatch: better-auth ^1.6.11 (HIGH #1260); hono <4.12.21 (#1264-1267);
brace-expansion (#673,#1180); file-type (#542); prismjs (#196);
ip-address (#1001); qs (#1202); webpack ^5.104.1 devDep (#247,#248)
typescript-sdk: protobufjs >=8.2.0 (#1191); postcss (#966); qs (#1206)
skills: ip-address (#1003); qs (#1204)
Excludes root esbuild/vite (needs a vite 5->6 major), the PIP/uv cluster,
mcp-server vitest (handled by #4479), and no-fix advisories.
Co-authored-by: Ubuntu <ubuntu@ip-10-0-3-183.eu-central-1.compute.internal>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(deps): close 6 langevals security alerts with minimal stable bumps (no prerelease, no downgrade) (#4654)
fix(deps): close 6 langevals security alerts with minimal stable bumps
Surgical uv lock upgrade (authlib 1.6.12, idna 3.17, pyarrow 23.0.1, pytest 9.0.3,
requests 2.34.2, starlette 1.2.1) closing 6 alerts (1 HIGH, 5 MODERATE). Avoids the
Dependabot uv-group resolution that pinned transformers to the 5.0.0rc3 prerelease
and downgraded arize-phoenix 14.6->9.0. No torch/native churn, no downgrades.
transformers (#831/833) and strawberry-graphql (#1186/1257/1258) deferred: their
only fixes need a torch-heavy major and an arize-phoenix downgrade respectively.
Also fixes a pre-existing ragas RougeScore call (measure_type -> mode) surfaced
during dogfooding.
Co-authored-by: Ubuntu <ubuntu@ip-10-0-3-183.eu-central-1.compute.internal>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(nlp): remove the python langwatch_nlp service, nlpgo is the only engine (#4653)
* refactor(nlpgo): strip the uvicorn child, nlpgo is Go-only
The Python langwatch_nlp service is being removed; nlpgo becomes the sole NLP engine. Remove the uvicorn child manager, the reverse proxy, and all NLPGO_CHILD_* config from the Go service. Any non-/go/* request now returns the go-only 502 unconditionally (no proxy fall-through). Delete the uvicornchild + proxypass adapter packages and their tests; update the lifecycle tests to assert the child-free service set.
Rewrite specs/nlp-go to the Go-only end state: drop the parallel-deployment, feature-flag, and front-door specs, add python-removal.feature, repoint topic-clustering at langevals unconditionally, and update the shared contract. Seed the code-block sandbox-requirements.txt (httpx, requests, pydantic, langwatch); the artifact keeps a minimal python3 only for code blocks.
* refactor(app): route all NLP traffic to nlpgo + langevals, drop the FF
With the Python langwatch_nlp service removed, nlpgo is the only NLP engine and topic clustering lives in langevals. Remove the release_nlp_go_engine_enabled feature flag and every Python-fallback branch it gated: studio execution and the playground proxy always hit nlpgo's /go/* paths; topic clustering always posts to langevals (dropping the langwatch_nlp / TOPIC_CLUSTERING_SERVICE fallback and the fetch-h2 path); execute_optimization always returns 410.
Delete the flag from the registry, the isNlpGoEnabled helper, and the TOPIC_CLUSTERING_SERVICE env. Update the affected unit tests (featureFlag fixtures, post-event routing, topic-clustering staged-fetch).
* build(nlpgo): rewrite the self-hosted image to distroless Go + sandbox python
Replace the python:3.12-slim + uvicorn + litellm + langwatch_nlp image with a distroless/python3 image carrying only the static Go binary plus a curated code-block sandbox (requests, httpx, pydantic, langwatch from sandbox-requirements.txt). 136MB, down from ~1GB+. The Go binary is the only long-lived process; python3.11 is spawned per code-block node and exits.
Verified end-to-end: /healthz 200, /go/version served, non-/go returns the go-only 502, and the sandbox python imports the curated libs (cp311 ABI match against distroless/python3-debian12). Babel's CLDR locale data is pruned to en+root (~30MB saved); each *.dist-info is kept because opentelemetry resolves its runtime context via importlib.metadata entry points.
* refactor(npx-server): delete langwatch_nlp, nlpgo is the only NLP runtime
Delete the Python langwatch_nlp service entirely and make the @langwatch/server npx self-host Go-only. The npx-server already ran nlpgo from the aigateway monobinary in its default 'go' mode; this removes the legacy 'python' (uvicorn) mode: delete startLangwatchNlp, the --nlp CLI flag + parseNlpMode, nlpMode from the runtime contract / env scaffold / cli, and the langwatch_nlp venv + project wiring.
Drop langwatch_nlp from the python feature-parity test roots and the package files list, and update the affected packages/server tests. Keep LANGWATCH_NLP_SERVICE (still the nlpgo base URL), langevals, and the uv predep (langevals needs it).
* deploy(helm): point the nlp deployment at the Go-only image
The langwatch_nlp image is now a distroless Go binary (no litellm/dspy/uvicorn), so the deployment drops the python-tier resources to requests 250m/256Mi, limits 1cpu/1Gi, repoints the probes at the Go endpoints (/healthz liveness, /readyz readiness, /startupz startup), and pins SERVER_ADDR to the service port. LANGEVALS_ENDPOINT is already wired on the app (topic clustering routes there). Updated the example/test overlays + README resource rows.
* chore(nlp): Go-only CI + dev environment, drop the python lambda Dockerfile
Finish the langwatch_nlp removal across CI and local dev. CI: langwatch-nlp-ci.yml drops the python lint/test/integration jobs (the dir is gone) and keeps the Go-image build smoke check (the Go source is covered by go-ci.yaml); strip the langwatch_nlp pyproject/uv.lock path filters from the npx-server workflows and the codeql path filter. Delete Dockerfile.langwatch_nlp.lambda: broken after the dir deletion and unbuilt by monorepo CI (the saas owns its own lambda runtime image). Dev: the all-local-nlp preset runs the Go nlpgo (make service svc=nlpgo) instead of uvicorn; compose services drop NLPGO_CHILD_* env; dev scripts + CLAUDE.md structure updated.
* docs(nlp-go): update the progress log to the Go-only deployment end state
* fix(nlpgo): route scenario adapters + helm smoke through go-only paths
The go-only 502 fallback (this PR) breaks live callers that bypass nlpgoFetch:
scenario code/workflow agents posted to ${nlpServiceUrl}/studio/execute_sync and
the scenario model factory proxied via /proxy/v1, both now 502. The helm
full-stack smoke also still hit the removed /health. Route the adapters through
/go/studio/execute_sync, the model factory through /go/proxy/v1, and the smoke
through /healthz; update the asserting unit tests.
* docs(self-hosting): nlp sizing to the lean go tier; topic clustering scales langevals
The nlpgo service is a lean Go binary, so the chart default and the prod/HA
overlays request far less than the old Python service. Update the sizing &
scaling tables/profiles to match (250m/1 CPU, 256Mi/1Gi), and fix the scaling
guide: topic clustering runs on langevals now, not the NLP service. Regenerate
llms-full.txt. Also trim a backward-looking comparison from the values.yaml
resources comment (describe current state, not history).
* Release docs (auto-release on merge to main) [skip ci]
* refactor(types): make zod the single source of truth, remove ts-to-zod (#4651)
* refactor(types): make zod the single source of truth, remove ts-to-zod
ts-to-zod generated Zod schemas from TypeScript types on every server
start, printing "File not found" / "fallback into z.any()" warnings and
leaving holes the inferred types could not express. Flip the direction:
schemas are defined in Zod and the TypeScript types are inferred with
z.infer.
- tracer, datasets and experiments types.ts are now Zod-first; the
generated *.zod.ts / *.generated.ts schema files are gone and consumers
import the schemas directly from types.ts
- langevals generates evaluators.generated.ts as Zod directly (new
get_field_type_to_zod emitter + unit test); the evaluators.zod.generated
ts-to-zod step is removed and its consumers fold into the one file
- evaluatorTypesSchema and reservedTraceMetadataMapping no longer collapse
to z.any()
- typescript-sdk copies the Zod-first modules verbatim (no ts-to-zod);
records use the two-arg form so they hold under Zod 4
- drop ts-to-zod from both package.jsons, delete both configs and
generate-zod-types.sh, and remove the types:zod:generate step from
start:prepare:files and compose.dev.yml
specs/dependencies/zod-first-schema-source-of-truth.feature pins the new
invariant; a no-mock test exercises the real collector and evaluator
schemas.
* fix(mcp-server): resolve zod for the now-Zod-first evaluator catalog
The langevals evaluator catalog the mcp-server bundles is Zod-first now and
imports `zod`, but it lives outside any node_modules tree so esbuild could
not resolve the import. Add the package's own node_modules to esbuild
nodePaths so zod still bundles into the standalone output.
* test(evaluators): real-Chromium render of the Zod-first catalog
Renders the unmocked evaluatorsSchema + AVAILABLE_EVALUATORS in chromium and
asserts the settings fields and defaults the forms build from, guarding the
schema-to-form path after the ts-to-zod removal.
* fix(deps): regenerate langwatch lockfile after dropping ts-to-zod
CI installs with --frozen-lockfile; langwatch/pnpm-lock.yaml still pinned
ts-to-zod (and its dependency subtree) after it was removed from
package.json, so the install failed before any check ran.
* fix(langevals): preserve explicit falsy defaults in emitted Zod; review nits
- emit .default(...) for every explicit default except a bare None, so 0/""/
[]/{} keep their runtime defaulting (was dropped as 'falsy'); add coverage
- drop unused import in the generator test
- use gpt-5-mini in the span test fixture per repo guideline
* fix(mcp-server): let tsc resolve zod for the out-of-tree evaluator catalog
The Zod-first catalog import made tsc fail with TS2307 (cannot find 'zod')
because the langevals file sits outside this package's node_modules. Map
`zod` to this package's own copy via tsconfig paths, mirroring the esbuild
nodePaths fix.
* fix(mcp-server): use paths without baseUrl so tsgo resolves zod
tsgo removed the baseUrl option (TS5102); paths resolve relative to the
tsconfig dir, which is what we want to point zod at this package's own copy.
* fix(evaluators,tracer): correct field classification and preserve pi-ai content
Two latent bugs surfaced during the ts-to-zod removal review, now fixed:
- generate_evaluators_ts.py read pydantic FieldInfo.is_required as an
attribute instead of calling it, so the "field has a default, therefore
optional" clause never fired. Calling is_required() restores the author's
intent and aligns the catalog with langevals' real contract: exact_match
and llm_answer_match entry fields that carry defaults are now optional
rather than required.
- chatRichContentSchema had two separate "text" union branches, so the
union's first-match-wins parsing stripped the pi-ai `content` key off
text parts. Merging them into a single text branch keeps both `text`
and `content`.
Adds catalog and ingest regression tests plus BDD scenarios, and repoints
the mapping-validation fixtures at an evaluator that still has required
fields so they keep exercising the required/optional distinction.
* test(evaluators): show entry-field required/optional classification in catalog browser QA
Renders each showcase evaluator's requiredFields/optionalFields in the
real-Chromium catalog and asserts exact_match's output/expected_output
are optional, giving the field-classification fix a visual QA artifact.
* fix(eval-v3): prompt-span parity + traces-v2 drawer opt-in routing (#4657)
* fix(experiments-v3): forward prompt identity so eval-v3 traces carry prompt spans
Evaluations v3 runs each row's target prompt through nlpgo via
execute_component, but the per-cell workflow builder forwarded only the
prompt's CONTENT (llm / instructions / messages) and dropped its IDENTITY.
The nlpgo engine emits the PromptApiService.get + Prompt.compile spans only
when the signature node carries data.configId, so eval-v3 LLM spans had no
prompt ancestry: the trace drawer's "Open in Prompts" resume was hidden and
you could not replay the same prompt + version + variables in the
playground. The engine emission was already complete (prompt_spans_emit_test
has zero skips); the gap was purely this app-side forwarding.
- buildSignatureNodeFromPrompt: forward configId / handle / versionMetadata
from the saved VersionedPrompt (new buildPromptIdentity helper, shape
mirrors signatureComponentSchema and the Go dsl.Component fields).
- buildSignatureNodeFromLocalConfig: forward the BASE prompt identity plus
promptDraft=true for inline-edited targets (the "(unsaved edits)" case,
wire format per specs/nlp-go/prompt-spans-unsaved-version.feature, locked
on the channel).
- nlpgo: un-skip the two core eval-v3 integration tests (per-row get+compile
emission + the no-configId negative); add an Origin field to the shared
prompt-spans test harness so eval-v3's origin="evaluation" dispatch is
exercised.
- workflowBuilder.test.ts: regression tests for both forwarding paths.
- specs/nlp-go/prompt-spans-eval-v3.feature: point the stale frontend
references at the real server path (workflowBuilder + orchestrator).
Issue 1 ("Evaluation V3 - Row N" repeated as a separate trace root per
component) is NOT a regression: the orchestrator dispatches target + each
evaluator as separate execute_component calls and nlpgo names each root
after the workflow, mirroring Python's optional_langwatch_trace(
name=workflow.name). Nesting them under one shared per-row parent span is a
follow-up enhancement, not a fix.
* fix(traces-v2): honor the drawer opt-in from every trace entry point
Trace-view buttons outside the traces table (eval-v3 results, workflow run
panels, feedback, command bar, annotations) called openDrawer("traceDetails")
directly, bypassing the v2 opt-in routing that lived inside
useTraceDetailsDrawer, so an operator who opted into the new explorer still
got the legacy drawer there.
Route at the single openDrawer chokepoint instead: a traceDetails open with a
traceId is rewritten to traceV2Details when the device opted in, fixing all
entry points at once (mirrors the central EXTERNAL-user restriction in
CurrentDrawer). useTraceDetailsDrawer becomes a thin delegate; the pure
routing helper is extracted to traceDrawerV2Routing.ts and unit-tested, with
an openDrawer integration test and a real ExecutionOutputPanel component test.
* test(experiments-v3): use gpt-5-mini in eval-v3 workflowBuilder fixtures
CodeRabbit flagged the new prompt-identity fixtures using openai/gpt-4o-mini;
switch them to gpt-5-mini per the project test-model rule. No assertion
references the model value, so the forwarding tests are unaffected.
* fix(traces-v2): guard non-object drawer query param in useUpdateDrawerParams
A malformed query like `?drawer=foo` makes qs parse `drawer` as a string; casting it to a record and mutating it in the update loop would throw at runtime. Guard the shape and default to an empty record. (CodeRabbit review on #4657.)
* test(traces-v2): user-perspective spec wording + given/when test nesting
Address CodeRabbit review nitpicks. Rephrase the 'every entry point' scenario to user-observable steps (open from any screen, the new drawer opens) instead of 'request is routed', and restructure the helper + openDrawer suites into given/when BDD nesting per the project convention. No behavior or assertion changes; the 7/7 @scenario bindings are preserved.
* refactor(experiments-v3): named params for signature builders + given/when test nesting
Address CodeRabbit nitpicks on the eval-v3 prompt-span work:
- buildSignatureNodeFromPrompt / buildSignatureNodeFromLocalConfig take a
single named-params object instead of 5-6 positional args (the latter grew a
6th positional basePrompt in this PR), per the project named-parameters rule.
Updated both call sites in buildTargetNode and the unit + integration tests.
- Restructure the two new workflowBuilder test suites into nested
describe("given ...") / describe("when ...") blocks per the BDD test rule.
* test(experiments-v3): use gpt-5-mini in workflowBuilder integration fixtures
CodeRabbit flagged the in-diff integration test still using openai/gpt-4o(-mini)
in its inline prompt-config fixtures. Sweep all four (the two fixtures and their
paired output assertions) to openai/gpt-5-mini per the test-model rule. The
max_tokens assertions are driven by the fixtures' explicit maxTokens, so the
model swap is inert (integration suite 26/26 green). Pre-existing gpt-4o-mini in
workflowBuilder.ts:113 (production default) and its coupled unit assertions are
intentionally left untouched.
* feat(governance): departments, tool catalog, and CLI path selection (#4646)
* refactor(governance): rename cost centers to departments (incl. DB)
Renames the org-chart spend-attribution concept from "cost center" to
"department" end to end: Prisma model CostCenter to Department, the costCenterId
columns + indexes on OrganizationUser/Team/Project to departmentId (migration
20260606120000), the service / repository / tRPC router (api.costCenters to
api.departments), the settings page + route (with an old-path redirect), the
picker / edit-drawer / column-hook components, the spend-by-department
activity-monitor card, docs, and the BDD spec.
Kept stable on purpose: the SCIM 2.0 external attribute stays "costCenter" (IdPs
send it; only the internal DepartmentService delegation was renamed); the
attribution sentinel value stays "unassigned"; no ClickHouse / data-plane keys
were touched (attribution is Postgres-only at read time).
typecheck clean; department service/attribution/spend/assignment + SCIM tests
green; feature-parity bound.
* chore: gitignore .claude agent scratch + dogfood-evidence dirs
* feat(cli): prompt for gateway vs direct-OTLP path on `langwatch <tool>`
The wrapper silently routed every run through the gateway virtual key even when
the org policy allowed both paths. It now resolves the path with a clear
precedence: an explicit --lw-path=gateway|otlp flag (or LANGWATCH_PATH env), a
remembered per-tool answer (cfg.tool_mode), exactly one allowed path, then an
interactive select when both are allowed on a TTY. The choice is remembered so
later runs do not re-ask, and non-TTY/CI defaults to the gateway.
When the user picks direct OTLP, the wrapper mints/reuses the personal ingest
key and offers to persist the OTEL exports to the shell rc (y/n/N, N = never
ask). On a mint/network failure it falls back to the gateway with a clear note.
Arg pass-through: only the wrapper-only --lw-path flag is stripped; every other
arg (--dangerously-skip-permissions, quoted positionals, ...) is forwarded to
the real tool verbatim and in order.
40 unit tests (path precedence, arg passthrough, shell-rc block idempotency);
typecheck clean; CLI suite green; dist rebuilt.
* feat(governance): overhaul AI Tool Catalog (department scope, overflow menu, CLI paths in tile)
Reworks the AI Tool Catalog admin surface end to end:
- Tile "Visible to" is now WHOLE ORGANIZATION or a set of DEPARTMENTS (people
groups), replacing the flat workspace/person checkbox list and the team axis.
New AiToolEntryDepartment join + migration; the /me filter shows a tile when
it is org-wide or the member's department matches. Legacy team-scoped rows
degrade to org-wide.
- The shared scope picker + badge gain a DEPARTMENT cut. ScopeChipPicker is
generic and defaults to the model-provider triad, so resource consumers
(model providers, VKs, budgets, routing, default models, retention) stay
narrow via ScopeTriadEntry; only opt-in surfaces see departments.
ProviderScopeChips renders a cyan "Department: X" chip.
- Row actions move to the standard vertical 3-dot overflow menu (Edit /
Enable-Disable / Delete via the existing archive endpoint); the tile drawer
drops its redundant Cancel button and shows the scope chip instead of a plain
"Org-wide" label.
- The per-tool gateway / direct-OTLP toggles fold into the coding-assistant
tile config (allowVk / allowOtelDirect; cursor forces direct-OTLP off). The
separate "CLI Paths" tab + ToolPathPolicyEditor are removed and cliBootstrap
derives the login toolPolicies map from tiles over the hardcoded defaults.
- AiToolEntryDepartment added to the multitenancy exempt list (org-scoped via
its entry, like AiToolEntryTeam) so department-tile writes pass the guard.
- Two dev/docs best-practices: scope-selector-and-badges + row-actions-overflow-menu.
typecheck clean; aiTools integration (20) + collapse/ProviderScopeChips/model-provider scope tests green.
* feat(settings): organization switcher on org-scoped header
On org-scoped routes (/settings/*, /governance/*) the header showed a static
org-name chip, so a user in more than one organization could not switch orgs
from there. The chip is now a menu listing the user's organizations (active one
checked); selecting another writes selectedOrganizationId and navigates to
/settings, the always-valid parent of every org-scoped route, so the page
re-resolves under the new org. Single-org users keep the original static chip
(no menu, no chevron). The "Organization-scoped, not tied to a project" pill is
unchanged.
* fix(cli): satisfy eslint in shell-rc ingestion-offer test
no-empty-function on the readline close mock + inline import() type
annotation; switch to a namespace type import and a returning no-op.
* feat(governance): drop untested cursor from the AI tools starter pack
Cursor's `langwatch cursor` wrapper exists but its telemetry/governance
path has never been validated, so we no longer seed a Cursor tile a fresh
org cannot stand behind (tracked for re-add in #4647). Starter pack is now
8 tiles (4 coding assistants + 4 model providers); the merge-dedup test
swaps its custom-icon fixture to Codex to keep that path covered.
* fix(cli): quiet dotenv, allow Ctrl+C during login, clearer shell-rc prompt
- dotenv config({ quiet: true }) so the 'injecting env ... tip' promo line
stops printing on every CLI run.
- the device-flow login spinner used ora's default discardStdin, which put
stdin in raw mode and swallowed Ctrl+C, making the 'waiting for approval'
wait unkillable; keep stdin cooked + handle SIGINT so it always cancels.
- shell-rc persist prompt reworded to one clear line and dropped the
redundant 'open a new shell / source' follow-up.
* feat(governance): give ingestion-only keys an ik-lw- prefix
Auto-minted ingest keys (keyType=ingest, traces:create only) now carry an
ik-lw- prefix so they're identifiable at a glance vs full-access sk-lw- keys.
Same ApiKey primitive + HMAC scheme; the prefix is cosmetic + for routing, the
DB lookup is by lookupId, so getTokenType/splitApiKeyToken just learn the new
prefix and resolution is unchanged.
* fix(cli): don't re-offer shell-rc persist when the block is already on disk
The ingestion shell-rc offer only checked process.env for the OTLP endpoint, so
it re-asked when the user had already installed the block but hadn't sourced the
rc in this shell yet. Also check the rc file for the `# >>> langwatch begin >>>`
marker and stay quiet when present.
* feat(me): show self-hosted `langwatch login --endpoint` step on the tile
On a self-hosted instance the CLI defaults to app.langwatch.ai, so the coding-
assistant tile now shows a 'point the CLI at this instance and sign in' step
(`langwatch login --endpoint <BASE_HOST>`) above the `langwatch <tool>` command
when not IS_SAAS. SaaS is unchanged.
* revert(governance): drop redundant Company-wide push ingestion panel
The company-wide push ingestion block (added in #4544) duplicated the existing
Push (OTLP / webhooks) ingestion source: that source already routes external
OTLP into the org's hidden Governance Project (ingestionRoutes via
ensureHiddenGovernanceProject), which is the intended Copilot Studio path. The
panel just minted a one-off org-level ingest key into the same project.
Removes CompanyWideIngestionPanel, the companyWide{List,Install,Rotate} tRPC
procedures, the ensureForOrganizationGovernanceProject /
listForOrganizationGovernanceProject service methods, their tests and spec
scenarios. No migration: the governance project is a lazy-created Project row
and the ingestSourceType columns are shared with the personal ingest keys we
keep. ensureHiddenGovernanceProject stays - the push sources use it.
* fix(me): cleaner CLI command rows (non-copyable $ prompt, drop Self-hosted?)
The $ shell prompt is a separate, non-selectable, lighter element that
top-aligns when the command wraps, so copying the command by hand never picks
up the '$ ' (the prior ::before lost the space and floated mid-wrap). Also
drops the redundant 'Self-hosted?' lead-in - the step only renders in the
!IS_SAAS branch, so the user is self-hosted by construction.
* fix(presence): Sharing presence toggle was a dead click (onSelect -> onClick)
The avatar-menu Sharing presence row used onSelect, which on a Chakra v3
Menu.Item is the DOM text-selection event, not the click handler - so clicking
did nothing (no toggle, no visual feedback). Use onClick like the rest of the
menu items; the icon/dot/label already re-render reactively from the store.
* fix(dev): pre-bundle Shiki in optimizeDeps so the trace drawer stops hanging
Dev-only (optimizeDeps never touches the prod build). Vite discovered Shiki's
Oniguruma WASM engine lazily on the first /traces navigation and re-optimized
mid-session, leaving the in-flight .vite/deps/wasm-*.js (onig.wasm, ~620KB)
request the span highlighter awaits pending - the trace drawer sat on 'loading
spans'. Pre-bundling the Shiki ecosystem at dev-server start removes the
mid-session re-optimization. Prod still bundles Shiki via the manualChunks rule.
* feat(cli): run the wrapped tool through the user's shell so aliases are honored
langwatch <tool> spawned the bare PATH binary, so a user's
alias claude='claude --dangerously-skip-permissions' (or a shell function) was
ignored. On zsh/bash it now routes through $SHELL -i -c '<tool> "$@"' so
aliases/functions resolve like a normal terminal invocation; the wrapper's env
(mode vars + clears) is re-applied after the rc sources so a user's rc can't
clobber the gateway/OTLP wiring, and args ride positional params. Fish and
Windows keep the direct spawn.
* refactor(cli): rename --lw-path to --tool-mode (aligns with tool_mode config)
The path-override flag/env read oddly next to the config key it pins
(cfg.tool_mode[tool]). Rename --lw-path -> --tool-mode and LANGWATCH_PATH ->
LANGWATCH_TOOL_MODE (+ parseToolModeFlag/ParsedToolMode/TOOL_MODE_FLAG and the
'Override with ...' tip). Values unchanged (gateway|otlp). Pre-launch flag, no
back-compat alias needed. Spec scenarios + @scenario tags renamed in lockstep.
* fix(workspace-switcher): preserve sub-route on project switch + org-scope selector
Switching projects from a project-anchored route (e.g. /[project]/traces) now
keeps the same view for the target project instead of bouncing home; only
org-level routes with no per-project equivalent land on the project home. The
route resolver is extracted into a shared buildProjectSwitchHref so the
WorkspaceSwitcher (via useWorkspaceData) and the legacy ProjectSelector no
longer carry diverging copies; that drift was the regression.
Org-scoped routes (/settings/*, /governance) now render the WorkspaceSwitcher
with the org as the current chip so the user can jump back into any project or
their personal workspace; multi-org users keep an in-place org switch. Drops
the redundant "Organization-scoped, not tied to a project" badge.
* feat(portal): governance getting-started hero for the empty /me portal
Replaces the bordered empty-state box + CLI install card with a MeshGradient
glass-card hero (matching the home announcements) when a catalog admin opens an
empty AI tools portal, pointing straight at the tool catalog where the starter
pack lives. Members with an empty portal get a plain "your admin hasn't added
tools yet" note. Drops the install-the-CLI affordance from this surface
entirely; InstallCliCard still ships on the coding-assistant tile, governance
settings, and the sessions page.
* feat(api-keys): API keys table above ingestion keys + name the legacy project key
Extracts the ingestion-keys table into its own IngestionKeysSection component
rendered below the regular API keys table (was above), so the primary keys lead
the page. The extraction keeps the API keys section in place rather than moving
the block, so the diff stays focused.
Also: the legacy per-project service key row now names its project via the same
scope chip as the user-scoped rows (was a bare "Project" badge), and the
ingestion-key secret preview shows the ik-lw- prefix the CLI actually mints
(was sk-lw-).
* feat(ingestion-keys): attribute each key to the CLI device session that minted it
Snapshots the device label from the CLI device flow's client_info at mint time
onto a new ApiKey.createdByDeviceLabel column, threaded through the ingestion-key
service and api-key service/repository. The API-keys settings page shows it as a
"Created from" column on the ingestion-keys table (falling back to "Unknown
device" for keys minted before device metadata existed). Denormalized so the
org-wide page can attribute keys without resolving another user's Redis sessions.
* fix(claude-otlp): parse the request body into a real conversation, never the raw blob
The claude-code log-to-span converter set gen_ai.prompt to the latest user turn
and, when claude truncated the request body inline (its own ~60KB cap makes the
JSON unparseable), fell back to the raw body string. The GenAIExtractor then
wrapped that string as a single user message, so the trace detail rendered
`{"model":...,"messages":[...]}` with claude's "[TRUNCATED ...]" marker as the
input instead of a conversation.
Now the converter parses the api_request_body into structured
gen_ai.input.messages: the system prompt plus every turn as { role, content },
each message's content flattened to text (text + tool_result + a compact
[tool_use: name] marker; thinking/images dropped). When the body is truncated it
falls back to the clean co-batched user_prompt text as the single latest turn,
and never to the raw JSON blob. Model/tokens/cost were already lifted onto the
span and are unchanged.
* fix(cli-wrapper): restore actionable tool-not-found error under alias-shell spawn
The #180 alias-shell invocation (spawn the login shell so aliases/functions
resolve) meant a missing tool no longer triggered Node's ENOENT handler: the
shell itself launches fine and only fails internally with a bare
`command not found`, swallowing our message. Add a `command -v` pre-flight
guard inside the same login shell so a missing tool surfaces the actionable
"<tool> not found in PATH - install it first" message and exits 127, exactly
as the direct-spawn ENOENT branch already does. The message is now defined
once and shared by both paths.
* refactor(tenancy): derive projectId-guard org exemptions from the org registry
The projectId guard hand-listed ~25 org-scoped models in EXEMPT_MODELS that the
org guard already classifies (ORG_SCOPED_MODELS + ORG_TENANCY_EXEMPT). That
duplication is a rot vector: a phantom 'GatewayAuditLog' (no such model) had
already crept in, and a new org-scoped model had to be hand-added in two places.
Make the org guard the single source of truth: export ORG_BEARING_MODEL_NAMES
(the union its DMMF partition test proves equals exactly the org-bearing set) and
derive the projectId guard's org exemptions from it, minus SCOPED_MODELS so those
keep their stricter row-id/scope/parent-FK check. What stays hand-listed shrinks
to honestly-named buckets: GLOBAL_MODELS (auth tables, top-level entities,
cluster-wide config), RELATIONAL_PARENT_SCOPED (join/membership tables), and
LICENSE_COUNTED_PROJECT_MODELS (project-scoped but org-rollup-read).
Stricter, not looser: the two AiToolEntry join tables move from blanket-exempt
into SCOPED_MODELS (validated by parent entryId), and a DMMF partition test now
proves every projectId-less model is classified into exactly one regime, no
org-bearing model is hand-listed, and no phantom names survive. Drops the
GatewayAuditLog phantom. Behaviour-preserving for all 40 real exempt models.
* fix(traces-v2): dedupe the discover SSE subscription so the drawer stops starving
useTraceFacets opened a tracesV2.onDiscoverUpdate SSE subscription, but it is
consumed by three sidebar components (SearchBar, FilterSidebar,
TokenValuePicker). tRPC subscriptions are not deduplicated across hook
instances the way queries are, so each consumer opened its own EventSource:
three identical onDiscoverUpdate connections. With traces.onTraceUpdate and
presence.onPresenceUpdate that is five persistent SSE connections, and on
HTTP/1.1 (dev) the six-per-origin pool leaves a single free slot, so the burst
of queries fired when a span/drawer opens sits pending behind the held
connections (a devtools resend works because it is a fresh connection).
Hoist the subscription into useTraceFreshness, the page-level coordinator
already mounted once in TracesPage that owns trace-event-to-cache
invalidation. One subscription invalidates the shared discover query, which
refreshes every consumer, so behaviour is unchanged but the connection count
drops from five to three. Reproduced the five-connection state via Playwright
before the fix.
* fix(claude-otlp): one trace per turn, not per session
Claude Code emits its events with no trace context, so the receiver synthesizes
ids. trace_id was sha256(session.id) - constant for the whole session, so every
user turn collapsed onto one ever-growing trace. Key it on the turn instead:
sha256(session.id || ':' || prompt.id). Every event of a single turn (the
user_prompt, the api_request/response model calls its agentic tool-loop drives,
and the tool events) shares one prompt.id, so they stay on one trace, while a
new turn (new prompt.id) starts a new trace. gen_ai.conversation.id = session.id
still groups the turns into one conversation thread. Pre-prompt session-setup
events (no prompt.id) keep falling back to the session-level id.
Validated against a real 2-turn claude-code 2.1.x session captured via an OTLP
intercept: turn 1 ("run three bash commands one at a time") drove 6 api_requests
across a 4-call tool-loop, ALL sharing prompt.id 47fcab35; turn 2 got a distinct
5fc69a28. request_id is per-API-call (8 distinct) and would have shattered turn
1 into 6 traces, so prompt.id is the correct turn key.
* fix(traces-v2): stop the drawer-mode sync from clobbering drawer.open on reopen
Reopening a trace drawer when the persisted view-mode differed from the URL
left the URL as just `?drawer.mode=trace` - no drawer.open/traceId - so a
refresh lost the drawer. openDrawer's `?drawer.open=traceV2Details&drawer.traceId=…`
push is an async shallow navigation; the store->URL effect fired mid-transition
and wrote drawer.mode off a stale asPath that had no drawer.open/traceId yet,
clobbering them. Gate the sync on drawer.open being present in the URL: the
effect re-fires once the open lands, and updateDrawerParams then reads an asPath
that carries drawer.open + drawer.traceId and preserves them.
* feat(claude-otlp): capture every attribute claude emits + map effort/request_id
The converter only lifted the canonical token/model/cost subset, dropping the
rest of what claude-code sends on each api_request (effort, speed, query_source,
duration_ms, terminal.type, user.id, …). Capture them all:
- effort -> gen_ai.request.reasoning_effort (new ATTR key; Anthropic's reasoning
knob, the analogue of OpenAI's reasoning_effort)
- request_id -> gen_ai.response.id (the Anthropic message/request id)
- everything else -> claude_code.<key> verbatim, so nothing is silently lost and
new attributes claude adds are captured automatically.
A CLAUDE_HANDLED_ATTRS set keeps the canonical keys + raw body payloads (already
carried as gen_ai.input/output.messages) from being double-copied. Applied to
both the collapsed-triplet and orphan span paths. Attribute shapes taken from a
live claude-code 2.1.x api_request capture.
* fix(traces-v2): sum cache/reasoning tokens + order models by last use
The trace-summary header popover was missing "Cache write" entirely and
showed an empty "Reasoning" row, and the Model pill surfaced the
title-generation haiku call instead of the opus turn it belonged to.
3.1 Cache + reasoning: the per-span gen_ai.usage.cache_* counts never
reach the trace attribute map (the accumulation allowlist only carries
identity/metadata keys), so the popover had nothing to read. Fold the
per-span cache read / cache write (creation) / reasoning counts into
trace-level sums under reserved keys; the drawer reads those first and
falls back to the raw keys for traces folded earlier. For Anthropic the
cache write lands only on the first span and the last span carries none,
so a latest-wins read would always drop it; the sum keeps it visible.
Cache/reasoning rows now render only when measured, so the permanent
empty "Reasoning —" row (Anthropic emits no reasoning tokens) is gone.
3.2 Models: the fold sorted models alphabetically, so claude-haiku-4-5
beat claude-opus-4-8 and every models[0] consumer showed the utility
call. Order models most-recently-used first so models[0] is the
conversational model; the drawer adds a "+N" hover badge (reusing the
table's ExtraModelsBadge) and a plural "Models" label, matching the
/traces list.
* feat(me): rebuild Recent activity on the /traces v2 table
The /me Recent activity card was a bespoke time/tool/summary/cost list
fed by a gateway-ledger-only query, so Path B OTLP traces (Claude Code
et al., which land directly in the personal project tenant) never
showed up. Rebuild it on the real /traces v2 table cells against the
personal project, showing only the columns that fit the card: time,
trace (summary), duration, cost, tokens. Clicking a row deep-links into
the full trace explorer with the detail drawer open on that trace.
- New PersonalRecentTracesTable reuses the /traces cell renderers
(Time/Trace/Duration/Cost/Tokens) so a personal trace reads exactly
as it does in the explorer; row click pushes
/<slug>/traces?drawer.open=traceV2Details&drawer.traceId=...&drawer.t=...
- Extract the list-payload -> TraceListItem mapping into a shared
mapTraceListPayload so the full list hook and this card map rows
identically (covered by a unit test).
- usePersonalContext exposes the personal project id + slug and drops
the old recentActivity shape.
- Retire the now-dead PersonalUsageService.recentActivity (plus its
ingestion-principal helper and type) and stop fetching it in
personalUsage; the card reads the personal tenant directly.
* fix(me): request page 1 (1-based) for the recent-traces list
The tracesV2.list endpoint paginates 1-based (page.min(1)); passing page 0
failed input validation, so the /me recent-activity table silently rendered
its empty state even when the personal project had traces. Caught dogfooding
the table against a real personal project.
* fix(claude-otlp): surface tool calls + drop cross-batch orphan bodies
Three claude-code Path B trace-quality fixes reported while dogfooding a
tool-using turn (`hi` -> `echo "test otlp"` -> `did it work`):
1. Tool calls were invisible. claude emits tool_decision / tool_result as
separate OTLP log events that stayed on the log path, so the Bash / Edit /
Read invocations an agent makes never appeared in the waterfall. They are
now trapped and converted into `tool` spans (one per tool_use_id) via
convertClaudeCodeToolLogsToSpans. The command lands on
gen_ai.tool.call.arguments, never langwatch.input, so a parentless tool span
cannot hijack the trace's headline input.
2. Duplicate / out-of-order model spans. claude logs api_request_body at call
START and the api_request anchor at call END, so for any call longer than
the OTLP export interval (every tool-using turn) the body lands in a
different export batch than its anchor. With no request_id on the body it
cannot be re-paired, and the old code emitted it as its own input-only span:
a duplicate of the model call that also sorted before the anchor in the
waterfall ("output earlier than input"). Orphan bodies are now dropped; the
turn input is already on the trace via the co-batched user_prompt event, and
the tool spans show what the call did. Anchor + response co-batch (both at
call END), so the output stays joined.
3. Mystery empty spans. Utility calls (generate_session_title,
prompt_suggestion) were named by model, so a turn showed extra
"claude-opus-4-8" spans carrying no conversation. They are now named by
query_source so the waterfall reads clearly.
* fix(traces-v2): latch the latest conversational output for claude turns
The trace summary showed input but no output for a claude-code tool-using turn
even though a middle span carried the real reply ("Done - output was test
otlp."). The fold's output accumulation treats a parentless span as the trace
root and lets any root override, so among the many parentless spans a
claude_code turn synthesizes (one per model call) the LAST-FOLDED root won
rather than the latest-finishing one. The real reply sits on a middle call,
with utility calls (prompt_suggestion) finishing after it, so the winner
depended on arrival order.
shouldOverrideOutput now keeps the latest-finishing reply among roots, so the
trace output is deterministic by end time instead of fold order. For a
conventional single-root trace this is a no-op (there is only ever one root); a
root still beats a non-root child that set the output.
* fix(governance): address CodeRabbit review quick-wins
- department.repository updateName: add archivedAt: null so an archived
department can't be renamed (the service reads back through getById, which
enforces archivedAt: null, and would otherwise return null after a successful
update count).
- ingestion-source-detail: stop advertising OTEL_RESOURCE_ATTRIBUTES=department=…
as a working attribution input. Department attribution is resolved from the
project's assignment at read time, not from an OTEL resource attribute, so the
env var had no effect. Keep team.id and note where department comes from.
- DashboardLayout: drop the unused Check import.
- departmentAssignment.integration test: hoist the dynamic imports to top-level
statements (vitest hoists the vi.mock calls above them, so the mocks still
apply). Matches the repo's no-inline-import rule.
* fix(governance): address review round 2 (path-policy visibility, scope quick-picks, shell-rc)
Three review-thread fixes on PR #4646:
- aiToolEntry.resolveToolPolicyOverrides now resolves the coding-assistant
tiles the calling user can actually see (department visibility plus
dept-shadows-org precedence) via a new shared resolveVisibleTilesForUser
helper that listForUser also uses; cliBootstrap threads userId through. A
department-scoped CLI path toggle no longer leaks into the bootstrap of
members who cannot see that tile, and the last visibility reader stops
depending on the lossy legacy scope/scopeId mirror. (P1 plus CR #9 and
schema #4)
- ScopeChipPicker quick-picks honour allowedScopeTypes: the Team / Project
chips are no longer offered when the caller restricts the picker to
ORGANIZATION / DEPARTMENT. (CR #6)
- rcHasLangwatchBlock is export-set aware: a stale or different langwatch
block no longer suppresses installing the current OTLP exports, so a plain
`<tool>` run stays captured for Path B. (CR #7)
* fix(governance): portal admin empty-state, CLI path policy, drawer back-button
Dogfood follow-ups on PR #4646:
- /me AI tools portal: a fresh-org admin saw the member "your admin hasn't
added any tools" note instead of the getting-started banner, because
hasPermission routed aiTools:manage to team-level checking. aiTools lives in
ORGANIZATION_ROLE_PERMISSIONS, so org-route it (extracted a testable
isOrgScopedPermission helper + unit test).
- CodingAssistantTile: the non-selectable `$` shell prompt is now lighter
(fg.subtle) and nudged 4px up to sit on the command baseline.
- CLI path-choice copy: "usage billed to the gateway" -> "usage billed per
token".
- CLI path-choice now re-checks the org policy at run time (best-effort
getCliBootstrap) when there is no remembered choice, so a gateway path the
admin disabled after login is honored without a re-login.
- CLI notices: replace the flat `langwatch:` prefix with a styled `✦ langwatch`
brand tag (chalk auto-strips color off a TTY).
- traces-v2 drawer: write view-state params (mode/viz/span/pinned) with history
replace instead of push. Pushing per pane switch created a back-button trap
where Back landed on the drawer URL minus drawer.mode and the sync effect
immediately re-added it.
* feat(traces-v2): recent-activity I/O rows, tool span input, reasoning chip, cache in token total
Dogfood follow-ups on PR #4646:
- /me Recent Activity now mirrors the /traces explorer: adds the ORIGIN column
and the per-row Input/Output summary preview, reusing the explorer's real
OriginCell + IOPreview so the two tables read the same. (TraceLensBody itself
is store-coupled to the /traces page, so a compact embed reuses the cells +
preview rather than the whole shell.)
- claude_code tool spans: surface the tool call on langwatch.input (reads like
an instrumented function call) instead of the non-standard
gen_ai.tool.call.arguments. The trace-IO fold now skips span_type=tool, so a
parentless tool span no longer hijacks the trace's headline input. Only the
real OTel attrs gen_ai.tool.name / gen_ai.tool.call.id are mapped; the rest
stay verbatim under claude_code.*. There is no output to show - claude
reports only tool_result_size_bytes, never the tool's stdout.
- Drawer header: add a Reasoning chip (when reasoning tokens are present) next
to Tokens / Models, and make the Tokens popover Total include cache read +
cache write. Anthropic reports input_tokens as the NON-cached portion, so
cache tokens are additive; a bare input+output Total sat below the cache
rows and read as wrong.
* feat(traces-v2): replace ingest_key trace origin with coding_agent / ai_tool
The ingestion-key provenance stamper hardcoded langwatch.origin to the
credential-shaped value "ingest_key", which reads oddly as a trace origin
(an ingest key is a credential, and the same key class can carry an app's
own tracing). Derive the origin from the key's ingestSourceType instead:
a CLI coding assistant (claude_code / codex / gemini / opencode / cursor)
becomes coding_agent, every other ingest source (claude_cowork, generic
OTLP push, templates) becomes ai_tool.
- ingestKeyProvenance.utils: originForIngestSourceType() + the two origin
constants; the stamper derives origin per source.
- governanceContentStrip: GOVERNED_ORIGINS now {gateway, coding_agent,
ai_tool}, importing the constants instead of a duplicated literal.
- traces-v2 origin labels / palettes / color map / autocomplete + the
TraceListItem origin union learn the two new values.
- specs + ingestion-templates doc reflect the derived provenance value.
The credential class still surfaces as "ingest_key" on /me/sessions; that
is the credential kind, distinct from the trace origin.
* feat(traces-v2): split trace cost into billed vs non-billed (bundled plans)
LangWatch bills per captured event, never per token, but a customer's own
LLM spend depends on how they pay the provider: gateway / virtual-key usage
is billed per token, while a coding assistant on a bundled subscription
(e.g. Claude Max) is not. Today the platform shows the full list-price token
cost everywhere, so a bundled coding session reads as enormous real spend it
mostly is not.
- Admin per-tool flag: coding_assistant tiles get a "Bundled subscription"
toggle (config.bundledPlan, default true). It only governs the direct-OTLP
ingest path; gateway usage is always billed and ignores it.
- Receiver stamp: for ingest-key traces the OTLP receiver resolves the tool's
bundledPlan (cached per org+source) and stamps langwatch.cost.non_billable.
- Trace summary: totalCost stays the grand list-price total; a derived
nonBilledCost holds the bundled (theoretical) portion, so billed =
totalCost - nonBilledCost. Surfaced on the v2 header + list item (the list
CH read projects the marker like langwatch.origin).
- UI: the drawer Cost pill shows the billed amount with a Billed / Non-billed
/ Theoretical-total popover; the traces-list cost cell shows billed with a
hover breakdown. A bundled session no longer reads as real spend.
Analytics billed/theoretical series is a follow-up on this same marker.
* feat(analytics): add billed vs non-billed (theoretical) cost metrics
Custom analytics now exposes two cost series alongside Total Cost, split on
the receiver-stamped langwatch.cost.non_billable marker:
- Billed Cost: list-price cost minus bundled traces (real per-token spend).
- Non-billed (theoretical) Cost: the bundled-subscription portion.
Both are derived in the trace_summaries aggregation (sumIf-style if() over
the existing TotalCost column + Attributes marker), so no new ClickHouse
column or migration is needed. Total Cost stays the grand list-price total
(= billed + non-billed).
* style(governance): drop em-dashes from cost-split copy and comments
* fix(me): render Recent activity through the real /traces table primitives
The card re-assembled the trace cells into its own Chakra Table.Root, so
the header typography, row borders, and the Input/Output summary row drifted
from /traces. Render through the explorer's own TraceTableShell (header) and
RegistryRow (rows + io-preview addon) with a local non-interactive tanstack
table instead, so a personal trace reads pixel-identical to the trace
explorer. The trace column is flexed for the compact card so it absorbs slack
instead of forcing horizontal scroll.
* fix(me): flush Recent activity table to card edges + stop trace overflow
Two follow-ups on the Recent activity card:
- Make its SectionCard content full-bleed (no side padding) with a gray top
divider under the title, so the table reads edge-to-edge like /traces while
the title stays inset.
- Flex the trace column correctly. The shared column def caps trace at
maxSize 820, so the flex sentinel (size 9999) was clamped to 820 and the
resize heuristic pinned it to a fixed 820px, overflowing the card with a
horizontal scrollbar. Lift maxSize on the card's copy so the column truly
flexes and the table fits the card width.
* fix(governance): tool-catalog Delete actually deletes, with confirm
Delete soft-archived a tile (archivedAt + enabled=false) but adminList
still read archived rows, so the 'deleted' tile stayed in the catalog editor
while it vanished from members' /me portals. Retire the archive split-brain:
- Service: remove() hard-deletes the row (department / team bindings cascade,
nothing else FKs the tile, so no orphans). listForAdmin filters archivedAt
so any legacy archived rows stop showing.
- Router + UI: rename archive -> remove; route Delete through a confirm
dialog since it is now permanent. Disable stays as the reversible hide.
- Banner: center the getting-started hero within its min-height so it stops
pooling empty space below the button.
- Specs + integration test updated: deleted entries are gone from both the
admin and user lists.
* feat(governance): add 'Add tools to the catalog' as the first setup step
The governance setup checklist skipped the catalog entirely. Add it as the
first step, checked once the org has at least one live catalog tile, so admins
start by publishing the tools their team installs from /me before wiring
routing, ingestion, and anomaly rules.
* fix(governance): stamp bundled-cost marker on log-based OTLP ingest
Claude Code (and other coding assistants) emit OTLP *logs*, not spans, so
their traces flow through the /logs receiver path. Only the /traces path
resolved and stamped langwatch.cost.non_billable, so a real claude_code
session's cost was never marked bundled and read as real spend. Resolve
nonBillable on the log path too and pass it to the log provenance stamper.
Proven via a real langwatch claude (Max plan) dogfood: the trace lands with
non_billable=true and the cost splits to the theoretical bucket.
* feat(traces-v2): show 'Bundled' in purple instead of an em-dash for bundled cost
A fully-bundled trace billed nothing, so the cost pill / cell rendered a bare
em-dash that read as missing data. Show 'Bundled' in purple (the drawer pill
via a purple Chip tone, the table CostCell via purple text) so a bundled
coding session is legible at a glance; the popover / hover still breaks down
billed vs theoretical.
* test(governance): cover bundled-cost marker on the OTLP log path
Logs are the path coding assistants (Claude Code) actually use, so guard that
stampIngestKeyProvenanceOnLogRequest carries langwatch.cost.non_billable.
* feat(me): plot billed vs theoretical spend on the /me cost charts
The Spending-over-time and By-tool cards showed a single total (list-price)
series, so a bundled Claude Max session read as real spend. Split each into a
theoretical series (full list price, includes bundled) and a billed series
(actual per-token spend), derived from the langwatch.cost.non_billable marker
in PersonalUsageService. Render both overlaid with distinct tones (purple
theoretical, green billed) and a clickable legend to toggle each: hide
theoretical and the chart shows only what was actually spent. The axis
rescales to the visible series so a near-zero billed series still reads.
* fix(traces-v2): include current-hour turns in the conversation tab
The conversation-turns query floored its upper time bound to the current
hour to keep the React Query key stable across re-opens. Any turn recorded
in the current hour then sat after that bound and was excluded, so the
Conversation tab showed "No turns found" while the pager (which queries up
to now) still counted them. Round the bound up to the next hour instead:
the key stays stable within the hour and current-hour turns are covered.
* feat(traces-v2): add Thread layout to the conversation tab
Adds a ChatGPT-style Thread layout as the first (and default) render mode
on the drawer's Conversation tab, alongside Bubbles, Markdown and
Annotations. Thread stacks both roles full-width in an 800px centered
column; bubbles keep spanning the pane. ChatTurnRow grows a `layout` prop
that swaps the side bubble for a full-width role row, reusing the same
tone/label/annotation inputs so toggling never changes what's shown.
Also fixes the turn-divider lines stopping short of the edge: the inline
hover actions reserved invisible width in flow, so the rules never reached
the container edge. They now float over one end, out of flow, and the lines
span the full width.
* feat(traces-v2): reuse the enriched cost breakdown on the list's bundled cell
The trace list's bundled cost cell showed a flat one-line title tooltip
("Bundled plan, not billed per token. Billed —, theoretical $0.63."). Extract
the trace-details cost pill's breakdown into a shared CostBreakdownTooltipContent
and render it from the list cell, so hovering a bundled cost shows the same
billed / non-billed / theoretical split. TooltipRow moves to components/shared
so both surfaces share it.
* feat(traces-v2): thread turn opens Summary, drop the selected highlight
Clicking a turn in the conversation Thread layout now jumps to that turn's
Summary tab instead of the raw Trace tab, and applies the mode transiently so
peeking at a turn doesn't repoint the user's remembered drawer tab (new
setViewModeTransient + a persistViewMode flag on navigateToTrace). The active
turn also no longer carries a persistent gray background — it reads flat like
the rest of the thread, with only a transient hover cue.
* fix(traces-v2): surface claude utility-call output on the span
Claude Code sends the reply for its utility model calls (prompt_suggestion
autosuggest, generate_session_title) via api_response_body, but the converter
gated that text off the span for non-conversational query sources, so drilling
into a prompt_suggestion span showed no output even though the model returned
one. Attach the reply to every model-call span; the trace-level headline output
stays conversational-only by skipping claude utility spans in the fold
accumulation (the same way tool spans are skipped), so a suggestion or title
never becomes the trace's displayed output.
* feat(traces-v2): show span_id as the first row in span attributes
Inject the span id as a synthetic, always-first row in the span attributes
table so it's one-click copyable. It isn't a real OTel attribute, so it sorts
ahead of the alpha/pin order, is searchable + included in copy-all, and renders
non-pinnable (a synthetic key can't resolve a pinned trace-header value).
* fix(traces-v2): roll up bundled cost per span, not a trace-global boolean
A trace's LLM cost splits into the portion actually billed per token
(gateway / pay-per-token) and the bundled portion covered by a flat
subscription (e.g. Claude Max), which is theoretical. This was derived
all-or-nothing from a single hoisted trace-summary attribute
(langwatch.cost.non_billable), which cannot represent a trace that mixes
billed and bundled spans.
Fold a real per-span NonBilledCost amount instead: each span (and claude
log-record) cost is classified at fold time from the receiver-stamped
marker, a resource-level default a span can override, so a trace rolls up
two real amounts (billed = TotalCost - NonBilledCost). The marker is no
longer hoisted onto the trace attribute map.
All read paths (trace list, drawer router, analytics metric-translator,
/me personalUsage) read the folded amount and fall back to the legacy
boolean for rows folded before the column existed, so nothing reprojects.
Dogfood-verified on a real Claude Max session: NonBilledCost == TotalCost
for the bundled trace and the boolean no longer appears in Attributes.
* fix(me): restore personal usage rollups broken by aggregate-alias collision
The dailyBuckets and breakdownByModel queries aliased the outer cost
aggregate with the same name as the inner per-trace column
(sum(SpentUsd) AS SpentUsd), so the SpentUsd inside
sum(coalesce(SpentUsd, 0) - NonBilledUsd) bound to the outer aggregate and
ClickHouse rejected the query ("aggregate function is found inside another
aggregate function"). The throw rejected the /me Promise.all, collapsing
the whole personal-usage dashboard to an all-zero empty state with an
empty spend chart. Rename the inner column to TraceSpentUsd so the billed
split resolves against the subquery. Adds an integration test that
executes all three rollup queries against ClickHouse.
* fix(me): highlight the personal sidebar Traces and library entries
The personal sidebar hardcoded isActive={false} on Traces, Evaluations,
Datasets, Annotations and Automations, so none of them lit up when you
were on their per-project page. Derive active state from the current path
the same way MainMenu does for project nav.
* fix(me): blue spend palette, zero baseline ticks, and bar tooltips
The /me spend charts now use one blue family (light blue theoretical,
blue billed) instead of purple plus green, render a thin baseline tick on
every day so empty days still read on the timeline, and show a real hover
tooltip with the day plus theoretical and billed amounts on Spending over
time and By tool (replacing the bare title attribute).
* feat(claude-otlp): full-turn converter + truncation-tolerant tool-output recovery
Rework the claude-code log->span converter to fold a whole turn's logs at
once instead of one export batch, so a model call whose request body and
anchor land in different batches keeps both its input and output. Recover
each tool span's output from the next model call's tool_result transcript
block, scanning the real 60KB-truncated request body that does not JSON.parse.
Render a tool-deciding call's tool_use as its span output, and nudge an
incomplete span's StartTime so a later, more complete re-fold wins the
stored_spans max(StartTime) read dedup. Spec + unit tests bound to the new
scenarios; grounded against the real raw OTLP dumps.
* feat(claude-otlp): save claude logs + reactor folds them into a hierarchy of spans
Stop trapping claude_code model/tool/user_prompt logs at ingest. The receiver
now SAVES them to stored_log_records marked with langwatch.claude_code.kind, and
a new claudeCodeSpanSync reactor (tied to the logRecordStorage projection) reads
the whole turn's saved logs and folds them into spans: one root turn span (the
user_prompt, carrying the turn input) with the model and tool calls as children.
This is the event-sourced, ingestion-stateless, idempotent re-architecture: the
receiver holds no cross-batch state, and the reactor re-derives over the full
turn so a model call split across export batches keeps both input and output,
…
0xdeafcafe
added a commit
that referenced
this pull request
Jul 3, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
Rebasing onto origin/main dropped three main-side changes that lived in files this branch deletes/relocates/restored; the merges that carried them are gone in a linear history, so re-apply them explicitly: - runEvaluation: port #4729 (unified scoped data privacy) — native evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation) + augmentEvaluationResult(droppedCategories) on both return paths. The branch extracted runEvaluation from the pre-#4729 worker, so without this the data-privacy augmenter silently vanished from evals. - topicClustering: take origin/main's topicClustering.ts (+ co-located ./topicClusteringQueue import) — the restored copy was a pre-#4653/ #4828/#4571 snapshot still routing to the removed python langwatch_nlp service. Restores nlpgo-only routing (#4653), the single argMax count pass (#4828), and the page-fetch read-stream cap (#4571). Align the two topicClustering tests with main (langevals mock) + co-located queue path. collectorWorker's #4729 data-privacy is already applied downstream in the event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs no change.
0xdeafcafe
added a commit
that referenced
this pull request
Jul 7, 2026
* refactor: remove Elasticsearch and the BullMQ legacy stack
The TraceService / EvaluationService / ExperimentRunService /
FilterService facades all had two backends — a ClickHouse one and an
Elasticsearch one — but every call was already going through ClickHouse.
The ES backends were constructed and never invoked. The legacy BullMQ
stack under src/server/background/ had been replaced piecewise by the
event-sourcing collector (recordSpan) but the old workers, queues, and
the tasks that wrote to ES were still wired up at boot.
This commit deletes both stacks from the application layer:
ES removal (live read/write paths):
- Drop the ElasticsearchTraceService / ElasticsearchEvaluationService /
ElasticsearchExperimentRunService / ElasticsearchFilterService /
ElasticsearchAnalyticsService backends (constructed but never called).
- Drop the evaluationEsSync + experimentRunEsSync reactors and the
annotationEsSync hook — no more dual-write.
- Drop the ES analytics timeseries + scenario-analytics query builders.
- Drop the ES experiments batch-evaluation repository + the ES
scenario-event service/repository (the SimulationFacade was already
routing reads to ClickHouse).
- Strip Elasticsearch URL/API-key fields from the org settings input
schema, the settings UI, and getApp().organizations.update.
- Drop the per-org ES-driven cron handlers
(scenario_analytics, trace_analytics msearch, retention_period_cleanup,
schedule_topic_clustering) and the ES task scripts they invoked
(elasticMigrate, cold/*, deleteTracesRetentionPolicy, syncAnnotations,
syncTraceCosts, backfillTraceAnalytics, backfillScenarioAnalytics,
fixCustomMetadataOutsideCustomField, clearTopics, rerunChecks).
- Collapse env.IS_QUICKWIT branches.
- Replace ES query-builder type imports
(@elastic/elasticsearch/lib/api/types) with neutral structural types
in the filter / analytics / common-router files that never actually
ship queries to ES anymore.
- Move the Protections type out of src/server/elasticsearch/protections
to src/server/traces/protections.
BullMQ legacy stack removal:
- Delete src/server/background/ entirely (worker entry, the six queue
files, collectorWorker, evaluationsWorker, topicClusteringWorker,
usageStatsWorker, anomalyDetectionWorker).
- Move the pure helpers under background/workers/collector/ (cost,
common, evaluations, evaluationNameAutoslug, metrics, piiCheck, rag)
into src/server/tracer/collector/ — these are consumed app-wide and
don't depend on BullMQ.
- Extract runEvaluation / runEvaluationForTrace / customEvaluation /
DataForEvaluation from the deleted evaluationsWorker into
src/server/evaluations/runEvaluation.ts, switching from ES
getTraceById / getTracesGroupedByThreadId to TraceService.getById /
getTracesByThreadId.
- Rewrite src/workers.ts: drop the 15-minute self-restart timer
(helm/docker already restart-on-exit), drop the BullMQ Worker
startups, initialize the worker-role app, and start the EE
governance ingestion-puller worker via a new
ee/governance/services/pullers/pullerQueue.ts.
- Migrate src/server/api/routers/httpProxyTracing.ts from
scheduleTraceCollectionWithFallback to getApp().traces.recordSpan via
CollectorSpanUtils — the only remaining caller of the old collector
queue.
- Strip the legacy collector dedup (fetchExistingMD5s + 256-version
cap) from src/server/routes/collector.ts; event sourcing handles
dedup downstream.
- Drop /api/start_workers, /api/rerun_checks,
/api/cron/schedule_topic_clustering,
/api/cron/scenario_analytics,
/api/cron/traces_retention_period_cleanup — the routes were
exclusively legacy stack triggers.
- Delete src/server/topicClustering/ (BullMQ-only scheduler) and the
/settings/topic-clustering page + command-bar entry + route.
- Rewrite TraceUsageService CH-only (drop the EsClientFactory
constructor parameter and the splitProjectsByFlag fallback).
Core ES dir + package:
- Delete src/server/elasticsearch.ts + src/server/elasticsearch/
(traces, transformers, patchOpensearch, patchQuickwit).
- Delete elastic/ (helpers, mappings, schema, all migrations).
- Remove @elastic/elasticsearch from package.json + lockfile.
- src/components/checks/TryItOut.tsx: drop the
transformElasticSearchSpanToSpan call; CH spans already arrive in the
domain Span shape.
Facade collapse:
- Collapse EvaluationService into a single CH-backed class
(delete clickhouse-evaluation.service.ts, rename methods to throw on
missing CH client instead of returning null, keep getEvaluationInputs
nullable for the legitimate empty case).
- Collapse ExperimentRunService into a single CH-backed class
(delete clickhouse-experiment-run.service.ts, fold
listRunsForExperimentSlugPaginated into the renamed class).
- Collapse FilterServiceFacade → FilterService into a single CH-backed
class (delete clickhouse-filter.service.ts).
- Delete SimulationFacade entirely — every consumer (suite router,
scenario events router, simulation-runs REST, cancellation router,
onboarding-checks service) now calls getApp().simulations.runs
directly.
193 files changed, -29.8K LOC. The only remaining BullMQ consumer in
the codebase is the EE governance ingestion puller, which keeps
BullMQ's repeatable-jobs cron primitive until it migrates to event
sourcing (follow-up).
* refactor(context): isolate the BullMQ adapter to the EE governance puller
After the legacy stack removal the only BullMQ-importing file left in
src/server/ was the context adapter at adapters/bullmq.ts, and the only
caller of its withJobContext helper was the EE governance ingestion
puller worker. The other three exports
(createContextFromJobData / getJobContextMetadata / JobDataWithContext)
have no BullMQ dependency — they're pure ALS plumbing.
- Move the pure helpers into src/server/context/core.ts so the
framework-neutral request-context module no longer touches BullMQ.
- Move withJobContext (the one piece that imports BullMQ's Job type)
into ee/governance/services/pullers/withJobContext.ts, co-located
with its only consumer.
- Move the existing withJobContext test cases (new __context shape +
legacy __payload migration paths) into
ee/governance/services/pullers/__tests__/withJobContext.unit.test.ts.
- Delete src/server/context/adapters/bullmq.ts and update
asyncContext.ts to re-export the pure helpers from core only.
Result: BullMQ is now contained entirely in
ee/governance/services/pullers/. The next time the puller migrates to
event sourcing, bullmq and bullmq-otel come out of package.json as a
one-liner with no other callers to migrate.
* fix: restore topic clustering (deleted in error)
Topic clustering was incorrectly bundled into the ES/BullMQ removal in
the parent commit. It is a live feature that:
- Reads from ClickHouse via TraceService (not ES).
- Uses BullMQ only for the worker queue, which is fine — the legacy
stack removal targeted the per-call collector / evaluations workers,
not every BullMQ consumer.
Co-locate the queue + worker under src/server/topicClustering/ (rather
than the deleted src/server/background/queues/ + workers/), and re-home
the shared BullMQ helpers (QueueWithFallback + withJobContext) under
src/server/queues/ since they are now used by both the EE governance
puller and topic clustering.
Restored:
- src/server/topicClustering/{topicClustering,types,topicClusteringQueue,topicClusteringWorker}.ts
- src/server/topicClustering/__tests__/* (3 files)
- src/tasks/runTopicClustering.ts
- /api/cron/schedule_topic_clustering handler
- project.triggerTopicClustering tRPC mutation
- /settings/topic-clustering page route + command-bar entry
Wired the topicClusteringWorker into src/workers.ts alongside the EE
ingestion puller.
Test-mock fixes for stale ~/server/background/... imports flagged in
review (3 integration tests + 2 unit tests, plus a now-defunct queues-dir
scan in no-retention-gc.unit.test.ts):
- httpProxyTracing.integration.test: rewrite to mock
getApp().traces.recordSpan and decode the OTLP span into the legacy
CollectorJob shape via a small facade — assertion bodies unchanged.
- evaluations.azure-byok.integration.test: drop mocks of long-gone
runEvaluationJob / startEvaluationsWorker symbols; mock just
runEvaluationForTrace at its new home.
- span-pii-redaction.service.test + topicClustering tests: update mock
paths from ~/server/background/... to the new locations.
* fix(lockfile): drop stale @elastic/elasticsearch importer + pin vite
CI was failing on `pnpm install --frozen-lockfile` because the lockfile
still listed @elastic/elasticsearch ~8.15.3 as a direct dep of langwatch/
even though the package.json no longer declares it. The previous rebase
resolution preserved main's lockfile but kept my branch's package.json,
so they drifted.
Surgical fix: remove the @elastic/elasticsearch entry from the importer
block. Orphan package + peer-dep references downstream are inert and
clear out next time someone regenerates the lockfile.
Also pin `vite` to 8.0.10 via overrides. vite 8.0.16 (published
2026-06-01) hasn't aged past the workspace's 7-day minimumReleaseAge
gate yet, so any `--no-frozen-lockfile` run hits the maturity check
trying to resolve vitest's peer. The pin doesn't help today's gate trip
(overrides apply after maturity checks), but it stops future regens from
churning between 8.0.10 and the latest unaged patch.
* fix: restore node:crypto import in httpProxyTracing.ts
The legacy-stack removal commit dropped `import crypto from "node:crypto"`
along with the other imports it was rewriting, but two callers of
`crypto.randomBytes` (line 96/97 — trace_id and span_id generators)
still reference it. Typecheck on the global Web Crypto sees no
`randomBytes` and fails.
* fix: chase test fallout from the ES/BullMQ removal
CI flushed out a pile of stale references that survived the previous
commits because nothing on the happy path touched them.
Production:
- evaluation.service.ts: getEvaluationInputs returns null when no
ClickHouse client is available (commit message claimed this, code
threw — restore the documented contract; the eval-inputs panel is
load-bearing on the trace drawer and a 500 cascade poisons polling).
- httpProxyTracing.ts: don't throw when the project lookup misses —
fall back to DEFAULT_PII_REDACTION_LEVEL. The tenantId is the
authoritative project id from the caller; the project row is only
consulted for the redaction setting and is fine to default.
Test fixes:
- internal-routes-auth.integration.test: repoint at
/api/cron/old_lambdas_cleanup (the deleted retention-cleanup,
start_workers, rerun_checks endpoints no longer exist; the kept
destructive cron exercises the same cronPolicy() wiring).
- metrics.test + metrics.unit.test: vi.mock paths were one level deep
too far after the collector move from src/server/background/workers/
to src/server/tracer/. Real getLLMModelCosts then hit Prisma in unit
tests with no DB.
- experiment-run.service.getRun-null.unit.test: the test was patching
service.clickHouseService (gone after the facade collapse). Rewrite
to mock getClickHouseClientForProject so the "client unavailable"
and "row not yet materialized" cases stay covered.
- experiment-run-dedup-safety + trace-dedup-oom-safety: the structural
guard read clickhouse-experiment-run.service.ts directly. Point at
the merged experiment-run.service.ts (clickhouse-trace.service.ts
still exists — only the experiment-run side collapsed).
- cancellation-event-sourcing.integration.test: vi.mock("~/server/redis")
was a partial factory that dropped makeQueueName. After the TC
restoration the test's transitive imports need it; spread
importOriginal() so the rest of the module passes through.
- httpProxyTracing.integration.test: also mock the relative-path
spelling of ~/server/app-layer/app so the route's own import is
intercepted regardless of how vitest canonicalises specifiers.
* fix(build): isolate makeQueueName so the client bundle stops pulling ioredis
The boot-smoke test was failing the entire PR because the simulations
route chunk (`__...path__-XXXXX.js`) tried to evaluate ioredis at
top-level in the browser and tripped `process is not defined`
(via `process.nextTick`).
Root cause: when the legacy stack was removed, `makeQueueName` moved
from `~/server/background/queues/makeQueueName.ts` (a tiny standalone
file) into `~/server/redis.ts`. `~/server/scenarios/scenario.constants.ts`
imports `makeQueueName`, and the SimulationsPage route transitively
pulls in scenario.constants for its queue identifiers. So every
SimulationsPage chunk now dragged the whole `~/server/redis` module
— and ioredis — into the client bundle.
Fix: pull `makeQueueName` back into its own file
(`~/server/queues/makeQueueName.ts`), re-export it from `~/server/redis`
for compatibility, and switch scenario.constants to the dedicated path.
Local `vite build` + `node scripts/smoke-boot.mjs` now reports
"BOOT SMOKE PASSED" with no externalised-node-builtin warnings.
* fix: hoist vi.mock factory + mark orphaned scenarios @unimplemented
httpProxyTracing.integration.test was failing every test ("There was
an error when mocking a module") because the vi.mock factories
referenced a top-level `fakeApp` variable. Vitest hoists vi.mock
above all top-level code, so closures over outer variables fail.
Move the shared mock fn into vi.hoisted() and inline the factories.
feature-parity: 19 orphaned scenarios survived the legacy-stack
removal because the tests that bound them lived under
src/server/background/ (deleted) or referenced ES storage layers
that no longer exist. Tag the orphans @unimplemented so the parity
check stops failing — re-binding them to event-sourcing-backed
tests is follow-up work, not in scope here:
- Redis Cluster compat: 4 scenarios tested BullMQ queue hash-tag
behaviour. After the legacy-stack removal the only BullMQ users
are the EE puller + topic clustering; the tests can be rewritten
against those queues but it's a fresh integration-test surface.
- Extensible scenario metadata: 11 ES-storage-round-trip and
metadata-namespace scenarios. ES is gone; equivalent ClickHouse
projection coverage is a follow-up.
- Monitor execution backend: 3 evaluator-settings scenarios that
were bound to evaluationsWorker.integration.test.ts (gone).
- API endpoint authorization: drop "Worker and ops trigger endpoints
reject callers..." (the endpoints don't exist) and repoint the
destructive-cron scenario at the surviving old-lambdas-cleanup
route.
* fix(topic-clustering): break circular import causing worker boot TDZ
Restoring topic-clustering reintroduced a runtime regression: workers
crashed on boot with `ReferenceError: Cannot access 'TOPIC_CLUSTERING_QUEUE'
before initialization`. The cycle is
topicClusteringWorker.ts
-> topicClustering.ts (line 15: clusterTopicsForProject)
-> topicClusteringQueue.ts (line 14: scheduleTopicClusteringNextPage)
-> topicClusteringWorker.ts
When workers.ts imports the queue first, queue.ts re-enters worker.ts via
topicClustering.ts. queue.ts line 18 then reads `TOPIC_CLUSTERING_QUEUE.NAME`
while worker.ts is still on its imports — the const is in TDZ.
Extract the queue name + job-type tuple to a leaf
`topicClusteringQueue.constants.ts` with no cyclic edges. Both worker.ts
and queue.ts now import the constant from there; the cycle on
`runTopicClusteringJob` survives but that's a hoisted function declaration,
so its binding is available regardless of which side enters first.
Caught by `make quickstart all-local`. With the fix the worker logs
"topic clustering checks worker registered" and "topic clustering worker
active, waiting for jobs!" cleanly.
* fix(prompts): stop polling traces.getById for failed assistant messages
When a prompt submission errors (e.g. NLP unreachable, provider 500,
upstream rate-limited), the chat renders an ErrorMessage instead of an
AssistantMessage — but TraceMessage was still mounted unconditionally
on every non-streaming assistant turn. That meant every failed
submission also mounted a trace polling component looking up a trace
that was never emitted, producing a chain of 404s on
`api.traces.getById` (retry: 10, exp-backoff to 60s) and a "Maximum
update depth exceeded" React warning in posthog session-replay.
Gate the TraceMessage mount on `!isError`. Errored assistant turns have
no backing trace; the only thing left to render is the ErrorMessage,
which is already shown above.
Verified live against `make quickstart all-local` with no NLP container
(repro path for "LangWatch NLP is unreachable"). Before: 15 console
errors per submit; after: 1 unrelated pre-existing SVG warning.
* chore(evaluations): drop dead imports and string throw in runEvaluation
- remove unused imports left over from the evaluations-worker extraction
(CostReferenceType, CostType, nanoid, getProtectionsForProject,
captureException, withScope, createLogger + the unused logger instance)
- throw new Error("trace not found") instead of a bare string
* fix(tests): chase moved modules in tests added on main
- Protections import: ~/server/elasticsearch/protections -> ~/server/traces/protections
- getRun integration test: clickhouse-experiment-run.service was folded
into experiment-run.service; use ExperimentRunService
* chore: purge remaining Elasticsearch remnants from the audit
- usage metering: drop the dead MeterBackend concept entirely —
resolveUsageMeter no longer takes clickhouseAvailable or returns a
backend (nothing ever consumed it; the only non-ClickHouse value was
the removed Elasticsearch fallback); UsageService and presets wiring
updated, backend-selection tests removed
- delete stale comments claiming ES is a live backend (org settings
migration note, experiment-run facade fallback, filter fallback,
license-enforcement counting source, mapper/type docs, stall
detection, synthetic error span, llmParameterMap, ee licensing)
- reword legacy snake_case mapper docs that described ES as current
Intentionally kept: history notes, regression guards asserting ES
absence, legacy ESBatchEvaluation/ElasticSearch* shape names, and the
unused Prisma elasticsearch* columns (follow-up migration).
* fix: chase moved cost module in model-cost-span-preview and drop stale UsageService ctor arg
* chore(lint): satisfy biome on files touched by the main merge
Biome's organizeImports ordering shifted after merging main's import
changes; re-sort the affected files. Also drop the dead `internalAuth`
const and collapse two `!x || !x.method()` guards to optional chaining
in misc.ts (biome noUnusedVariables + useOptionalChain).
* fix(rebase): re-integrate main features that landed in rebased-over code
Rebasing onto origin/main dropped three main-side changes that lived in
files this branch deletes/relocates/restored; the merges that carried
them are gone in a linear history, so re-apply them explicitly:
- runEvaluation: port #4729 (unified scoped data privacy) — native
evaluator short-circuit (isNativeEvaluatorType/executeNativeEvaluation)
+ augmentEvaluationResult(droppedCategories) on both return paths. The
branch extracted runEvaluation from the pre-#4729 worker, so without
this the data-privacy augmenter silently vanished from evals.
- topicClustering: take origin/main's topicClustering.ts (+ co-located
./topicClusteringQueue import) — the restored copy was a pre-#4653/
#4828/#4571 snapshot still routing to the removed python langwatch_nlp
service. Restores nlpgo-only routing (#4653), the single argMax count
pass (#4828), and the page-fetch read-stream cap (#4571). Align the two
topicClustering tests with main (langevals mock) + co-located queue path.
collectorWorker's #4729 data-privacy is already applied downstream in the
event-sourcing recordSpanCommand path, so collector.ts (recordSpan) needs
no change.
* fix(rebase): chase main-renamed modules + narrowed APIs after rebase
The dropped main-merges also carried module renames and a signature
narrowing that the branch's deleted/relocated files and main's newer
files now straddle. Re-apply across the affected files so typecheck +
biome pass clean (0 type errors):
- Protections moved out of src/server/elasticsearch/ -> src/server/traces/;
retarget 10 main-added files (projection, tracesV2, redactAttributes,
data-privacy tests) that still imported the old path.
- main consolidated the generated zod/type modules: evaluators.zod.generated
-> evaluators, {tracer,experiments}/types.generated -> {tracer,experiments}/
types. Retarget the legacy-route files (collector, misc, evaluations-legacy).
- piiCheck relocation background/workers/collector -> tracer/collector:
retarget the two stray test imports.
- captureException narrowed to Error|string (#2155): wrap the unknown
catch args via toError() in evaluations-legacy + misc.
- Project.piiRedactionLevel removed by #4729: collector + misc track-event
spans now pass DEFAULT_PII_REDACTION_LEVEL (the policy is resolved
downstream in the recordSpan pipeline).
biome --write applied (import sort + optional-chain); one pre-existing
noUnusedVariables warning remains (also present on origin/main).
* fix(evaluations): enrich trace evaluations in runEvaluationForTrace (P1)
The worker→runEvaluation extraction swapped the legacy
getTraceById({ includeEvaluations: true }) for TraceService.getById,
which routes through getTracesWithSpans and does NOT enrich evaluations
(that only happens in the paginated getAllTracesForProject path). The
field mapper reads `trace.evaluations ?? []`, so any evaluator whose
input maps from the `evaluations` source (a prior evaluator's result)
was silently fed [] — a quiet correctness regression with no crash.
Fetch evaluations via TraceService.getEvaluationsMultiple (already mapped
to the legacy Evaluation[] shape the mapper expects) and attach them to
the trace before buildDataForEvaluation, restoring parity with the worker.
Note: this path still has no executing test (its spec is @unimplemented);
a regression test that asserts the `evaluations` source maps real data is
the right follow-up.
* fix(ci): repair rebase regressions in unit + integration tests
Four CI failures, all from the rebase taking older snapshots of branch-only
files or losing main's schema/query evolution:
- evaluations-legacy.ts: restored the branch's working getEvaluatorDataForParams
(coercedString/coerceEvaluatorScalar preprocessing + the CODE_EVALUATOR_CHECK_PREFIX
code-evaluator branch were dropped to an older snapshot). Re-applied the
main-driven module renames + captureException(toError) narrowing on top.
Fixes evaluator-input-coercion.unit.test.ts (10).
- misc.ts: restored the RoleBinding-aware /mcp/authorize handler (the old
legacy-TeamUser check had been re-introduced, returning 500/false-403).
Re-applied renames + toError + DEFAULT_PII_REDACTION_LEVEL. Fixes
mcp-authorize.rolebinding.unit.test.ts (4).
- experiment-run.service.ts: getRun now uses main's scalar
UpdatedAt = (SELECT max(UpdatedAt) ...) single-run read instead of the
IN-tuple. Fixes experiment-run-dedup-safety.unit.test.ts (2).
- httpProxyTracing.ts: drop the now-invalid select { piiRedactionLevel }
(#4729 removed Project.piiRedactionLevel) that threw PrismaClientValidationError;
use DEFAULT_PII_REDACTION_LEVEL. Fixes httpProxyTracing.integration.test.ts.
biome --write applied across the rebased diff to clear reviewdog lint.
* fix(p2)+test: stale ES string, document worker boot-ordering, P1 regression test
- settings.tsx: drop the stale "or Elasticsearch" from the org-update error
toast (the ES config fields were removed).
- workers.ts: document why the puller/topic-clustering worker + queue modules
are imported via await import() — they construct Redis-connecting
QueueWithFallback instances at module load, so they must load only after
verifyRedisReady(). (Not converted to top-level: that would connect to Redis
before it's verified.)
- runEvaluation.evaluations-enrichment.unit.test.ts: regression test locking the
P1 fix — an evaluator mapping from the `evaluations` source receives the
trace's real evaluations (via getEvaluationsMultiple), not a silent []. Verified
it fails when the enrichment is removed.
* fix(remediation): restore behavior dropped in the ES/BullMQ removal
Verified P0 + behavioral P1s from the PR #4547 audit (the blocker set). Each path regressed because logic was dropped while extracting code out of the deleted BullMQ worker.
- workers.ts: re-wire worker-role boot — scenario execution pool (P0: simulations were silently skipped on workers), worker /metrics listener, anomaly detector, ClickHouse storage-stats collection; add graceful SIGINT/SIGTERM shutdown with handle cleanup.
- runEvaluation.ts: restore code-evaluator routing (code/<id> -> runCodeEvaluator) and the embeddings_model env setup + openai/moderation guard the extracted module lost vs the canonical service.
- evaluation-execution.service.ts: enrich trace.evaluations in executeForTrace before building mapped data (parity with runEvaluationForTrace; the monitor path fed an empty evaluations source).
- routes/collector.ts: route REST ingestion through ingestNormalizedSpan so it shares the (tenant,trace,span) dedup gate + ADR-022 spool with OTLP, instead of calling recordSpan directly.
- tracer/collector: restore unit coverage for the moved span-tree/text-extraction helpers (deleted common.test.ts had no committed replacement).
typecheck clean; 235 evaluation unit tests green.
* refactor(analytics,experiments-v3): remove dead Elasticsearch query and mapper builders
Audit §B6 follow-up to PR #4547. No live callers remain after the ES backends were dropped: generateTracesPivotQueryConditions/generateFilterConditions (analytics/common.ts) and mapEsRunToExperimentRun/mapEsBatchEvaluationToRunWithItems + ESRunAggregationBucket (experiments-v3/mappers.ts). Keeps the still-live mapEsTargetsToTargets (used by routes/evaluations-legacy.ts). Drops the now-orphaned tests and a stale filter-conditions comment.
* docs(specs): drop stale Elasticsearch/BullMQ references from scenario specs
Audit §C follow-up. extensible-scenario-metadata: neutralize/remove the ES-storage scenarios (scenario events persist to ClickHouse via event sourcing). monitor-execution-backend: note evaluationsWorker + its test were removed with the BullMQ stack (monitor evals now run via executeEvaluation.command.ts) and correct results-stored-in-Elasticsearch -> ClickHouse. redis-cluster-compatibility: minor comment wording.
* fix(ui): restore canonical ModelOption import in LlmConfigField
Audit P2 follow-up. Replace the hand-rolled ModelOption type with the import from ~/server/topicClustering/types; topic clustering was restored in this PR, so the canonical type exists.
* fix(lint): satisfy biome organize-imports on PR-changed files
* fix(test): stub traceService.getEvaluationsMultiple in EvaluationExecutionService integration mocks
The executeForTrace evaluations-enrichment (getEvaluationsMultiple, added in 689be85) broke three integration tests whose hand-rolled traceService mocks didn't stub the new method (TypeError: getEvaluationsMultiple is not a function across shards 1/2/5). Stub it to resolve {} (no-op enrichment), mirroring the unit-test mock. typecheck clean.
* fix: address /review-pr re-audit findings (P1 collector + P2 worker hardening)
- collector.ts (P1): REST /api/collector counted ingestion failures via Promise.allSettled 'rejected', but ingestNormalizedSpan catches its own errors and RESOLVES with {status:'failed'} (never rejects) — so rejectedSpans was always 0 and failed spans were reported to the SDK as success with no error log. Now inspects the resolved SpanIngestionResult.status (still treats an unexpected rejection as a failure).
- workers.ts /metrics (P2 security): the worker metrics listener served the full prom-client registry unauthenticated on 0.0.0.0 while the web path is bearer-gated. Extract isMetricsAuthorized into server/metrics.ts and apply it to the worker listener too (fail-closed in prod).
- workers.ts shutdown (P2): gracefulShutdown discarded the topic-clustering + ingestion-puller worker handles and never closed the App; now registers both close handles and closes getApp() (CH/Redis/Prisma) last.
- evaluation-execution.service.unit.test.ts (P2 test): the A4 executeForTrace evaluations-enrichment was untested (every mock stubbed getEvaluationsMultiple to {}); add a test asserting prior evaluations reach the mapped data.
typecheck + biome clean; 208 evaluation unit tests pass.
* chore: drop unused prisma import in httpProxyTracing (codeql/code-quality)
* docs(experiment-run.service): document getRun's deliberate null-on-no-CH (#4547 review)
Per Aryansharma28's review (thread PRRT_kwDOKRXhvM6L6N-6): getRun's null-on-no-CH was a deliberate UX call (PR #3483 dogfood — fold-window polling race) but the class-level JSDoc still claimed all methods return null, and getRun's own JSDoc didn't explain the divergence from listRuns / getRunAggregates / listRunsForExperimentPaginated (which throw). Update the class JSDoc to call out the exception, expand getRun's JSDoc with the rationale + the route caller's intentional 404 mapping, and add an inline comment at the no-CH guard.
* fix(tests): retarget imports from deleted background/collector and elasticsearch/protections (rebase fixup)
PR #4547 moves src/server/background/workers/collector/cost -> src/server/tracer/collector/cost and src/server/elasticsearch/protections -> src/server/traces/protections. Upstream commits landed 3 files that import from the old paths; the rebase onto current main surfaced the dangling imports. Retarget to the new locations.
* fix(evaluations): preserve evaluator-specific extras (pairwise candidate_a_id/output)
Upstream #5142 (pairwise_compare) added a 'preserve evaluator-specific fields' block to background/workers/evaluationsWorker.ts so evaluators that declare extras via required/optional fields (e.g. langevals/pairwise_compare's candidate_a_id, candidate_a_output, candidate_b_id, candidate_b_output) have those fields flow through to langevals via ...extras instead of being stripped by the canonical-6 forward.
This PR extracts runEvaluation out of the deleted worker; without this port, pairwise-compare evaluations silently strip the candidate fields and 422 from langevals. Port the block into runEvaluation.ts (kept path) at the same relative location — just before the stagedLangevalsFetch call — bounded to the evaluator's declared requiredFields ∪ optionalFields so a stray mapping output on a non-pairwise evaluator can't ride through.
* docs(specs): reconcile feature specs with the ES/BullMQ removal (/review-pr audit)
Rewrites scenarios that mandated merging BullMQ jobs with ES scenario
events, restores tags where behaviour ships again with test coverage
(redis-cluster hash-tags, extensible scenario metadata), and drops
implementation-detail phrasing flagged by the audit.
* test: restore coverage lost with the deleted ES/BullMQ test files (/review-pr audit)
Re-homes the deleted suites' assertions against the surviving subjects:
TraceUsageService (incl. the CH-unavailable ?? 0 branch), makeQueueName
hash-tags, metrics auth fail-closed + getWorkerMetricsPort, REST
collector result handling + 429 span cap, googleDLP redaction, stall
derivation, RUN_STARTED metadata pass-through, extensible-metadata
schemas, evaluator-extras allowlist, and monitor settings resolution.
* fix: address /review-pr full-audit findings (P0 spend-spike + P1s)
Restores behaviours silently dropped with the BullMQ stack: the
5-minute spend-spike anomaly evaluation tick, daily self-hosted usage
telemetry, and OTel instrumentation in the worker process. Forwards the
bearer token on the /workers/metrics proxy, returns 500 when every span
in an ingest request fails, fixes googleDLPClearPII discarding its own
redactions, and gates decrypted s3SecretAccessKey behind manage
permission. Deduplicates the track_event route against the shared
service, extracts member-classification helpers, honours traceIds in CH
timeseries, and deletes the dead ES-era helpers, builders, and modules
the audit flagged.
* test(specs): fix @Scenario bindings broken by the ES/BullMQ spec reconciliation
- Retarget five scenario-run-utils annotations to the renamed scenarios
in unified-run-table.feature (ES/BullMQ wording was rewritten)
- Drop the vacuous 'does not import Elasticsearch' source check; the
module no longer exists and the spec scenario was removed
- Restore the missing binding for 'Uses cached count within TTL' on the
trace-usage cache test
- Mark the event_details aggregation-keys scenario @unimplemented: its
ES binding died with the ES backend and events.event_details is not
yet supported by the ClickHouse metric translator
* fix(dev): stop build-mcp-server.sh timestamps breaking under FORCE_COLOR
concurrently (pnpm dev) sets FORCE_COLOR, which makes console.log wrap
numbers in ANSI colour codes even when piped — the coloured start
timestamp then got interpolated into the elapsed-time eval and killed
the whole prepare chain with a SyntaxError. Write raw strings instead.
* feat(nlpgo): route xai, groq, cerebras, and deepseek providers
The playground proxy's provider inference only knew the six original
providers, so a default model like xai/grok-4.3 hard-400'd with
missing_provider on /go/proxy/v1/chat/completions. Map the remaining
TS-registry providers on both nlpgo paths (x-litellm headers for the
playground proxy, inline credentials for llmexecutor/Studio):
- xai / groq / cerebras are Bifrost-native and ride the ProviderID
passthrough with a plain API key
- deepseek is absent from Bifrost's enum; route it through the vLLM
(openai-compat) adapter defaulting to https://api.deepseek.com
- plain api-key providers share one Generic inline-credentials slot
instead of four copy-pasted per-provider slots; the duplicated
provider switches in gatewayproxy are merged into providerForPrefix
Spec: specs/nlp-go/proxy.feature (provider-prefix routing scenarios)
* feat(scenarios): surface typed generation errors from the Go gateway
nlpgo's handled-error envelope (herr) was flattened to a 500 with a
bare string by the scenario-generate route, leaving the browser with
"bad_request" and no way to react. Now:
- nlpgoHandledErrorFrom() parses the envelope off the AI SDK's
APICallError (unwrapping RetryError) into a DomainError keyed by
meta.reason, forwarded as domainError on the response
- the browser service throws ScenarioGenerationError carrying
kind + meta; classifyGenerationError matches typed kinds before
string sniffing (missing_provider → config tier + configure CTA)
- unknown kinds expand kind/message/meta into the raw-message surface
instead of showing an opaque status word
* feat(scenarios): show which model Generate uses, with a settings link
Both the scenario editor panel and the AI create modal now render
"Uses <model> · Change" next to their Generate buttons (resolved via
the scenarios.generator feature key), linking to model provider
settings. Config-tier generation failures get a "Model settings"
toast action. AICreateModal grows an optional footerHint slot.
* fix(ui): dark-mode tokens and layout stability on scenario surfaces
- CurrentDrawer: the lazy-drawer Suspense fallback rendered a 200px
block in page flow, shoving the whole page down while the chunk
loaded; it is now a fixed overlay where the drawer appears
- SimulationCard: reserve clearance for the floating status pill so it
no longer covers the last preview lines
- SaveAndRunMenu, Stop buttons (RunRow, ScenarioGridCard), and the
variable-mapping editor swapped hard-coded light-palette literals
(blue.50, gray.100, orange.50, bg white, raw gray-500 vars) for
semantic tokens so dark mode is readable; missing-mapping error uses
fg.error
* chore: regenerate pnpm lockfile after rebase onto main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the Python
langwatch_nlpservice entirely. The Gonlpgoservice (already serving production via the/go/*paths) becomes the only NLP engine. The self-hosted Docker image and the production per-tenant Lambda drop the whole Python application stack (uvicorn, litellm, dspy, langwatch_nlp), keeping only a minimalpython3+ a small curated set of libraries to sandbox user code blocks.Pairs with langwatch/langwatch-saas#570 (the per-tenant lambda + k8s service infra). Merge order: this PR first, then #570 (its submodule re-bumps to this PR's merge commit).
Why — the win
Per-tenant lambdas are created on a project's first Studio/eval run after each deploy (CreateFunction -> poll Active -> first invoke), so a customer pays the full image-distribution + optimization cost. Measured on lw-dev, fresh never-optimized digests, 1024MB arm64, n=3 median (credit @ash):
~8.2s / ~40% faster per new tenant (or per project's first run after each deploy), with much tighter variance (Go 12.0-13.1s vs fat 16.5-20.9s). The entire delta is
CreateFunction -> Activeand scales with image size. Plus: steady cold-start +102ms (the image lazy-loads the python it never reads on a no-code-block run) and memory 4Gi -> 1-2Gi with no background uvicorn process.Self-hosted image: ~1GB-class -> 136MB.
What changed
services/nlpgo): strip the uvicorn child manager, the reverse proxy, and allNLPGO_CHILD_*config. nlpgo is unconditionally go-only; any non-/go/*request returns a typed 502.release_nlp_go_engine_enabledfeature flag and every Python-fallback branch. Studio execution + the playground proxy always hit/go/*; topic clustering always routes to langevals;execute_optimizationalways 410s.Dockerfile.langwatch_nlp): distroless/python3 + the static Go binary + a 35MB sandbox. Verified end-to-end (/healthz200,/go/version200, non-/go502, sandbox imports work).runner.py+ the curatedsandbox-requirements.txt(the only Python in the artifact).@langwatch/server): runsservice nlpgo(already its defaultgomode); the legacypythonmode + thelangwatch_nlp/directory are deleted./healthz/readyz/startupz.go-ci.yaml), compose, Makefile, scripts. The orphan monorepoDockerfile.langwatch_nlp.lambdais deleted (the saas runtime Dockerfile owns the prod lambda image).Code-block sandbox libraries
services/nlpgo/app/engine/blocks/codeblock/sandbox-requirements.txtis a deliberately small, fixed, documented set installed into both the self-hosted image and the saas lambda image from one file:requests,httpx,pydantic, and thelangwatchPython SDK (prompt/dataset fetch). No pandas/numpy (langwatch does not require them). The old image leaked its entire dependency tree (litellm, dspy, ...) into user code by accident; that is not a contract and is not carried here.Scope note
langevalsstays. It is a separate Python service (evaluators + presidio PII + topic clustering, which was deliberately migrated there). "Remove the Python NLP" meanslangwatch_nlpspecifically.Verification
go build ./cmd/service+go test ./services/nlpgo/...green.pnpm typecheckclean; updated unit tests pass; feature-parity gate passes (1549 enforced scenarios bound).helm lint+helm templateclean.Dogfood — self-hosted Go-only pod on lw-dev EKS
Deployed the Go-only distroless image to the lw-dev
langwatch-langwatch-nlpDeployment (the self-hosted helm shape) and exercised it end-to-end.Before (old pod):
nlpgo-9114f90ac4c2-arm64(fat ~1GB python), probes on/health, logs showservice=uvicorn-child+uvicornchild/manager.go+ litellm on python3.12.After (Go-only distroless, this PR's image):
/healthz/readyz/startupz— a wrong probe path would hang the rollout, so a cleansuccessfully rolled outproves the helm probe change on real infra.nlpgo_starting :5561, lifecycle services =otel+httponly,transport=in_process_dispatcher, zero uvicorn/child/litellm lines.python3.11):/healthz/readyz/startupz/go/versionall200; a non-/gopath returns the typed502go-only.POST /go/studio/execute_sync, a Studioexecute_flowwhose code node imports the curated sandbox libs and builds a pydantic model) →status: success,result: httpx=0.28.1 requests=2.34.2 pydantic=2.13.4 langwatch=True model_n=18. All four curated libs import (includinglangwatch— the dist-info fix), pydantic model built.httpx=0.28.1 requests=2.34.2 pydantic=2.13.4 langwatch=True model_n=18.The deployed-pod result is byte-identical to the saas lambda result (langwatch/langwatch-saas#570) because both artifacts install the curated sandbox from the one shared
sandbox-requirements.txtbuild stage.Code-block workflow in the real lw-dev Optimization Studio, executed against the Go-only EKS pod:
After the dogfood the lw-dev nlp Deployment was reverted to its prior image and the dev nodegroup scaled back to zero.