Purpose: Regression-test Codemap internals by comparing codemap query output to checked-in expectations (or subset matchers) on fixed corpora — not an LLM-in-the-loop eval. Latency / tokens vs scanning: benchmark.md.
Operational docs: CONTRIBUTING § Golden queries · benchmark § Fixtures · benchmark § Agent eval harness (agent-eval harness — probe + live — reuses scenarios via goldenId) · Runner: scripts/query-golden.ts · Setup: scripts/query-golden/run-setup.ts · Schema: scripts/query-golden/schema.ts
| Goal | How scenarios help |
|---|---|
| Catch regressions | Parser or schema drift → JSON diff vs golden |
| Encode good answers | Human-reviewed rows for representative queries |
| Stress realistic size | Optional second corpus beyond fixtures/minimal |
| Stay deterministic | Assertions on query output, not model prose |
- Chat / SSE / auth harnesses — out of scope here
- Proving agents follow rules — measure in the IDE or another project
- Replacing
src/benchmark.ts— that stays SQL vs glob/read time; goldens add correctness snapshots
| Piece | Role |
|---|---|
fixtures/minimal/ |
Tier A corpus; stable for CI |
scripts/agent-eval/ |
Tier A agent-eval harness (test:agent-eval; reuses golden goldenId) |
src/benchmark.ts |
Speed comparison (not golden row equality) |
bun test |
Unit tests for parsers, CLI, DB |
CODEMAP_ROOT / --root |
Index any tree; Tier B uses env + optional gitignore |
We do not commit another product’s source tree, paths, business strings, or golden JSON derived from a private app (or any repo we do not own and license for redistribution).
| Safe to commit here | Not committed here |
|---|---|
fixtures/minimal/ (trees we control) |
Clones of private apps |
Generic SQL / --recipe ids |
App-specific path literals in assets |
| Goldens from our fixtures only | Snapshots keyed to proprietary names |
Abstract prompt text (intent labels) |
Verbatim customer prompts |
Tier B: Point CODEMAP_ROOT at a local clone; goldens for that tree stay gitignored (or private automation) — see .gitignore. For agent-eval on external fixtures, see benchmark § Agent eval harness and .github/workflows/agent-eval-external.yml.
| Tier | Corpus | When | Purpose |
|---|---|---|---|
| A (in-repo test bench) | fixtures/minimal + fixtures/golden/ |
Every PR / bun run check |
Canonical for Codemap development — see fixtures/README.md |
| B (consumer-only) | Local path via CODEMAP_* |
Private app validation | Goldens gitignored — not required to develop Codemap |
| Bench growth | fixtures/minimal + CAPABILITIES.json |
Shipped (Phases 1–3) | testing-coverage.md, fixtures/README.md; optional scale: roadmap.md (in-repo test bench scale) |
Scenarios live in fixtures/golden/scenarios.json (Tier A) or optional scenarios.external.json / example (Tier B). The file may be a bare array of scenarios (legacy) or an object { "setup": [...], "scenarios": [...] }. Optional top-level setup runs once after index, before scenarios — today ingest-coverage, clear-coverage, and seed-file-churn (see run-setup.ts); missing coverage files are skipped with a warning. Per-scenario preSetup runs after global setup (global setup restores after the scenario when preSetup mutates the index). Each scenario has id, sql or recipe, optional match (exact, minRows, everyRowContains, everyRowFieldEquals), optional budgetMs. Goldens: fixtures/golden/minimal/*.json etc. Refresh: bun scripts/query-golden.ts --update.
Prompts in JSON are intent labels, not pasted chat logs — pair with queries whose literals come from fixture-owned data (see fixtures/qa/prompts.external.template.md for optional chat QA).
Some bundled recipes add optional reason (TEXT) and evidence_json (TEXT, JSON array) columns on each row — factual detection path for agents, not engine verdicts (Moat A — not pass/fail verdicts). Bounded subqueries cap evidence at three hops; list caps append {"truncated":true}. unimported-exports reasons: no_direct_import, reexport_chain_possible, unresolved_import_blind_spot. Goldens assert these columns when the recipe ships evidence (boundary-violations, deprecated-symbols, unimported-exports).
high-crap-score adds coverage_source (measured | estimated) and effective_coverage_pct on each row — measured when coverage has a matching symbol row after ingest-coverage; otherwise graph-estimated tiers from test reachability. Goldens assert coverage_source when the recipe ships coverage semantics (high-crap-score); measured override is covered by scripts/high-crap-score-measured.test.mjs.
coverage-confirmed-dead adds confidence (high | medium) on each row — high when static dead and ingested coverage_pct = 0; medium when static dead but the symbol has no ingested coverage row. Also reason, caller_count. Goldens: coverage-confirmed-dead (post-ingest mix) and coverage-confirmed-dead-no-ingest (preSetup: clear-coverage, everyRowFieldEquals on confidence: medium).
churn-complexity-hotspots ranks indexed files or symbols by git churn × cyclomatic complexity. File grain (default): file_path, max_complexity, weighted_commits, commit_count, churn_trend, hotspot_score, hotspot_score_normalized. Symbol grain (by_symbol=true): per-symbol name, kind, line_start, cyclomatic_complexity plus file churn fields. Goldens: churn-complexity-hotspots, churn-complexity-hotspots-by-symbol (fixture churn seeded via setup step seed-file-churn).
duplicates returns one row per function-shaped symbol in a body_hash collision group: name, kind, file_path, line_start, line_end, body_hash, body_line_count, duplicate_count (in-scope group size after path_prefix / min_body_lines). Substrate column symbols.body_hash is populated at index for function-shaped symbols (function, method, getter, setter) when body_line_count >= 2. Goldens: duplicates (includes src/bench/duplicate-body-{a,b}.ts pair). False positives possible when unrelated functions share control-flow skeleton or sync vs async/generator bodies match — triage with snippet. Recipe caps at 50 rows (no truncation marker).
| Area | State |
|---|---|
| Tier A runner + CI | bun run test:golden + bun run test:agent-eval in check (CI Test job runs both; agent-eval reuses golden index when present) |
| Tier A scenario coverage | Scenarios cover core parser/schema surfaces, every bundled recipe (templates/recipes/*.sql), and SQL pin-down for substrate tables — see testing-coverage.md. Guard: scripts/query-golden-coverage-matrix.test.mjs. Inventory: scenarios.json. |
| Tier B external + schema | test:golden:external, Zod in scripts/query-golden/schema.ts |
| Subset matchers + budgets | match, budgetMs, --strict-budget |
| Optional CI for public corpus | Deferred — roadmap § Backlog |
- benchmark.md — speed methodology, Tier B, fixtures
- architecture.md — schema, parsers
- roadmap.md — backlog