Commit 560390b
authored
feat(fts5+mermaid): full FTS5 + Mermaid plan implementation (slices 1-7) + docs audit (#69)
* feat(fts5): Slice 1 — substrate (schema + Zod + CLI flag + indexer write path)
First slice of the FTS5+Mermaid plan implementation per
docs/plans/fts5-mermaid.md. Ships the FTS5 substrate end-to-end so
follow-up slices (demo recipe, Mermaid formatter, MCP/HTTP plumbing)
have something to layer on.
Schema (per Q1):
- SCHEMA_VERSION 6 → 7 (forces dropAll on first upgrade).
- New `source_fts` virtual table — columns (file_path UNINDEXED,
content), tokeniser 'porter unicode61'. Always created; near-zero
space when empty.
- Helpers: upsertSourceFts (DELETE+INSERT — FTS5 doesn't accept
INSERT OR REPLACE on virtual tables), deleteSourceFts (manual
mirror of FK CASCADE since virtual tables can't be FK targets),
clearSourceFts.
Config (per Q6):
- New `fts5: boolean` field in Zod schema (default false, optional).
- `ResolvedCodemapConfig.fts5` resolved from config + CLI; CLI wins
per `--root` / `--state-dir` precedent. Logs stderr override line
when CLI overrides config.
- New `getFts5Enabled()` runtime accessor.
CLI:
- `--with-fts` flag in parseBootstrapArgs; threaded through
bootstrapCodemap → resolveCodemapConfig.fts5Cli; rest also pushed
into validateIndexModeArgs allowlist.
Worker plumbing (per Q2):
- WorkerInput.fts5Enabled propagated from worker-pool via
getFts5Enabled().
- ParsedFile.content optional; worker tees source into it only when
fts5Enabled (zero serialization cost on default-OFF path).
Indexer (per Q2):
- insertParsedResults writes source_fts in same transaction as files
row insert.
- Single-threaded path (parse-on-main, used in incremental
per-relPath loop) also calls upsertSourceFts gated on
getFts5Enabled().
- deleteFilesFromIndex mirrors DELETE to source_fts.
- indexFiles fullRebuild path re-seeds meta (fts5_enabled,
schema_version) after dropAll wipes meta.
Toggle-change auto-detect (per Q3):
- run-index reads meta.fts5_enabled; mismatch with current resolved
config upgrades incremental → full and logs stderr line. First-run
(no value) seeds silently. Already-full mode skips the upgrade
message but still syncs.
Verification:
- bun run typecheck passes.
- bun test: 746 pass, 0 fail.
- End-to-end smoke test (/tmp/fts5-smoke-test):
- Index without --with-fts → source_fts empty ✓
- Re-index with --with-fts → toggle-change auto-detect upgraded
incremental to full → MATCH 'TODO' returns the seeded file ✓
Slice 1 deliberately does NOT include:
- Telemetry on first FTS5 populate (Q7) — defer.
- Mermaid formatter (slices 3-5).
- Demo recipe text-in-deprecated-functions (slice 2).
- Docs/agents lockstep (slice 6) — defer until the demo recipe lands
so the rule update names a real recipe.
- Patch changeset (slice 7).
Files changed: 9 (db.ts, config.ts, runtime.ts, bootstrap.ts,
bootstrap-codemap.ts, main.ts, cmd-index.ts, parsed-types.ts,
parse-worker-core.ts, worker-pool.ts, index-engine.ts, run-index.ts).
* feat(fts5+mermaid): slices 2-7 — recipe + Mermaid + plumbing + telemetry + lockstep
Completes the FTS5+Mermaid plan (docs/plans/fts5-mermaid.md). Builds on
slice 1's substrate.
Slice 2 — demo recipe text-in-deprecated-functions:
Bundled recipe: @deprecated functions in files containing TODO/FIXME/
HACK markers AND coverage <50%. Demonstrates FTS5 ⨯ symbols ⨯ coverage
JOIN composability that ripgrep can't match. Returns empty when FTS5 is
off (source_fts empty). Action template: review-cleanup-priority.
Slices 3+4 — Mermaid formatter with bounded-input contract:
- formatMermaid(rows, opts) in output-formatters.ts. Renders
{from, to, label?, kind?} as flowchart LR. Reuses existing formatter
plumbing pattern from SARIF / annotations.
- MERMAID_MAX_EDGES = 50 hard-coded const (Q4). Auto-truncation
explicitly out of scope (would be a verdict masquerading as output
mode, violating moat A).
- Reject error names recipe + count + scoping knobs (LIMIT / --via /
WHERE) so agent knows how to scope.
Slice 5 — MCP + HTTP plumbing:
- formatEnum gains "mermaid"; QueryArgs / QueryRecipeArgs format unions
extended.
- tool-handlers.runFormattedQuery branches on "mermaid" → wraps
formatMermaid in try/catch (bounded-input rejection becomes a
structured ToolResult error).
- MCP wrapToolResult needs no change — generic non-JSON passthrough
handles "mermaid" same as "sarif" / "annotations".
- HTTP writeToolResult needs no change — fall-through "text/plain;
charset=utf-8" applies to mermaid (not sarif+json).
- Tool descriptions in mcp-server.ts updated to mention "mermaid"
format.
Slice 6 — agent rule + skill lockstep (Rule 10):
Both templates/agents/ AND .agents/ codemap rule + skill updated:
- --format mermaid example row in CLI table
- --with-fts row in CLI table
- text-in-deprecated-functions in trigger pattern + recipe id list
Slice 7 — Q7 telemetry + patch changeset:
- Stderr line on first FTS5 populate: [fts5] source_fts populated:
<N> files / <X> KB (uncompressed content). Cheap (single
SUM(length(content)) on source_fts at end of full reindex); only
fires when fts5 just became populated (fts5WasEmpty pre-check).
- Patch changeset (.changeset/fts5-mermaid.md) per pre-v1 lesson:
additive feature, default-OFF, behaviour-preserving for existing
users.
Verification end-to-end:
- bun test: 754 pass, 0 fail (8 new mermaid formatter tests)
- bun run check passes (format, lint, typecheck, 23/23 golden queries)
- Smoke test with @deprecated function + TODO comment:
* codemap --with-fts --full → "[fts5] source_fts populated: 2
files / 138 B (uncompressed content)" ✓
* query --recipe text-in-deprecated-functions → returns the
deprecated function ✓
* query --format mermaid 'SELECT from_path AS "from", to_path AS
"to" FROM dependencies LIMIT 50' → renders flowchart LR ✓
* query --format mermaid 'SELECT from_path AS "from", to_path AS
"to" FROM dependencies' (unbounded) → rejects with scope-
suggestion error ✓
Plan slices all complete. PR #69 description gets refreshed in a
follow-up commit (this commit covers the impl — the doc-audit pass on
docs/ comes next per user request).
* docs: audit and lift FTS5+Mermaid decisions to canonical homes
Per docs-governance lifecycle (docs/README.md Rule 3 + Lifecycle table:
"Plan: Deleted when work ships"):
LIFTED (decisions promoted from docs/plans/fts5-mermaid.md to canonical
homes):
- docs/architecture.md
- Schema version 6 → 7
- meta table description: added fts5_enabled key
- New "source_fts" table section under § Schema, after meta —
documents tokenizer, file_path UNINDEXED, opt-in toggle precedence,
auto-detect, telemetry, and the bundled demo recipe
- application/ engines list: output-formatters.ts now mentions
Mermaid alongside SARIF + GH annotations
- docs/glossary.md
- New "source_fts (FTS5 virtual table) / --with-fts / opt-in
full-text" entry — covers schema shape, toggle precedence,
auto-detect, telemetry, default-OFF rationale, the JOIN
composability the FTS5 / ripgrep distinction tracks
- New "--format mermaid / formatMermaid / MERMAID_MAX_EDGES" entry
— covers the {from, to, label?, kind?} input contract, bounded-
input rejection (50 edges), why auto-truncation is out of scope
- docs/roadmap.md
- Backlog: "Optional FTS5 for opt-in full-text" line removed (work
has shipped per Rule 2 — backlog items move OUT when shipped)
- Non-goals: "Full-text search across all file bodies — use ripgrep
/ IDE / opt-in FTS5 (see backlog)" REWORDED to "Full-text search
default-on" non-goal — FTS5 ships per --with-fts, but default-on
is still out of scope until v2 size-tax measurements
DELETED:
- docs/plans/fts5-mermaid.md (per Rule 3 + Lifecycle: plans deleted
when work ships; decisions live in canonical homes above)
- docs/README.md File Ownership row updated to drop fts5-mermaid.md
from in-flight plans list (still has c9-plugin-layer.md)
Audit method: grep [Ff]allow / FTS5 / source_fts / --with-fts /
--format mermaid across docs/ — categorised hits as documentation
(lift to architecture.md / glossary.md), historical (research notes
keep their content; closed records like fallow.md stay closed), or
plan-residue (delete fts5-mermaid.md per lifecycle).
Verification:
- Schema version assertion in architecture.md matches SCHEMA_VERSION
in db.ts (7).
- meta.fts5_enabled key documented matches META_FTS5_ENABLED_KEY in
db.ts.
- source_fts column shape (file_path UNINDEXED, content) +
tokenizer ('porter unicode61') match the actual CREATE VIRTUAL
TABLE in db.ts createTables.
- --with-fts CLI flag documented matches parseBootstrapArgs +
validateIndexModeArgs allowlists.
- --format mermaid documented matches OUTPUT_FORMATS + formatEnum in
cmd-query.ts + tool-handlers.ts.
- 50-edge ceiling matches MERMAID_MAX_EDGES in output-formatters.ts.
- bun run check passes (format, lint, typecheck, 23/23 golden
queries).1 parent 73ffa86 commit 560390b
29 files changed
Lines changed: 634 additions & 230 deletions
File tree
- .agents
- rules
- skills/codemap
- .changeset
- docs
- plans
- src
- application
- cli
- templates
- agents
- rules
- skills/codemap
- recipes
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
94 | 96 | | |
95 | 97 | | |
96 | 98 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
120 | 123 | | |
121 | 124 | | |
122 | 125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
0 commit comments