Skip to content

Commit 6712af1

Browse files
committed
feat(context): add index_freshness trust metadata (slice 1)
Surface commit drift, watcher pending sync, and disk-ahead signals on codemap context / MCP context so agents know when structural queries may lag the checkout. Watcher exposes debouncer state; slice 2 will extend MCP/HTTP tool responses per plan.
1 parent df31144 commit 6712af1

7 files changed

Lines changed: 417 additions & 1 deletion

File tree

docs/agents.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,8 @@ Recipe ids cited in the playbook are machine-validated in tests against the live
103103

104104
## MCP tool allowlist
105105

106+
**`context.index_freshness`** — session bootstrap includes index-level freshness metadata: `commit_drift` (HEAD ≠ `last_indexed_commit`), `pending_sync` (watcher debounce queue or in-flight reindex), optional disk-drift counts when watch is off, and a single `warning` string when agents should pause or re-index. Complements per-file `validate` / snippet `stale`. Plan: [`plans/index-freshness-trust-bundle.md`](./plans/index-freshness-trust-bundle.md).
107+
106108
**`CODEMAP_MCP_TOOLS`** — comma-separated snake_case MCP tool names. When set, only listed tools register (stderr lists the active set). Unknown names are ignored with a warning. Unset = all tools (default). **`query_batch`** registers only when listed or when unset (eval ablation).
107109

108110
Example: `CODEMAP_MCP_TOOLS=query,context,show codemap mcp --no-watch`
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Index freshness trust bundle — plan
2+
3+
> **Status:** in progress · **Priority:** agent session · **Effort:** S (~3–5 days) · **Roadmap:** [§ Index staleness surfacing](../roadmap.md#agent-session--warm-path-economics), [§ HEAD / index freshness warning](../roadmap.md#agent-session--warm-path-economics)
4+
>
5+
> **Motivator:** Agents treat MCP / HTTP / `context` output as ground truth. Today they can query during watcher debounce (disk ahead of index), after a branch switch (`last_indexed_commit``HEAD`), or with a dirty working tree when watch is off — with no signal except running `validate` manually. Wrong structural verdicts follow.
6+
7+
---
8+
9+
## Pre-locked decisions
10+
11+
| # | Decision | Source |
12+
| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
13+
| L.1 | **`index_freshness` is metadata, not a verdict** — structured fields + optional `warning` string; never `pass`/`fail`. | [Moat A](../roadmap.md#moats-load-bearing) |
14+
| L.2 | **Canonical shape in one module**`src/application/index-freshness.ts`; `context`, MCP, HTTP, and CLI all call `computeIndexFreshness()`. | Same seam as `validate-engine` / `context-engine` |
15+
| L.3 | **`pending_sync` = watcher queue OR in-flight reindex** — true when debouncer has paths **or** a targeted `--files` reindex is running. Not “SQLite mid-transaction” (writes stay transactional). | [roadmap § No split-brain](../roadmap.md#floors-v1-product-shape) |
16+
| L.4 | **Cheap vs full freshness** — every transport gets cheap signals (HEAD drift, pending sync, watch active). **Disk drift** (`getChangedFiles`) runs on `context` and opt-in full mode only (git cost). | Avoid git subprocess on every `query` row |
17+
| L.5 | **JSON tool payloads stay backward-compatible in v1 slice** — slice 1 enriches `context` only; slice 2 adds HTTP headers + MCP `_meta` wrapper without breaking array-shaped `query` results. | Agent eval / golden harness consume raw arrays today |
18+
19+
---
20+
21+
## Freshness envelope
22+
23+
```typescript
24+
interface IndexFreshness {
25+
head_commit: string | null;
26+
last_indexed_commit: string | null;
27+
commit_drift: boolean;
28+
watch_active: boolean;
29+
pending_sync: boolean;
30+
pending_paths: number;
31+
reindex_in_flight: boolean;
32+
/** Present when `include_disk_drift: true` */
33+
disk_ahead_of_index?: boolean;
34+
unindexed_change_count?: number;
35+
history_incompatible?: boolean;
36+
/** Single agent-readable line when any concern is active; null when fresh */
37+
warning: string | null;
38+
}
39+
```
40+
41+
---
42+
43+
## Shipping cadence (tracer bullets)
44+
45+
### Slice 1 — `context` + watcher state (this PR)
46+
47+
1. **`getWatchSyncState()`** on `watcher.ts` — expose debouncer pending count + reindex-in-flight flag (module-scoped; test reset hook).
48+
2. **`index-freshness.ts`**`computeIndexFreshness(db, { include_disk_drift })`.
49+
3. **`ContextEnvelope.index_freshness`**`buildContextEnvelope` calls full mode (`include_disk_drift: true`).
50+
4. **CLI `codemap context`** — inherits via `buildContextEnvelope` (no separate code path).
51+
5. **Unit tests** — freshness permutations (drift, pending, disk-ahead, history rewrite).
52+
6. **Docs** — one paragraph in [`agents.md`](../agents.md) + roadmap checkbox note when shipped.
53+
54+
**Acceptance**
55+
56+
- [ ] `codemap context --json` includes `index_freshness` with `warning` when HEAD ≠ `last_indexed_commit`
57+
- [ ] Fake watcher with pending debounce → `pending_sync: true`
58+
- [ ] Non-git fixture → `head_commit: null`, no throw
59+
60+
### Slice 2 — all MCP / HTTP tool responses
61+
62+
1. **HTTP response headers** on `POST /tool/*`: `X-Codemap-Pending-Sync`, `X-Codemap-Commit-Drift`, `X-Codemap-Warning` (when set).
63+
2. **MCP `wrapToolResult`** — for JSON tools, wrap as `{ result, index_freshness }` **or** append a second `content` block with type `text` prefixed `@codemap/index_freshness` (pick in plan-PR review; default: wrapper object for object payloads, header-equivalent second block for arrays — document in MCP instructions).
64+
3. **`/health`** — include cheap freshness when DB exists (optional).
65+
66+
### Slice 3 — stderr + MCP initialize (optional)
67+
68+
1. **`codemap mcp` / `serve` boot** — one-line stderr when commit drift detected at prime time.
69+
2. **MCP `instructions` / `codemap://mcp-instructions`** — document freshness fields + agent guidance (“if `pending_sync`, retry after debounce or call `validate`”).
70+
71+
---
72+
73+
## Dependencies
74+
75+
- [`watcher.ts`](../../src/application/watcher.ts) — debouncer + `isWatchActive()`
76+
- [`index-engine.ts`](../../src/application/index-engine.ts)`getChangedFiles`, `getCurrentCommit`
77+
- [`context-engine.ts`](../../src/application/context-engine.ts) — envelope builder
78+
- [`validate-engine.ts`](../../src/application/validate-engine.ts) — conceptual sibling (per-file staleness vs index-level)
79+
80+
---
81+
82+
## Out of scope
83+
84+
- Blocking queries when stale (agents decide).
85+
- Changing incremental indexing semantics ([No split-brain floor](../roadmap.md#floors-v1-product-shape)).
86+
- Shared daemon / multi-session lifecycle (separate roadmap item).

src/application/context-engine.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import { getMeta, SCHEMA_VERSION } from "../db";
22
import type { CodemapDatabase } from "../db";
33
import { CODEMAP_VERSION } from "../version";
4+
import { computeIndexFreshness } from "./index-freshness";
5+
import type { IndexFreshness } from "./index-freshness";
46
import { QUERY_RECIPES } from "./query-recipes";
57

68
/**
@@ -32,6 +34,7 @@ export interface ContextEnvelope {
3234
content: string;
3335
}[];
3436
recipes: { id: string; description: string }[];
37+
index_freshness: IndexFreshness;
3538
intent?: {
3639
input: string;
3740
classified_as: string;
@@ -136,6 +139,7 @@ export function buildContextEnvelope(
136139
id,
137140
description: meta.description,
138141
})),
142+
index_freshness: computeIndexFreshness(db, { include_disk_drift: true }),
139143
};
140144

141145
if (!opts.compact) {
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
import { afterEach, beforeEach, describe, expect, it, spyOn } from "bun:test";
2+
import { mkdirSync, mkdtempSync, rmSync } from "node:fs";
3+
import { tmpdir } from "node:os";
4+
import { join } from "node:path";
5+
6+
import { resolveCodemapConfig } from "../config";
7+
import { closeDb, createTables, openDb, setMeta } from "../db";
8+
import { initCodemap } from "../runtime";
9+
import { buildContextEnvelope } from "./context-engine";
10+
import * as indexEngine from "./index-engine";
11+
import { computeIndexFreshness } from "./index-freshness";
12+
import { _resetWatchStateForTests, runWatchLoop } from "./watcher";
13+
import type { WatchBackend } from "./watcher";
14+
15+
let benchDir: string;
16+
17+
beforeEach(() => {
18+
benchDir = mkdtempSync(join(tmpdir(), "index-freshness-"));
19+
mkdirSync(join(benchDir, ".codemap"), { recursive: true });
20+
initCodemap(resolveCodemapConfig(benchDir, undefined));
21+
_resetWatchStateForTests();
22+
});
23+
24+
afterEach(() => {
25+
rmSync(benchDir, { recursive: true, force: true });
26+
_resetWatchStateForTests();
27+
});
28+
29+
function withEmptyDb<T>(fn: (db: ReturnType<typeof openDb>) => T): T {
30+
const db = openDb();
31+
try {
32+
createTables(db);
33+
return fn(db);
34+
} finally {
35+
closeDb(db);
36+
}
37+
}
38+
39+
function fakeBackend(): WatchBackend & {
40+
fire: (kind: "add" | "change" | "unlink", abs: string) => void;
41+
} {
42+
let onEvent:
43+
| ((k: "add" | "change" | "unlink", p: string) => void)
44+
| undefined;
45+
return {
46+
start(opts) {
47+
onEvent = opts.onEvent;
48+
},
49+
async stop() {},
50+
fire(kind, abs) {
51+
onEvent?.(kind, abs);
52+
},
53+
};
54+
}
55+
56+
describe("computeIndexFreshness", () => {
57+
it("reports no warning when HEAD matches last_indexed_commit", () => {
58+
const head = "abc123def456789012345678901234567890abcd";
59+
const revParse = spyOn(indexEngine, "getCurrentCommit").mockReturnValue(
60+
head,
61+
);
62+
63+
try {
64+
withEmptyDb((db) => {
65+
setMeta(db, "last_indexed_commit", head);
66+
const f = computeIndexFreshness(db);
67+
expect(f.commit_drift).toBe(false);
68+
expect(f.warning).toBeNull();
69+
expect(f.head_commit).toBe(head);
70+
});
71+
} finally {
72+
revParse.mockRestore();
73+
}
74+
});
75+
76+
it("warns on commit drift", () => {
77+
const revParse = spyOn(indexEngine, "getCurrentCommit").mockReturnValue(
78+
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
79+
);
80+
81+
try {
82+
withEmptyDb((db) => {
83+
setMeta(
84+
db,
85+
"last_indexed_commit",
86+
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
87+
);
88+
const f = computeIndexFreshness(db);
89+
expect(f.commit_drift).toBe(true);
90+
expect(f.warning).toContain("HEAD is bbbbbbb");
91+
});
92+
} finally {
93+
revParse.mockRestore();
94+
}
95+
});
96+
97+
it("reports pending_sync when the watcher debouncer has queued paths", async () => {
98+
spyOn(indexEngine, "getCurrentCommit").mockReturnValue("");
99+
mkdirSync(join(benchDir, "src"), { recursive: true });
100+
const backend = fakeBackend();
101+
const handle = runWatchLoop({
102+
root: benchDir,
103+
excludeDirNames: new Set(["node_modules", ".git", "dist"]),
104+
onChange: () => {},
105+
debounceMs: 60_000,
106+
backend,
107+
});
108+
109+
backend.fire("change", join(benchDir, "src/a.ts"));
110+
111+
withEmptyDb((db) => {
112+
const f = computeIndexFreshness(db);
113+
expect(f.pending_sync).toBe(true);
114+
expect(f.pending_paths).toBe(1);
115+
expect(f.warning).toContain("pending");
116+
});
117+
118+
await handle.stop();
119+
});
120+
});
121+
122+
describe("buildContextEnvelope", () => {
123+
it("includes index_freshness with disk drift opt-in", () => {
124+
const head = "cccccccccccccccccccccccccccccccccccccccc";
125+
const revParse = spyOn(indexEngine, "getCurrentCommit").mockReturnValue(
126+
head,
127+
);
128+
const changedFiles = spyOn(indexEngine, "getChangedFiles").mockReturnValue({
129+
changed: [],
130+
deleted: [],
131+
existingPaths: new Set(),
132+
sourceCache: new Map(),
133+
existingHashes: new Map(),
134+
});
135+
136+
try {
137+
withEmptyDb((db) => {
138+
setMeta(db, "last_indexed_commit", head);
139+
const envelope = buildContextEnvelope(db, benchDir, {
140+
compact: true,
141+
intent: null,
142+
});
143+
expect(envelope.index_freshness.head_commit).toBe(head);
144+
expect(envelope.index_freshness.disk_ahead_of_index).toBe(false);
145+
expect(envelope.index_freshness.warning).toBeNull();
146+
});
147+
} finally {
148+
revParse.mockRestore();
149+
changedFiles.mockRestore();
150+
}
151+
});
152+
});

0 commit comments

Comments
 (0)