Skip to content

Commit 5be25cd

Browse files
denfryclaude
andcommitted
docs: rebuild README around real output and the public benchmark; audit and launch kit
- README: one-sentence proposition, agents list, terminal card rendered from a real Flask transcript, 60-second try-it, why-this-exists with grep / repo-map / cloud comparison, evidence section with the public baseline chart, then architecture, privacy, languages, status, contributing, roadmap. - BENCHMARKS.md leads with the public baseline run and withdraws the private 55k LOC Java headline (asymmetric accounting, not reproducible); COMPARISON.md and the older result logs point at the new run. - docs/COMMUNITY_AUDIT.md: P0-P3 findings with status and before/after scores. - docs/COMMUNITY_LAUNCH.md: HN / Reddit / X / Discord / release drafts using only logged numbers, a day-0 to month-1 sequence, and the feedback loop; replaces the stale docs/SEO.md. - examples/demo-project (a project that did not exist) removed in favour of examples/demo; package keywords extended (code-graph, impact-analysis, mcp-server, claude-code-plugin). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDogWFhCH69TMddDcrtwHL
1 parent 87146cf commit 5be25cd

10 files changed

Lines changed: 998 additions & 401 deletions

CHANGELOG.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,69 @@ All notable changes to this project are documented here. The format is based on
66

77
## [Unreleased]
88

9+
### Added
10+
11+
- **Public baseline benchmark.** `tests/eval/run_baselines.py` compares the index with
12+
a disciplined `rg` + 80-line-window agent and with repo-map-style context on
13+
Flask, Gson and Fastify at pinned commits, using git-derived ground truth, one
14+
tokenizer on every side, and paired bootstrap / permutation significance. The
15+
logged run is committed under `tests/eval/results/`: pooled over 450 queries,
16+
hit@3 0.547 vs 0.304 and MRR 0.456 vs 0.263 (p < 0.001) at 3.8k vs 3.5k context
17+
tokens per query. `scripts/gen_benchmark_chart.py` renders the chart from the log.
18+
- **Reproducible demo.** `examples/demo/run_demo.sh|.ps1` run Find / Trace / Predict
19+
on `pallets/flask` at a pinned commit; `EXPECTED_OUTPUT.md` is the captured run and
20+
`assets/demo-terminal.svg` is rendered from it (`scripts/gen_terminal_svg.py`).
21+
`docs/DEMO.md` documents GIF/video recording.
22+
- **Release and CI gates.** `scripts/check_versions.py` (package, plugin manifest,
23+
lock tag, skill stamps and changelog must agree), `scripts/check_links.py`
24+
(relative Markdown links must resolve), `scripts/release_notes.py` (GitHub release
25+
body comes from the changelog section and an empty section fails the release), and
26+
a `package` CI job that builds, `twine check`s and runs the clean-venv install smoke
27+
on every pull request.
28+
- **Contributor onboarding.** `docs/DEVELOPMENT.md`, rewritten `CONTRIBUTING.md`,
29+
benchmark-report and language-support issue templates, `SUPPORT.md`, a labels
30+
manifest, and `docs/COMMUNITY_AUDIT.md` / `docs/COMMUNITY_LAUNCH.md`.
31+
32+
### Changed
33+
34+
- **Read plan is bounded.** `recommended_reads` entries are capped at
35+
`retrieval.max_read_lines` (default 120) and carry `truncated: true` plus
36+
`line_end_full` when capped. A symbol-aligned chunk can be a whole class; on the
37+
public baseline benchmark the uncapped read plan cost 6.8k tokens per query against
38+
3.5k for grep, the capped one 3.8k, with identical ranking quality. Set the option to
39+
0 for the previous behaviour.
40+
- **README, docs and examples show real output only.** Fabricated tables (an
41+
`AuthService.ts` example with a `Score` column, an invented `doctor` transcript,
42+
the mock-up `assets/demo.png`) are replaced by output captured on Flask.
43+
Duplicate pages (`DATABASE_SCHEMA`, `RETRIEVAL_PIPELINE`, `docs/SECURITY`) are
44+
redirect stubs; `SCHEMA.md` now matches `storage/schema.sql`.
45+
- **Benchmark headline.** The "13× fewer tokens on a 55k LOC Java repo" figure is
46+
withdrawn: the repository is private and the accounting was asymmetric. The
47+
defensible claim is the public baseline run above.
48+
- `SECURITY.md` points to GitHub private vulnerability reporting and states the
49+
supported line as 1.9.x.
50+
51+
### Fixed
52+
53+
- **MCP server did not start on mcp 2.x.** The SDK renamed `FastMCP` to
54+
`MCPServer` and removed the old import path, so `codebase-index mcp` reported
55+
"needs the optional extra" even with the extra installed, and the MCP tests
56+
skipped silently in CI because the skip guard wrapped our own module's import.
57+
The server now imports `MCPServer` first and falls back to `FastMCP` on 1.x
58+
(verified against mcp 1.29 and 2.1); the tests skip only when the SDK itself is
59+
missing.
60+
- **`codebase-index mcp --root <repo>` was rejected.** Every client template
61+
used that form, but `--root` was only a global option (`codebase-index --root
62+
<repo> mcp`). The subcommand now accepts it as well.
63+
- **Plugin wrappers refused four documented commands.** `bin/cbx` and `bin/cbx.ps1`
64+
whitelisted ten subcommands while the skill allowed fourteen; `architecture`,
65+
`diff-impact`, `path` and `describe` now work from the plugin. A parity test pins
66+
the two lists together.
67+
- **Benchmark leakage.** `gen_queries` documented that changelog-style files were
68+
excluded from the evaluation corpus, but the harness never applied the list, and
69+
Flask-style `CHANGES.rst` was not covered. Both are fixed; `CHANGES*`, `HISTORY*`,
70+
`NEWS*` and `RELEASE_NOTES*` are refused as answers and excluded from the corpus.
71+
972
## [1.9.0] - 2026-09-02
1073

1174
### Added

0 commit comments

Comments
 (0)