Skip to content

Commit 7f158e9

Browse files
authored
Merge branch 'main' into fix/index-worker-explicit-memory-budget
2 parents 0e91b3d + ae5de7f commit 7f158e9

135 files changed

Lines changed: 18120 additions & 1000 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,7 @@ internal/cbm/vendored/grammars/**/scanner.c linguist-generated=true diff=false
1111
# the git hooks are extensionless and need explicit entries. (#1272)
1212
scripts/git-hooks/commit-msg text eol=lf
1313
scripts/hooks/pre-commit text eol=lf
14+
15+
# Chialisp grammar corpus: one case asserts CRLF comment termination, so this
16+
# file must keep its exact bytes on every platform (no eol conversion). (#919)
17+
tools/tree-sitter-chialisp/test/corpus/comments.txt -text

.github/workflows/_build.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ jobs:
9797
9898
- name: Attest release candidate provenance
9999
if: ${{ inputs.attest }}
100-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
100+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
101101
with:
102102
subject-path: |
103103
release-candidates/${{ matrix.goos }}-${{ matrix.goarch }}/candidate-provenance.tsv
@@ -155,7 +155,7 @@ jobs:
155155
156156
- name: Attest release candidate provenance
157157
if: ${{ inputs.attest }}
158-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
158+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
159159
with:
160160
subject-path: |
161161
release-candidates/windows-amd64/candidate-provenance.tsv
@@ -217,7 +217,7 @@ jobs:
217217
218218
- name: Attest release candidate provenance
219219
if: ${{ inputs.attest }}
220-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
220+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
221221
with:
222222
subject-path: |
223223
release-candidates/windows-arm64/candidate-provenance.tsv
@@ -280,7 +280,7 @@ jobs:
280280
281281
- name: Attest release candidate provenance
282282
if: ${{ inputs.attest }}
283-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
283+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
284284
with:
285285
subject-path: |
286286
release-candidates/linux-${{ matrix.arch }}-portable/candidate-provenance.tsv
@@ -432,7 +432,7 @@ jobs:
432432
433433
- name: Attest selected containers and decision evidence
434434
if: ${{ inputs.attest }}
435-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
435+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
436436
with:
437437
subject-path: |
438438
release-containers/**/*.tar.gz

.github/workflows/codeql.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
run: sudo apt-get update && sudo apt-get install -y zlib1g-dev
3030

3131
- name: Initialize CodeQL
32-
uses: github/codeql-action/init@f205ea1c3313d32999d8d6a48b4f6530d4437b38 # v4.37.4
32+
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
3333
with:
3434
languages: c-cpp
3535
build-mode: manual
@@ -38,6 +38,6 @@ jobs:
3838
run: scripts/build.sh
3939

4040
- name: Perform CodeQL Analysis
41-
uses: github/codeql-action/analyze@f205ea1c3313d32999d8d6a48b4f6530d4437b38 # v4.37.4
41+
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
4242
with:
4343
category: "/language:c-cpp"

.github/workflows/release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,7 +202,7 @@ jobs:
202202
run: scripts/ci/append-legacy-alias-checksums.sh checksums.txt
203203

204204
- name: Attest checksum provenance
205-
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
205+
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
206206
with:
207207
subject-path: checksums.txt
208208

.github/workflows/scorecard.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,6 @@ jobs:
2727
publish_results: true
2828

2929
- name: Upload SARIF results
30-
uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
30+
uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
3131
with:
3232
sarif_file: results.sarif

Makefile.cbm

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,7 @@ PIPELINE_SRCS = \
354354
src/pipeline/pass_similarity.c \
355355
src/pipeline/pass_semantic_edges.c \
356356
src/pipeline/pass_complexity.c \
357+
src/pipeline/pass_importance.c \
357358
src/pipeline/pass_cross_repo.c \
358359
src/pipeline/artifact.c \
359360
src/pipeline/pass_ensemble_routing.c \
@@ -560,7 +561,7 @@ TEST_DISCOVER_SRCS = \
560561

561562
TEST_GRAPH_BUFFER_SRCS = tests/test_graph_buffer.c
562563

563-
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
564+
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
564565

565566
TEST_WATCHER_SRCS = tests/test_watcher.c
566567

README.md

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
55
[![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml)
66
[![Tests](https://img.shields.io/badge/tests-6768_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp)
7-
[![Languages](https://img.shields.io/badge/languages-161-orange)](https://github.com/DeusData/codebase-memory-mcp)
7+
[![Languages](https://img.shields.io/badge/languages-162-orange)](https://github.com/DeusData/codebase-memory-mcp)
88
[![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-10_languages-blue)](#hybrid-lsp)
99
[![Agents](https://img.shields.io/badge/agent_surfaces-45-purple)](https://github.com/DeusData/codebase-memory-mcp)
1010
[![Pure C](https://img.shields.io/badge/pure_C-no_language_runtime-blue)](https://github.com/DeusData/codebase-memory-mcp)
@@ -16,7 +16,7 @@
1616

1717
**The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a native executable with a small verified runtime-asset set for macOS, Linux, and Windows — download, run `install`, done.
1818

19-
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 161 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
19+
High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 162 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 15 MCP tools. No language runtime, hosted service, or API key. Plug and play across 45 supported automatic/conditional client surfaces.
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
@@ -32,7 +32,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
3232

3333
- **Extreme indexing speed** — Linux kernel (28M LOC, 75K files) in 3 minutes. RAM-first pipeline: LZ4 compression, in-memory SQLite, fused Aho-Corasick pattern matching. Memory released after indexing.
3434
- **Plug and play** — native executable plus authenticated release-owned assets for macOS (arm64/amd64), Linux (arm64/amd64), and Windows (amd64). The native install needs no Docker, language runtime, or API keys. Download → `install` → restart agent → done.
35-
- **161 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
35+
- **162 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks.
3636
- **120x fewer tokens** — 5 structural queries: ~3,400 tokens vs ~412,000 via file-by-file search. One graph query replaces dozens of grep/read cycles.
3737
- **45 supported automatic/conditional client surfaces**`install` configures detected clients and safely activates conditional clients only when their documented platform, marker, or explicit existing config path is present. See [Multi-Agent Support](#multi-agent-support) for the complete matrix and manual/UI-only boundaries.
3838
- **Built-in graph visualization** — 3D interactive UI at `localhost:9749`, served from the binary itself.
@@ -289,6 +289,8 @@ Benchmarked on Apple M3 Pro:
289289

290290
**Token efficiency**: Five structural queries consumed ~3,400 tokens via codebase-memory-mcp versus ~412,000 tokens via file-by-file grep exploration — a **99.2% reduction**.
291291

292+
To measure comparable quality, latency, and agent-efficiency metrics on your own workload, see [Measuring quality, latency, and agent savings](docs/MEASURING_SAVINGS.md). Exact reproduction of the figures above requires the original inputs and raw artifacts.
293+
292294
## Troubleshooting & Diagnostics
293295

294296
codebase-memory-mcp runs **100% locally and collects no telemetry** — your code, queries, environment, and usage never leave your machine. That privacy guarantee also means that when you hit something we can't reproduce on our side (a slow memory climb over hours, a performance regression, a leak that only appears after days of real use), **we have no data at all unless you choose to send it.** Here is how to capture it yourself.
@@ -482,11 +484,11 @@ overwrite user-modified agents.
482484

483485
| Agent | Activation | MCP config | Durable context / augmentation |
484486
|-------|------------|------------|--------------------------------|
485-
| Claude Code | Detected | `~/.claude.json` | Skill + three exact-tool graph agents; `SessionStart`, `SubagentStart`, non-blocking `PreToolUse` for `Grep`/`Glob`, and post-`Read` coverage |
487+
| Claude Code | Detected | `~/.claude.json` | Skill + three exact-tool graph agents; `SessionStart`, `SubagentStart`, non-blocking `PreToolUse` for `Grep`/`Glob`/`Bash`, and post-`Read` coverage |
486488
| Codex CLI | Detected | `$CODEX_HOME/config.toml` | `AGENTS.md`, skill, three read-only agents; `SessionStart` + `SubagentStart` |
487489
| Gemini CLI | Detected | `.gemini/settings.json` | `GEMINI.md`, three explicit read/graph-tool subagents; `BeforeTool`, `AfterTool` `read_file` coverage, and `SessionStart` |
488490
| Zed | Detected | platform `settings.json` (JSONC) | `AGENTS.md` + shared skill |
489-
| OpenCode | Detected | `$OPENCODE_CONFIG` or resolved global config | `AGENTS.md`, skill, three deny-by-default read-only agents |
491+
| OpenCode | Detected | `$OPENCODE_CONFIG` or resolved global config | `AGENTS.md`, skill, three deny-by-default read-only agents; plugin adds grep/glob graph lookup, post-`read` coverage, first-tool-result session context, and post-compaction reinjection |
490492
| Antigravity | Detected | `.gemini/config/mcp_config.json` | `.gemini/GEMINI.md` |
491493
| Aider | Detected || `CONVENTIONS.md` via `.aider.conf.yml` |
492494
| KiloCode | Detected | `.config/kilo/kilo.jsonc` | Rule + three graph-tool subagents with deny-by-default permissions |
@@ -531,7 +533,7 @@ overwrite user-modified agents.
531533
### Sessions, compaction, and subagents
532534

533535
Hooks installed by this project are fail-open and context-only. Claude Code's
534-
`PreToolUse` observes `Grep`/`Glob` and injects matching graph symbols as
536+
`PreToolUse` observes `Grep`/`Glob`/`Bash` and injects matching graph symbols as
535537
`additionalContext`; `PostToolUse` on `Read` adds targeted coverage context when
536538
the graph could not fully parse or index that file. It never denies or replaces
537539
the requested tool call.
@@ -664,9 +666,11 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
664666
| `get_code_snippet` | Read source code for a function by qualified name. |
665667
| `get_architecture` | Codebase overview: languages, packages, routes, hotspots, clusters, ADR. |
666668
| `search_code` | Grep-like text search within indexed project files. |
667-
| `manage_adr` | CRUD for Architecture Decision Records. Query modes do not wait behind a same-project reindex; writes remain serialized. |
669+
| `manage_adr` | CRUD for Architecture Decision Records (`get` reads, `update` replaces the whole document, `set_sections` rewrites only the named sections and leaves every other byte untouched, `sections` lists headings). Query modes do not wait behind a same-project reindex; writes remain serialized. |
668670
| `ingest_traces` | Ingest runtime traces to validate HTTP_CALLS edges. |
669671

672+
`manage_adr(mode='set_sections')` writes one or more sections by name and splices them into the stored document, so text outside the named sections — including a preamble, code fences and section ordering — is preserved byte-for-byte. Any `## Heading` works, not just the conventional PURPOSE / STACK / ARCHITECTURE / PATTERNS / TRADEOFFS / PHILOSOPHY set; names match exactly, including case. Writing the same section twice is a no-op, so a retry after a lost response cannot duplicate content.
673+
670674
`manage_adr` query modes (`get` and `sections`) use the server's cached query store so they can proceed while a same-project reindex is running. If another process publishes a replacement store during reindexing, they can return the pre-publication ADR until idle eviction refreshes that cache. Updates remain serialized through the project mutation guard.
671675

672676
## Graph Data Model
@@ -796,14 +800,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso
796800

797801
**Two-layer architecture:**
798802

799-
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 161 languages. Extracts definitions, calls, imports.
803+
1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 162 languages. Extracts definitions, calls, imports.
800804
2. **Hybrid LSP pass** — type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, so you always get *some* answer.
801805

802806
The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls — without paying for a language server process per project.
803807

804808
## Language Support
805809

806-
161 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
810+
162 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes):
807811

808812
| Tier | Score | Languages |
809813
|------|-------|-----------|
@@ -829,7 +833,7 @@ src/
829833
traces/ Runtime trace ingestion
830834
ui/ Local HTTP server + verified external 3D-UI asset pack
831835
foundation/ Platform abstractions (threads, filesystem, logging, memory)
832-
internal/cbm/ Vendored tree-sitter grammars (161 languages) + AST extraction engine
836+
internal/cbm/ Vendored tree-sitter grammars (162 languages) + AST extraction engine
833837
```
834838

835839
## Security

THIRD_PARTY.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,15 @@ License summary:
5252
- `clojure` ([sogaiu/tree-sitter-clojure](https://github.com/sogaiu/tree-sitter-clojure)) is **CC0-1.0**;
5353
`fennel` is **CC0-1.0**; `jinja2` and `just` are **Apache-2.0**;
5454
`pine` is **ISC** (declared by its upstream).
55-
- The grammars authored in-house for this project (`cobol`, `form`, `janet`,
56-
`magma`, `protobuf`, `wolfram`) are **MIT** under the project's own license,
57-
(c) DeusData. Seven further grammars (`arkts`, `assembly`, `cfml`,
55+
- The grammars authored in-house for this project (`chialisp`, `cobol`, `form`,
56+
`janet`, `magma`, `protobuf`, `wolfram`) are **MIT** under the project's own
57+
license, (c) DeusData. Each ships the repository's own LICENSE, byte-identical
58+
to the root copy; they carry no third-party copyright because there is no
59+
third party. `chialisp` is a generic s-expression grammar for the Chia
60+
smart-coin language, written for this project because no usable public
61+
grammar exists; its source and corpus tests live in
62+
`tools/tree-sitter-chialisp/`.
63+
- Seven further grammars (`arkts`, `assembly`, `cfml`,
5864
`cfscript`, `dotenv`, `pine`, `qml`) are self-maintained forks that retain
5965
their original upstream authors' licenses — see the manifest for per-grammar
6066
provenance. `arkts` is a first-party derivative of

0 commit comments

Comments
 (0)