Full index build allocates ~11.5 GB on a ~14k-file TypeScript monorepo regardless of the configured memory budget (0.10.8)

[shafty023]

Version

codebase-memory-mcp 0.10.8 (aarch64 static build)

Platform

Linux (arm64)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard

What happened, and what did you expect?

A full index_repository of a single TypeScript monorepo (13,969 files) peaks at
about 11.0 to 11.9 GB RSS on a 16 GB host, and the configured memory budget does not
bound it. The default ram_fraction budget on this box is 3,916 MB and an explicit
CBM_MEM_BUDGET_MB=2500 was also honored in the logs but exceeded by roughly 3 to 4x
in both cases. The same binary indexes five smaller repositories on the same box with
memory flat (the largest of those is 34,829 nodes / 100,940 edges in 8.1 s), so this is
specific to scale, not to the install.

I expected the build to either stay near its stated budget or fail with an explanatory
error naming the pass that overshot. Instead it produced three distinct problems, which
I am reporting together because they share the one allocation.

The allocation is not a concurrency artifact. A single-threaded run
(CBM_INDEX_SINGLE_THREAD=1, CBM_MEM_BUDGET_MB=2500) still reached 11.4 GB, and the
worker log attributes it to one pass:

parallel.registry.done entries=787665 defines=804677 imports=99865
pass.timing pass=registry_build elapsed_ms=2480
mem.phase phase=registry_build rss_mb=11419 peak_mb=11419

787,665 registry entries cost about 11.4 GB with a single worker, in 2.5 seconds. That
matters for triage: backpressure or worker-count tuning cannot fix a single pass that
wants 11.4 GB on its own, so I do not think this is the same defect as a parallel
over-admission overshoot.

A separate multi-worker run got through extraction and died entering resolve at a
comparable peak:

pass.timing pass=lsp_cross_prepare elapsed_ms=5366
mem.phase phase=lsp_cross_prepare rss_mb=11025 peak_mb=11025
parallel.resolve.start files=13969 workers=4

with the daemon having started under:

daemon.start version=0.10.8 memory_budget_bytes=4107238400 physical_job_limit=4
mem.init budget_mb=3916 total_ram_mb=15665 source=ram_fraction

Consequence 1: multi-worker mode hangs rather than failing

When memory pressure stalls forward progress, the supervisor does not surface an
out-of-memory condition. It times the worker out and reports a hang:

index.supervisor.worker_failed outcome=hang exit_code=-1 signal=15
index.supervisor.reap outcome=hang exit_code=-1 signal=15

This reads as "one file wedged the worker", which sent me looking for a bad file for
some time. It is not one file. Across attempts, extraction stalls at a different
position each run: 12,360, then 12,890, then 13,220 of 13,969 files, with no file in
common between the stall points. The position tracks how much memory was already
committed when that run started, not the content at that offset.

If the supervisor can see that RSS is far past budget when it reaps a hang, saying so
in the worker_failed line would remove the entire wrong-turn diagnosis.

Consequence 2: auto_watch=true runs this build inside the serving daemon

auto_watch defaults to true, so the watcher inside the long-lived MCP session daemon
noticed new upstream commits on the watched monorepo and began rebuilding the index
in process. The build wants about 11.5 GB, the box had roughly 10 to 11 GB free
with services resident, and the kernel killed the daemon. The supervising service
restarted, a fresh session daemon started, the watcher fired again, and the cycle
repeated. Four kills in fourteen minutes:

Out of memory: Killed process 22295 (codebase-memory) total-vm:17120976kB,
anon-rss:11338244kB ... oom_score_adj:200
Out of memory: Killed process 26839 (codebase-memory) total-vm:17121040kB,
anon-rss:11848124kB ... oom_score_adj:100

Two things make this worse than a failed index. The daemons self-set oom_score_adj
to 200 and 100, which is what kept the kernel selecting them. And because the victim is
the server, every client reaching it got HTTP 502 for roughly ten seconds per restart,
so the visible symptom was an intermittently unavailable MCP server rather than
anything that looked like an indexing problem.

A serving daemon probably should not run an unbounded full build in its own address
space at all. At minimum, an in-daemon watcher rebuild seems like it should be capped,
or run out of process, or be off by default at this repository size.

Consequence 3: the CLI hint points away from the evidence

Every one of these failures surfaces to the caller as only:

{"status":"error","hint":"Pipeline failed. Check repo_path exists and contains source files. ..."}

repo_path existed and contained source files in all of them. The real evidence is in
undocumented files under <cache_dir>/logs/ (cbm-daemon.log and .worker-log-*),
which I found by guessing. Nothing in the CLI output mentions either path.

The knobs that actually resolve this are also undocumented. I found them by running
grep -aoE 'CBM_[A-Z0-9_]+' over the 281 MB static binary: CBM_INDEX_SINGLE_THREAD,
CBM_MEM_BUDGET_MB, CBM_INDEX_WORKER_TIMEOUT_S, CBM_DISABLE_LSP_CROSS,
CBM_INDEX_MAX_RESTARTS, CBM_MAX_FILE_BYTES, CBM_RETAIN_PER_FILE_MB,
CBM_RETAIN_TOTAL_MB, CBM_SEMANTIC_ENABLED and others. The config command exposes
only auto_index, auto_index_limit, auto_watch, ui-lang, ui_enabled and
ui_port, none of which touch memory. Documenting the first three would have saved the
whole investigation.

What works

Stated plainly, because it narrows the problem:

• CBM_INDEX_SINGLE_THREAD=1 with about 15 GB free (every other service stopped)
  completes the identical full index in 1m33s, status=indexed, 164,323 nodes and
  834,119 edges, both matching the expected counts, parse_partial_count 278, six
  indexes and none quarantined. One caveat so the timing is not overread: that run
  reused cached extraction from the preceding failed attempts, so 1m33s is not a
  cold-start number. The memory behavior is the point, not the speed.
• mode: fast succeeds on the same repository.
• The five smaller repositories on the same box index normally, memory flat throughout.

So the build is entirely capable of completing. It just needs about 11.5 GB of headroom
that neither budget setting reserves or enforces, and nothing in the product tells an
operator that in advance.

Reproduction

I cannot share this tree, and I want to be straight about that rather than imply a
public repro I have not run. What I can characterize precisely:

1. Code being indexed: a private TypeScript monorepo, 13,969 files, producing
   787,665 registry entries, 164,323 nodes and 834,119 edges. Ordinary application and
   library TypeScript with dense cross-package imports; no generated blobs, no vendored
   trees, nothing that .gitignore should have excluded.

2. Host: 16 GB RAM (total_ram_mb=15665), 4 vCPU, aarch64, Linux.

3. Command:

   codebase-memory-mcp cli index_repository '{"repo_path":"<repo>","mode":"full"}'
   

4. Result: with services resident, either a supervisor hang (multi-worker) or a
   kernel OOM kill, at 11.0 to 11.9 GB RSS in both cases, against budget_mb=3916 by
   default or 2,500 when set explicitly. With CBM_INDEX_SINGLE_THREAD=1 and about
   15 GB genuinely free, success in 1m33s.

Expected: the build respects CBM_MEM_BUDGET_MB, or fails with an error naming
registry_build and the observed RSS.

My guess at a public equivalent, untested: any TypeScript monorepo in the 12k to 15k
file range with heavy cross-package imports, indexed on a host with under about 16 GB
free, should show the same mem.phase phase=registry_build peak. If it would help, tell
me which public repository you would like measured and I will run it on this box and
post the phase table.

Diagnostics trajectory

Not captured during these runs, since I did not know CBM_DIAGNOSTICS=1 existed until
after the fact. I can reproduce with it set and attach the ndjson on request; the
repository and the box are both still available, and I can also test a patch or a build
with extra mem.phase instrumentation.

Project scale

164,323 nodes / 834,119 edges / 13,969 files (787,665 registry entries)

Confirmations

• I searched existing issues. This overlaps several open reports but I do not believe it
  is a duplicate of any of them, so I am listing what I found and why I filed separately.
  Please close this into whichever one you prefer if you disagree.
  • Daily OOM crash-loop: worker RSS far exceeds its own memory budget, then full reindex retriggers on every crash #1864 (daily OOM crash-loop, worker RSS far exceeds its budget) is the closest:
    same version, same platform class, same "budget is advisory" conclusion. It differs
    in the loop mechanism, which there is invalid_existing_db retriggering a full
    reindex on a schedule, and here is auto_watch rebuilding inside the serving
    daemon. It also has no phase attribution, whereas the single-threaded 11.4 GB
    registry_build measurement above says the overshoot is one pass rather than
    parallel over-admission. Happy to have this merged into Daily OOM crash-loop: worker RSS far exceeds its own memory budget, then full reindex retriggers on every crash #1864 if you read it as one
    defect; the registry_build and single-thread numbers are the part I would want
    carried over.
  • index_repository: OOM kill instead of degradation — nested .gitignore below another .gitignore is dropped, and the memory budget is advisory (mem.backpressure.futile → soft_overshoot) #1973 also finds the budget advisory (mem.backpressure.futile to
    soft_overshoot) and also notes the caller sees nothing useful. Its over-collection
    root cause is a nested .gitignore discovery bug, so it indexes files it should
    have skipped. This repository has no such excess; the 13,969 files are all files we
    intend to index.
  • PHP: lsp_cross/parallel.resolve livelocks + unbounded RAM on large codebases (no PHP step-budget guard) — public repro: Shopware #1072 hits lsp_cross / parallel.resolve unbounded RAM but is PHP specific,
    and argues the TypeScript resolver self-limits. The lsp_cross_prepare peak above
    is TypeScript, which may be worth a look in that context, though registry_build is
    the larger allocation here.
  • watcher.changed/strategy=git self-triggers infinite reindex loop on v0.10.8 (looks like a #1254 regression) #1953 is also a 0.10.8 auto_watch loop, but there the watcher fires with no
    real changes. Here the watcher was correct that upstream commits existed; the defect
    is the cost and the location of the rebuild it triggers, not the detection.
  • Worker continuously hangs and indexing process gives up eventually #1130 is a supervisor hang, but on 0.9.0 and attributable to specific quarantined
    files. Here the stall position moves between runs with no file in common.
• My reproduction does not use a public repository. It is a private monorepo and I
  have not managed to reduce it to a shareable case, since the trigger appears to be
  aggregate scale rather than any single construct. I have described the scale and the
  environment as precisely as I can above, and I would rather say so than check a box
  that is not true. Glad to run whatever public repository you nominate.

Same deployment as #1991 (the per-request latency floor), if that is useful context for
the environment. Thanks for the work on this project.

[shafty023]

I built 0.10.8 from source (tag v0.10.8, commit 46ae198) with scripts/build.sh on macOS arm64 and indexed a TypeScript monorepo of the same shape as the one above, so the pass numbers line up closely enough to compare:

parallel.registry.done entries=787954 defines=804999 imports=100027
pipeline.done nodes=164684 edges=835879

(reported above: entries=787643, defines=804653, nodes=164628, edges=835682)

Then I added two temporary local patches: a census of what is actually live in the per-file result arenas at mem.collect phase=post_extract, and per-callsite attribution of every cbm_arena_alloc byte by the caller's return address. Neither is anything I would submit; they exist to answer where the 11 GB is.

CBM_CACHE_DIR=$PWD/cbm-cache CBM_PROFILE=1 \
  ./build/c/codebase-memory-mcp cli --json index_repository \
  --repo-path $PWD/<repo> --mode full --name census

Worth noting on its own: CBM_PROFILE=1 is what keeps <cache_dir>/logs/.worker-log-* after a clean exit. Without it the supervisor unlinks the log at index_supervisor.c:601 and none of these lines survive a successful run.

What is live at post_extract

files=13999 mapped_mb=11226 handed_mb=8185 item_structs_mb=903
retained_src_mb=157 tail_slack_mb=3040 blocks=48045

• mapped_mb is the sum of every block behind the 13,999 live CBMFileResult.arena instances. 11,226 MB, in 48,045 malloc'd blocks. That is within 0.6 percent of the 11,161 MB post-extract RSS reported above, so I believe this array is the entire balloon.
• item_structs_mb 903 MB is the fixed item arrays (defs.cap * sizeof(CBMDefinition) and the other thirteen). sizeof(CBMDefinition) is 240, sizeof(CBMCall) 320, sizeof(CBMUsage) 40.
• retained_src_mb 157 MB is retain_sources. Already capped, not the problem.
• A strlen sweep over every string the cached results still point at (def names, QNs, signatures, docstrings, body tokens, call and usage strings) totals 417 MB.

So about 1.5 GB of the 11.2 GB is reachable data. The remaining 6.7 GB was handed out of the arenas and then abandoned, and a bump allocator has no way to hand it back.

Where the abandoned bytes come from

Top callsites by bytes, symbolized:

1584 MB  12673 calls  scan_string_consts_js  internal/cbm/extract_channels.c:104
1584 MB  12673 calls  extract_channels_js    internal/cbm/extract_channels.c:376
 676 MB  25308 calls  cbm_calls_push         internal/cbm/cbm.c:109 (GROW_ARRAY)
 586 MB  15312 calls  cbm_defs_push          internal/cbm/cbm.c:109 (GROW_ARRAY)
 349 MB  14.1M calls  cbm_node_text
 346 MB  38019 calls  ts_reg_method
 222 MB  40042 calls  cbm_usages_push        internal/cbm/cbm.c:109 (GROW_ARRAY)
 198 MB  12673 calls  walk_es_imports
 157 MB  13999 calls  extract_worker         (the retain_sources copy)

The top two are the same allocation twice: ts_nstack_init(&stack, ctx->arena, CHAN_STACK_CAP), with CHAN_STACK_CAP 4096 and sizeof(TSNode) 32, so 128 KiB per call, once in scan_string_consts_js and once in extract_channels_js, per JavaScript/TypeScript file, out of ctx->arena, which is result->arena. 12,673 TS/TSX files times two times 128 KiB is 3.09 GB.

That 3.09 GB produced 23 Channel nodes out of 164,684.

extract_node_stack.h says of those stacks:

Uses the arena allocator for zero-fragmentation growth: old blocks are abandoned (freed when the arena is destroyed at end of file extraction).

The arena is not destroyed at end of file extraction. In the parallel path cache[] is allocated in run_parallel_pipeline (src/pipeline/pipeline.c:1150), filled at ec->result_cache[file_idx] = result (src/pipeline/pass_parallel.c:963), and freed only after parallel_resolve and the infra passes (src/pipeline/pipeline.c:1322). So a scratch structure written for a one-file lifetime is in fact held for the whole index, 14k times concurrently. ts_nstack_init appears 55 times across internal/cbm/extract_*.c; channels is simply where the constant cap is largest and the yield smallest.

The TSTree is not the problem, for the record: cbm_free_tree(result) at pass_parallel.c:908 releases it right after the defs loop, exactly as intended.

Experiment

I routed ts_nstack_init and ts_nstack_push to a per-thread arena that is reset at the top of cbm_extract_file_ex, leaving all 55 callsites untouched. About 30 lines.

before:  mapped_mb=11226  handed_mb=8185  tail_slack_mb=3040  blocks=48045
after:   mapped_mb=5680   handed_mb=3973  tail_slack_mb=1706  blocks=34186

Retained set down 49 percent. nodes=164684 identical, parse_partial_count=280 identical, entries/defines/imports identical, and the per-file defs/calls/imports/usages/typerefs counts identical. Edges came out 835873 against 835879; five unmodified runs on this same tree gave 835868, 835869, 835873, 835877 and 835879, so that gap is the existing run to run spread rather than an effect of the change.

One caveat about my measurements. Peak RSS here did not reach 11 GB, because this macOS build logs mem.allocator.bound_populations_only owned_classes=0/6 and ordinary malloc is served by the system allocator, which never commits the untouched tails of those blocks. Resident stayed at 2.4 GB while 11.2 GB was malloc'd. On the aarch64 build in this report, which logs mem.allocator.owned classes=all, mapped and resident look like the same number, which is why the two figures land 0.6 percent apart.

Questions

1. Would you take a PR that gives extraction its own scratch arena for AST traversal stacks, reset per file, so ts_nstack_* stops writing into the result arena? It is contained to extract_node_stack.h plus a few lines in cbm_extract_file_ex, no callsite churn, and it is half the retained set on this repo. Happy to open a separate issue first if that is the process you want.
2. Add memory-aware concurrent indexing for large repositories #1925 is taking on bounded two pass indexing for the retention half of this. Is the scratch split better kept out of that PR, given it needs none of that machinery and lands independently?
3. Would a mem line reporting the retained arena total at post_extract be welcome? mem.collect phase=post_extract rss_mb= reports the symptom; the arena total is the number that points straight at the cause, and it is a cheap sum over cache[].