Add memory-aware concurrent indexing for large repositories

[zhiyuzhang001-a11y]

Summary

• scope daemon index workers to the canonical request repository instead of inheriting the daemon starter workspace boundary
• add bounded two-pass indexing so large repositories do not retain the full extraction set in memory
• add a global memory-aware scheduler for concurrent indexing across projects, while preserving exact output parity

Motivation

This enables multiple project-scoped MCP clients to share the Provider safely. Small daily repositories can run concurrently; large jobs are admitted according to measured source bytes and memory budget instead of starting without a global bound.

Validation

• rebased onto current upstream main
• git diff --check passes
• ASan/UBSan focused suites: 788 passed, 1 platform skip
• suites: subprocess, daemon_application, discover, pipeline, extraction
• the same three commits previously passed the full local Provider suite: 7477 passed, 4 platform skips
• downstream Codebase Atlas product suite: 237/237
• independent root acceptance suite: 47/47

No release or binary distribution is included in this PR.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Read in full and routed for a maintainer fit decision. Since the only thing on this thread so far is the automated acknowledgement, here is honestly where it stands rather than silence.

CI is green and the validation you supplied is unusually thorough for a first contribution — ASan/UBSan focused suites, the full local suite, and named downstream results. That is not what is holding it.

What needs a maintainer ruling is that this is three separable changes in one PR, and one of them is a one-way door:

1. Scoping daemon index workers to the canonical request repository. This moves a workspace boundary, which is a security surface here — it was hardened deliberately in a recent round, on both entry points.
2. Bounded two-pass indexing so a large repository does not retain the full extraction set in memory. This one is self-contained and the easiest to reason about on its own.
3. A global memory-aware scheduler admitting concurrent indexing across projects by measured source bytes against a memory budget. This is the one-way door: a scheduler with a budget is a permanent maintenance surface, and its knobs outlive whoever tuned them.

Each of those is reviewable on its own evidence; together, a reservation about any one blocks all three. That is why the split matters and it is not a formality — but I am not going to ask you to do the splitting work before the direction call on (3) is made, because the answer changes what the split should look like.

One thing that would genuinely help the decision, if you want to write it while this is queued: what does the scheduler do when the budget is wrong — too low, or too high on a machine it mis-measures? Failing open, failing closed, and degrading to sequential are three different products, and which one it is matters more than the admission arithmetic.

I will come back with the ruling rather than leaving this to age.

[DeusData]

The maintainer ruling is in. Short version: please split this into three PRs, and the third one needs to work on all three platforms before it can be reviewed.

Split into three

1. Daemon workers scoped to the canonical request repository. Reviewable on its own. It moves a workspace boundary, which is a security surface here — it was hardened deliberately on both entry points recently — so it deserves to be looked at without a scheduler in the same diff.
2. Bounded two-pass indexing. Take this one first if you want the quickest path to a merge. It is self-contained, platform-independent, and it addresses a problem this project demonstrably has on its own benchmarks: a single Linux kernel index has peaked at 16.45 GB, and a 2× run OOM'd on a 36 GB machine. Not retaining the full extraction set is squarely on the roadmap regardless of anything else here.
3. The global memory-aware scheduler. See below.

What (3) needs first: cross-platform measurement

The admission logic rests on application_worker_observed_rss, and that path is:

DIR *directory = opendir("/proc");
if (!directory) {
    return false;
}

…then a walk of /proc/<pid>/stat and /proc/<pid>/statm.

/proc does not exist on macOS or Windows, both of which are first-class platforms here. On those, opendir fails, the probe returns false, job->observed_rss_bytes is never advanced from zero, application_active_observed_rss_locked therefore sums to zero, and daily_memory_busy can never become true. Admission falls back to slot counting with no memory awareness at all.

Failing open is the right choice over failing closed — nobody wants indexing blocked because a probe is unavailable. The problem is that the result is a scheduler which ships its code, its five environment knobs and its maintenance cost to all three platforms while protecting only one, and does so invisibly: there is no signal at runtime that the memory half is inert.

So (3) is reviewable once the RSS probe either works natively on macOS (task_info / proc_pidinfo) and Windows (GetProcessMemoryInfo), or explicitly degrades to a stated conservative bound and says so in a log line, rather than silently to none.

To be honest with you about scope: that is a substantial piece of platform work, and it is entirely reasonable to decide it is more than you want to take on right now. Splitting (1) and (2) out means neither is held hostage to that decision.

One design question for whenever (3) comes back

CBM_INDEX_RESOURCE_MODE, CBM_LARGE_REPOSITORY_FILE_THRESHOLD, CBM_LARGE_REPOSITORY_BATCH_FILES, CBM_LARGE_REPOSITORY_SOURCE_BYTES and CBM_STREAMING_BATCH_FILES are five new environment variables, and an environment variable is a one-way door — once it is documented, it is supported. Worth asking which of them are genuinely operator-facing policy and which are internals that could be derived or fixed constants.

Credit where it is due

The validation you supplied is well beyond what a first contribution usually carries: ASan/UBSan focused suites, the full local suite, and named downstream results rather than "tests pass". The /proc walk correctly filters by process group rather than assuming a flat child list, and the probe returns a status instead of a bogus zero — which is exactly why the platform gap was findable by reading rather than by an incident.

None of the above is a rejection of the idea. It is a request to let the two uncontroversial pieces land on their own evidence while the third gets the platform work it needs.