refactor: extract the shared manifest walk behind tracked_files

[LuciferYang]

tracked_files walks every present manifest with a four-stage pipeline: a lister that enumerates manifest locations and applies min_version, a reader that fetches them with bounded parallelism under a memory budget, an emitter that turns each manifest into file rows, and an index lister that materializes index directories. Only the last two are about the rows it emits. The first two are about walking manifests, and a second consumer needs exactly them.

That consumer is Dataset::referenced_files in #8097, the keep-set an external orphan-cleanup driver uses to decide what it may delete. In the discussion there the suggestion was to factor out the reusable part before rebasing that PR onto it, which is what this does. Nothing in this PR depends on #8097; tracked_files is the only caller here and its behavior is unchanged.

What moved

The lister and reader now live in dataset::files::scan, which yields a ScannedManifest per present manifest: the manifest, its own path, and the index metadata read alongside it. tracked_files keeps its emitter and index lister and consumes that stream. Channel capacities, the can_launch predicate, the biased select ordering, and the min_version filter are carried over unchanged.

Why the budget accounting changed shape

Previously the reader charged bytes before sending and the emitter released them after processing. That worked because the emitter was the only consumer and sat in the same file, so the charge was bounded by the reader's in-flight reads plus two channel slots.

A shared walk cannot rely on that: a second consumer that forgets to release would silently stall the reader. The charge now lives in a MemoryPermit held by ScannedManifest and released on drop, so backpressure follows the manifest's lifetime rather than a convention. Field order is load-bearing and commented: the permit drops after the manifest it accounts for.

The bound is on the reader's prefetch, not on what a consumer retains. One read is always allowed when nothing is in flight, which is what keeps a manifest larger than the whole budget from deadlocking the walk, so a consumer that holds every manifest gets serial reads rather than a stall. That escape hatch is unchanged from before, and the module doc now states this rather than promising a bound it does not provide.

Tests

Six cases in scan::tests, covering what the previous arrangement had no way to observe:

• the budget returns to zero once every manifest is dropped, and stays charged while a consumer holds them;
• min_version really does skip manifests, which is why a keep-set must leave it unset;
• total counts every manifest the walk yields;
• a failed manifest read surfaces one Err per manifest rather than being skipped, asserted as errors == 3 because errors > 0 would also pass on a reader that stopped at the first failure or on a listing failure;
• dropping the stream early releases every in-flight permit.

Each fails against the corresponding mistake: a leaked permit, a bypassed filter, a reader that aborts on first error.

cargo clippy -p lance --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps, and cargo fmt --all --check are clean; dataset::files (18) and dataset::cleanup (41) pass. The full suite is left to CI.

Reviewing this

The second commit is the result of reviewing the first, so the two are worth reading separately. It removes an unreachable error branch the extraction left behind, moves the index fan-out after the row batches so a full index channel cannot block row output while holding budget, makes the test-only budget accessor private, and corrects the module doc described above.

[lance-gatekeeper]

✅ Gate recommendation: approve.

This extraction puts manifest listing, bounded reads, and memory ownership behind the right crate-private stream while preserving tracked_files output and cancellation behavior. Drop-owned permits are a cleaner fit for reuse than consumer-managed accounting.

[LuciferYang]

cc @wjones127

[LuciferYang]

friendly ping @wjones127

[LuciferYang]

Could you please take a look at this patch when you get a chance, thanks. @wjones127

[wjones127]

This looks good! Thanks for working on this.

[LuciferYang]

Thank you @wjones127