feat: add initial lance-context python/rust scaffold and uv workflow - #1
Merged
Conversation
beinan
added a commit
that referenced
this pull request
Sep 2, 2026
## Problem `Compact` tasks could sit queued indefinitely whenever `MergeWal` load was present. In production the 600s auto-sweep kept ~50–80 MergeWal tasks queued/running, fragments grew to **1,085,065** (1–10 rows each), and because every worker's open `Dataset` handle keeps the full manifest resident, **18 of 20 workers were OOMKilled**. Scaling the master 3→8 replicas did not help — every added slot was consumed the same way. ## Root cause: admission, not concurrency `MERGE_WAL_CONCURRENCY` already gave merge its own semaphore, but a single dispatch loop called one **unfiltered** `claim_next()` and only chose a pool *after* the claim. Claiming is destructive — it deletes the queue key, grants a lease, and takes the per-experiment target lock. So a claim made because the *general* pool had permits could return a `MergeWal`, then park it on the saturated merge semaphore **while holding its target lock**. With MergeWal numerically dominant, nearly every claim returned one, and the loop's `while` guard stayed true only because the general pool was idle. The dispatcher spun claiming work it could not run while a lone `Compact` sat queued behind it. ### One correction to the issue's diagnosis The issue lists "single shared queue, interleaved only by queue key order" as root cause #1. That part is not accurate: task ids are **UUIDv7** (`lance-context-core/src/id.rs`), so the queue is genuinely time-ordered FIFO — ordering was never arbitrary. This matters, because plain FIFO turns out to be sufficient once admission is fixed. **No priority, fairness policy, or reserved capacity was added** — those would solve a problem that isn't there. ## Fix Add `TaskKinds`, a small kind set threaded into `claim_next`, and run **one poller per pool** — `GENERAL` (Compact + IndexId) and `MERGE_WAL` — each claiming only the kinds it can actually run. The filter is applied *before* the dependency probe, so skipped kinds cost nothing. FIFO order within a kind is unchanged, and **no new config knob** is introduced. One caveat, now documented in the config docs rather than left implicit: `MERGE_WAL_CONCURRENCY=0` shares a single pool, which necessarily reinstates single-poller behavior. Prefer a non-zero value when both kinds are in play. ## Tests Both etcd-backed, and **both verified to fail with the fix reverted**: - `filtered_claim_reaches_compact_behind_merge_wal_backlog` — 30 MergeWal enqueued *ahead* of one Compact (worst case for FIFO). Asserts an unfiltered claim still returns MergeWal, so the test cannot silently stop exercising the scenario; that `GENERAL` reaches the trailing Compact; that `GENERAL` never returns MergeWal; and that the merge pool still drains its own backlog. - `compact_runs_while_merge_wal_pool_is_saturated` — end-to-end through the dispatcher against a stub worker that accepts and never responds. Asserts the Compact reaches `Done` **and** that MergeWal work is still outstanding, so it cannot pass merely because the backlog drained. **Negative verification:** with the kind filter removed, the first fails on the kind assertion and the second hangs to timeout — the production symptom exactly. Full suite green: 22/22 etcd-backed master tests, 222 core, 68 server, `fmt` + `clippy -D warnings` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
python/andrust/subprojectsContextAPI and PyO3 bindingsTesting