feat(index): support MinHash LSH scalar index - #9042
Conversation
A `minhashlsh` scalar index returns the rows whose token shingles have the highest estimated Jaccard similarity to a query text, ordered by `_distance = 1 - estimated Jaccard`. Text is tokenized with the full text search tokenizer, shingled, hashed and permuted into a 16-bit MinHash signature that LSH banding groups into candidates. A segment stores `signatures.lance` (one signature per document) and `bands.lance` (fixed-width `(band_key, doc_id)` rows with a page table); every signature parameter lives in `MinHashLshIndexDetails`, whose tokenizer reuses `InvertedIndexDetails`. Queries read one page per band, refine the candidates by signature under the prefilter, merge segments by distance, and score rows the index does not cover on the fly unless the scan uses `fast_search`. Builds spill sorted runs and merge them by partition, so memory does not grow with the table; merges, updates and remaps rebuild from stored signatures. Exposed as `Scanner::minhash_search` in Rust, `MinHashQuery` as `nearest` in Python and `IndexType::MinHashLsh` in Java. Framework additions are `ScalarIndexPlugin::validate_new_segments_against_existing` and `CacheBackend::capacity_bytes`, both defaulted. The format specification follows separately.
github-actions
Bot
added
A-python
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
zhangyue19921010
changed the title
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The MinHash/LSH direction addresses the near-duplicate use case, but the durable index contract needs to land through Lance’s format-change process before this implementation can be accepted. Please first submit the persisted details together with the corresponding docs/src/format/ specification in a dedicated format PR; after that contract lands, rebase this implementation onto it and remove the overlapping schema change.
| // signatures written by different segments (and computed on the query side) | ||
| // are comparable; the index files repeat them as schema metadata only to | ||
| // detect a details/file mismatch. | ||
| message MinHashLshIndexDetails { |
There was a problem hiding this comment.
This introduces a persisted index-details schema without the matching format specification and in the same PR as the full implementation. The accepted format process (#7399) requires a dedicated PR containing the proto, docs/src/format/ contract, and only compile-required library edits, followed by the implementation PR; otherwise voters cannot review the durable reader/writer contract independently before code starts emitting it. This matters here because discussion #8820 describes a different on-disk layout (bucket postings and u32 signatures) from this branch’s repeated (band_key, doc_id) rows and u16 signatures, so it cannot substitute for the versioned spec. Please land that contract PR first, then rebase this implementation onto it.
lance-gatekeeper
Bot
added
K-changes
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The latest commit changes only the protobuf comment syntax; the format-contract blocker remains. Please first submit MinHashLshIndexDetails with its matching docs/src/format/ specification in a dedicated format PR. Once that contract is approved, rebase this implementation onto it and remove the overlapping schema change.
lance-gatekeeper
Bot
added
the
K-changes
1 participant
Please refer to #8820 for the full discussion
A
minhashlshscalar index returns the rows whose token shingles have the highest estimated Jaccard similarity to a query text.Text is tokenized with the full text search tokenizer, shingled, hashed and permuted into a 16-bit MinHash signature that LSH banding groups into candidates. A segment stores
signatures.lance(one signature per document) andbands.lance(fixed-width(band_key, doc_id)rows with a page table); every signature parameter lives inMinHashLshIndexDetails, whose tokenizer reusesInvertedIndexDetails. Queries read one page per band, refine the candidates by signature under the prefilter, merge segments by distance, and score rows the index does not cover on the fly unless the scan usesfast_search. Builds spill sorted runs and merge them by partition, so memory does not grow with the table; merges, updates and remaps rebuild from stored signatures.Exposed as
Scanner::minhash_searchin Rust,MinHashQueryasnearestin Python andIndexType::MinHashLshin Java.