What should be in scope for Lance?

[wjones127]

Lance today spans a wide range. At one end is functionality that's clearly core to a file and table format: encodings (lance-encoding), the file reader/writer (lance-file), the manifest and fragment model (lance-table), object store I/O (lance-io), and index structures.

At the other end is functionality that looks a lot like a query engine:

• Scanner — filter and projection pushdown, limits, vector/FTS/hybrid search planning (dataset/scanner.rs, ~16k lines)
• SQL parsing and planning (dataset/sql.rs, lance-datafusion/src/sql.rs)
• Merge-into, update, and delete as full operations with their own logical and physical plans (write/merge_insert, write/update.rs)

I don't want to argue for a particular end state in this thread. What I do want to put on the table is that scope has a maintenance cost, and I think we're paying it:

• We have a very large volume of PRs, and they are a mix of both. To review them, one has to know a lot about query engines and about the fundamental design of the format.
• We ship a full query engine—DataFusion—in our binaries and Rust dependency tree. This is a heavy dependency for a query engine to take on just to support a table format.
• The two largest source files in the repo are query-layer files (scanner.rs and merge_insert.rs, ~16k and ~15k lines with tests). That's also where a lot of the churn and breakage concentrates. I can't help but feel these need a major refactor anyways.
• Meanwhile several low-level crates — lance-encoding, lance-file, lance-io, lance-select, lance-arrow — have no DataFusion dependency at all. The gradient already half exists.

There's a "what is Lance for" question underneath this too. There are already a lot of good query engines: DuckDB, DataFusion, Spark, Trino, LanceDB. I'm skeptical that Lance should try to be another one, when the thing only Lance can provide is the format and the primitives to read it well. delta-kernel-rs is one example of the alternative scope. It contains engine-agnostic building blocks, leaving planning and execution to the engines.

But I know the high-level APIs within Lance, including pylance, get a lot of use. "Narrow the scope" might mean different things depending on which APIs are your main entrypoints to Lance. Whatever we do has to start from real use cases rather than from an architecture diagram. So I'd like to hear:

• What do you use from the high-level surface today — Scanner, SQL, merge insert, update/delete? What would it look like to use a downstream engine (DuckDB, Spark, etc.) for that instead?
• If you integrate Lance with another engine, where do you wish you could plug in? What do you currently have to reach into internals to get?
• Where would you draw the line between core and high-level?

Nothing is being removed and no decision has been made. This is the start of the conversation.