feat(scanner): ignore selected scalar indices

[majin1102]

Summary

• add a scanner option to ignore selected scalar indices while leaving all other scalar indices eligible
• apply the ignored set by logical index name before scalar filter parsers and index details are loaded
• expose idiomatic APIs in Rust (with_ignored_scalar_indices), Java (ignoredScalarIndices), and Python (ignored_scalar_indices)

Motivation

External query planners can consume a scalar index during task planning and pass the resulting row selection to Lance workers. For example, lance-spark can query a ZoneMap on the driver, produce fragment-local physical slices, and then avoid querying that same ZoneMap again on executors without disabling useful BTree or other scalar indices.

This fills the gap between the existing all-or-nothing use_scalar_index option and retaining every available scalar index.

Semantics

• ignored indices are identified by logical scalar index name
• the original expression filter is still evaluated, so ignoring an index only changes planning/performance
• unknown names have no effect
• use_scalar_index = false continues to disable all scalar-index planning
• vector index segment selection and explicit full-text search are unaffected

Validation

• cargo fmt --all --check
• cargo check -p lance --tests
• cd java && ./mvnw spotless:check
• cd java && ./mvnw -Dtest=ScannerTest,AsyncScannerTest test (54 Java tests and 22 JNI tests)
• cd python && cargo check --release
• Python Ruff formatting and lint checks
• Python runtime test (local release extension linking stalled; left for CI)

The PR is intentionally opened as a draft so the API shape and naming can be discussed before lance-spark adopts it.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T12:24:07.332032Z	9503cee	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9503ceeb27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lance-gatekeeper]

✅ Gate recommendation: approve.

The Java/JNI naming issue is fixed in 89c43eca: the public option and its full binding path now consistently use ignoredScalarIndices. The selective exclusion remains at the scalar-parser construction boundary, preserving the original filter and all unaffected index paths.