fix(fts): use substring-safe ngram defaults

[lance-gatefixer]

Summary

• make the NGRAM tokenizer disable stemming and stop-word removal by default so valid substring tokens are not discarded
• preserve explicit filter overrides and analyzer settings persisted by existing indexes
• document the conditional defaults and extend the existing multi-fragment Python FTS regression test

Root cause

Selecting base_tokenizer="ngram" changed the lexical tokenizer but inherited the text analyzer defaults, which enable word-oriented stemming and stop-word removal. Both indexed content and queries therefore dropped valid NGRAM tokens such as the.

Existing NGRAM indexes retain their persisted analyzer behavior and must be rebuilt to adopt the corrected defaults.

Validation

• cargo test -p lance-index scalar::inverted::tokenizer::tests
• uv run make build
• uv run pytest python/tests/test_scalar_index.py::test_fts_ngram_tokenizer -q
• uv run make lint
• cargo fmt --all
• cargo clippy --all --tests --benches -- -D warnings

Fixes #8777

[lance-gatefixer]

Blocked: the current-head Python macOS 3.14 ARM check failed in a MemWAL test outside the FTS repair, with no evidence supporting an attributable code change.

Remote head 2609852 contains current main tip 7b80136. Python macOS 3.14 ARM (run 32948474973, job 98114307860) failed only python/tests/test_mem_wal.py::test_shard_writer_lsm_scanner_includes_own_sstables because _pk_index/page_lookup.lance was not found; the job otherwise reported 1,346 passing tests, while the repair diff is limited to FTS tokenizer defaults, documentation, and regression coverage.

I fetched and verified both refs, inspected the complete failure evidence, and reran the exact failed test on this head, where it passed. Please rerun the failed check; alternatively, investigate the independent MemWAL missing-file failure if it reproduces.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The merge from main preserves the previously verified NGRAM behavior: omitted word filters remain substring-safe, while explicit and persisted settings stay authoritative.