Performance regression: test_conditional_update[v1_indexed] (~517% latest-run slowdown)

[westonpace]

Lance SHA: dabd971\nBenchmark run: https://github.com/lancedb/lance-bench/actions/runs/32656832705\nBenchmark: python/ci_benchmarks/benchmarks/test_merge_insert.py::test_conditional_update[v1_indexed]\nCrate/suite: Python CI benchmarks (merge_insert)\n\nThe regression analysis flags this benchmark at p=0.000750, with the recent four-result window 102.45% slower than older history.\n\nThe latest individual result shows a sharper step:\n- 2c1500f (2026-08-23 10:56 UTC): 8.68 ms\n- dabd971 (2026-08-23 17:57 UTC): 53.54 ms\n\nRecent preceding results were generally 8.4–12.0 ms, so the latest result is approximately 517% slower than the immediately previous run.\n\nThe suspect range is 2c1500f..dabd971. In that range, 30cbafc ("feat(merge_insert): add write_mode to select how merged rows are written", #8423) is directly related and landed shortly before the benchmark, making it the leading candidate.\n\nPlease reproduce the v1 indexed conditional-update benchmark and compare 2c1500f against 30cbafc/dabd971.

[LuciferYang]

Let me look into it.

[LuciferYang]

Please disregard the previous conclusion for now. I will get back to you after verification.

[Sadik00789]

Hi @westonpace, I can take this on.

I will reproduce the test_conditional_update[v1_indexed] benchmark locally across 2c1500f and dabd971, profile the changes introduced in #8423 (write_mode dispatch in merge_insert), and open a PR with the fix to restore baseline latency.

Could you please assign this to me?

[LuciferYang]

I could not reproduce this locally, and what I did measure points at the runner rather than at a commit in that range.

Taking the two endpoints the issue names, both built in release mode and run about fifteen minutes apart on the same machine against the 10M-row merge_insert_narrow dataset:

commit	conditional_update[v1_indexed]
2c1500f (reported good)	10.15 ms
dabd971fe (reported bad)	8.79 ms

The end reported as bad is the faster one. Driving the same operation on current main (897a8bdbd) today gave 11.1 ms. All three sit inside the 8.4 to 12.0 ms band this case has held for months, while CI reported 53.54 ms at dabd971fe. Bisection needs the two ends to differ, and locally they do not.

The reason I doubt it is a commit at all: in that window the whole suite moved, not this case. Counting the complete high-concern list rather than the top of it, the total went from 137 entries to 296, the Python subset from 80 to 197, and the pure-CPU codec group (decode_*, encode_*, zip_*, bitpacking) from 26 to 57. Those share no code with merge_insert. A suite-wide doubling that includes pure-CPU work looks like a property of the machine rather than of a diff.

On #8423 specifically, it cannot reach this benchmark. The case is a full-schema source with write_mode="auto" and use_index(True). The new routing check requires is_subset_schema, which is false here; Auto returns RewriteRows on the first line of select_write_sink, so the plan is unchanged; and the only new write-side work is an offsets bitmap gated on uses_stable_row_ids(), while the benchmark dataset is created with enable_stable_row_ids at its default of False.

What I cannot check from here is the runner itself. This machine is aarch64, so the local numbers say nothing about an x86_64-specific effect or about a level shift in the CI environment. If the raw per-run series from lance-bench-results is shareable, I am happy to look at where the level moved. #8472 may be relevant background, since it tracks noise from #8377 lowering rustflags to x86-64-v2.

Chasing this did turn up a real problem in the same area, though not the one reported here. With a composite key whose columns are all indexed, MapIndexExec evaluates every per-key probe as a single AND, so a key with few distinct values materializes a candidate set the size of the table while pruning almost nothing: 198.9 ms for the composite case where the v2 hash path takes 128.7 ms, and an outright Resources exhausted at the default 150 MB pool when that key is probed on its own. Filed as #8718, with a fix in #8719 that probes the most selective key first and stops early (198.9 ms to 10.2 ms, paired release measurements against pristine main). It leaves the single-key path untouched, so it is not a fix for the case in this issue. @westonpace, a review would be welcome when you have time.

I posted two comments here earlier and deleted them. One of them blamed #8587, which was wrong: I read a truncated list and drew a conclusion the full data does not support. Sorry for the churn.

[westonpace]

I'm going to close this particular issue for now. I've reverted the rustflags change and we can see how benchmark results look after that change. The follow-ups can be tracked in their own issues.