Skip to content

fix(arrow): deep-copy the right rows from a sliced boolean column - #9021

Open
MaxFreedomPollard wants to merge 1 commit into
lance-format:mainfrom
MaxFreedomPollard:fix/deepcopy-sliced-offset
Open

fix(arrow): deep-copy the right rows from a sliced boolean column#9021
MaxFreedomPollard wants to merge 1 commit into
lance-format:mainfrom
MaxFreedomPollard:fix/deepcopy-sliced-offset

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown

deep_copy_array_data_sliced in rust/lance-arrow/src/deepcopy.rs copied from the wrong place: mutable.extend(0, data.offset(), data.offset() + data.len()). MutableArrayData::extend already indexes from the source array's first logical element. ArrayData::buffer hands the value extenders a slice that starts at self.offset, and the null-bit extender adds nulls.offset() itself. Passing the offset in as start applies it a second time, so the copy begins offset elements past the slice.

Most arrays hide this, because to_data() folds a slice into the buffer pointer and leaves ArrayData::offset at 0. A boolean array cannot: a bit offset has no byte pointer to fold into, so From<BooleanArray> for ArrayData sets .offset(array.values.offset()) and a sliced boolean column keeps a non-zero offset. Its deep copy comes back shifted by that many bits, values and validity alike.

Two public entry points reach it. RecordBatchExt::shrink_to_fit is a direct call into deep_copy_batch_sliced. And rechunk_stream_by_size_deep_copy deep-copies every slice it produces, which is how HardCapBatchSizeExec caps batch sizes ahead of a DataFusion sort, so a large batch with a boolean column comes back through that node with the wrong booleans and no error.

The fix is to copy from 0. The regression test slices a boolean array that has nulls at offset 3, asserts the slice really does carry ArrayData::offset == 3, and compares the copy against the slice element by element.

Verification, run from rust/ on 1.97.0:

cargo test -p lance-arrow passes, 102 unit tests and 6 doctests. Reverting just the one-line fix and keeping the test makes test_deep_copy_array_sliced_boolean_keeps_offset fail with left: [Some(true), Some(false), Some(true), Some(true)] against right: [Some(true), Some(true), Some(false), Some(true)], while the five deepcopy tests that were already there still pass, which is why this went unnoticed.

cargo clippy -p lance-arrow --all-targets --all-features -- -D warnings is clean, and so is cargo fmt --all -- --check.

deep_copy_array_data_sliced called mutable.extend(0, data.offset(), data.offset() + data.len()). MutableArrayData::extend already indexes from the source array's first logical element: ArrayData::buffer hands the value extenders a slice starting at self.offset, and the null-bit extender adds nulls.offset() itself. Passing the offset in as start applies it twice, so the copy begins offset elements past the slice.

Most arrays hide this because to_data() folds a slice into the buffer pointer and leaves ArrayData::offset at 0. A boolean array cannot, since a bit offset has no byte pointer to fold into, so From<BooleanArray> for ArrayData sets .offset(array.values.offset()) and a sliced boolean column keeps a non-zero offset. Its deep copy comes back shifted by that many bits, values and validity alike.

Two public entry points reach it: RecordBatchExt::shrink_to_fit calls deep_copy_batch_sliced directly, and rechunk_stream_by_size_deep_copy deep-copies every slice it produces, which is how HardCapBatchSizeExec caps batch sizes ahead of a DataFusion sort.

Copy from 0 instead, and cover it with a sliced boolean array that carries a non-zero ArrayData::offset.
@github-actions github-actions Bot added the bug Something isn't working label Sep 6, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The fix restores the logical-slice contract: MutableArrayData::extend receives 0..len, so non-zero-offset boolean values and validity are copied once into compact buffers. The regression test covers the offset case, and the shared copy path remains correct for existing array types.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant