feat: add has_non_empty_nulls helper function in OffsetBuffer#9711
Merged
alamb merged 5 commits intoApr 27, 2026
Merged
Conversation
Member
Author
|
Would appreciate a review |
alamb
reviewed
Apr 23, 2026
| return last_offset != initial_offset; | ||
| } | ||
|
|
||
| let mut valid_slices_iter = null_buffer.valid_slices(); |
Contributor
There was a problem hiding this comment.
this code would be a lot simpler if there were a NullBuffer::invalid_slices 🤔
Member
Author
There was a problem hiding this comment.
yes but I feel the code is pretty straight forward and this have comments as well that explaining the logic
| /// # Panics | ||
| /// | ||
| /// Panics if the length of the `null_buffer` does not equal `self.len() - 1`. | ||
| pub fn is_there_null_pointing_to_non_empty_value( |
Contributor
There was a problem hiding this comment.
I think this function name is a little excessively long and doesn't fit with the rest of this codebase. Can you please rename it to something shorter and more concise. Here are some ideas:
- has_null_data
- has_non_empty_nulls
- nulls_have_values
- has_populated_nulls
Member
Author
There was a problem hiding this comment.
renamed to has_non_empty_nulls
| // --------------------------------------------------------------- | ||
|
|
||
| #[test] | ||
| fn null_pointing_none_null_buffer() { |
Contributor
There was a problem hiding this comment.
Do we need all these tests? It seems like a single test for each representative case would be more than adequate and trim this PR down substantially
rluvaton
changed the title
is_there_null_pointing_to_non_empty_value helper function in OffsetBufferhas_non_empty_nulls helper function in OffsetBuffer
Rich-T-kid
pushed a commit
to Rich-T-kid/arrow-rs
that referenced
this pull request
Jun 2, 2026
…ache#9711) # Which issue does this PR close? N/A # Rationale for this change In variable-length array types (e.g., `StringArray`, `ListArray`), null entries may have non-empty offset ranges, meaning the underlying data buffer contains data behind nulls. This matters when wanting to work on the underlying values of variable length data for example when unwrapping (flattening) a list array, as the child values are exposed, including those behind null entries. If null entries point to non-empty ranges, the unwrapped values will contain data that may not be meaningful to operate on and could cause errors (e.g., division by zero in the child values). Usages when this will be helpful: - flattening list array - casting lists/map - we don't wanna cast values that are not used so this is a check if there is one - explode on list - we don't want the null values behind it so this give us a check if it exists (will have another pr to cleanup empty values) - gc on lists/map/strings to remove unneeded data # What changes are included in this PR? Add `OffsetBuffer::is_there_null_pointing_to_non_empty_value` method that checks if any null positions correspond to non-empty offset ranges # Are these changes tested? Yes # Are there any user-facing changes? Yes, a new public method `OffsetBuffer::is_there_null_pointing_to_non_empty_value` is added. ------- Related to: - apache/datafusion#18921 as it need to unwrap the list values and only get the reachable values
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
In variable-length array types (e.g.,
StringArray,ListArray), null entries may have non-empty offset ranges, meaning the underlying data buffer contains data behind nulls. This matters when wanting to work on the underlying values of variable length data for example when unwrapping (flattening) a list array, as the child values are exposed, including those behind null entries. If null entries point to non-empty ranges, the unwrapped values will contain data that may not bemeaningful to operate on and could cause errors (e.g., division by zero in the child values).
Usages when this will be helpful:
What changes are included in this PR?
Add
OffsetBuffer::is_there_null_pointing_to_non_empty_valuemethod that checks if any null positions correspond to non-empty offsetranges
Are these changes tested?
Yes
Are there any user-facing changes?
Yes, a new public method
OffsetBuffer::is_there_null_pointing_to_non_empty_valueis added.Related to: