Skip to content

fix(datagen): align random primitive buffers - #8776

Merged
Xuanwo merged 2 commits into
mainfrom
gatekeeper/fix-7911-1
Sep 2, 2026
Merged

fix(datagen): align random primitive buffers#8776
Xuanwo merged 2 commits into
mainfrom
gatekeeper/fix-7911-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • allocate random primitive bytes in an Arrow cache-line-aligned mutable buffer
  • cover empty and nonempty Float16, Decimal128, and Decimal256 generation

Root cause

RandomBytesGenerator filled a Vec and reinterpreted it as wider Arrow native values. Vec only guarantees byte alignment, and its empty dangling pointer is deterministically misaligned for these types.

Fix

Use Arrow MutableBuffer storage, which remains cache-line aligned for both empty and allocated buffers, before constructing the typed ScalarBuffer.

Validation

  • cargo test -p lance-datagen
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #7911

@github-actions github-actions Bot added A-deps Dependency updates bug Something isn't working labels Aug 26, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The change fixes the reported alignment failure at the allocation boundary: Arrow-managed mutable storage supplies aligned pointers for both empty and populated buffers while preserving the generator’s existing byte and declared-type semantics.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 26, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: The Ready PR remains blocked by two unrelated current-head CI failures.

Remote head fa7b17f53040fd213d5d180c37c008f93a4cf3e0 contains current main tip 7b801361cf4448a19c31b9e2cf58c1227591c1bd. mac-build (stable) failed test_build_ivf_pq_4bit::case_1_l2 because recall was 0.4 below the 0.5 threshold; the unchanged test passed in focused reproduction and the current-base macOS job succeeded. Python windows failed test_torch_index_with_nans[V3] with StopIteration; the failing files are unchanged, and the current-base Windows run passed this exact test while failing a separate Torch temporary-cache permission path. All 17 lance-datagen tests, including the alignment regression, pass, so there is no evidence-backed repair attributable to either failure.

Smallest next action: re-run the two failed jobs on this head. Alternatively, investigate and fix the independent vector-index flakes separately if either reproduces consistently, then re-run this PR.

@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The merge from current main leaves the reviewed patch unchanged. Arrow-managed mutable storage still fixes the alignment contract for both empty and populated buffers while preserving generated bytes and declared types.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@Xuanwo
Xuanwo merged commit 901edc1 into main Sep 2, 2026
38 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-7911-1 branch September 2, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lance-datagen panics when generating empty Float16 or decimal arrays

1 participant