fix(encoding): reject out-of-bounds variable-width offsets on decode

[Xuanwo]

Variable-width block decoding trusted offsets that come straight from file bytes, so a file with corrupt offsets could be accepted by the default reader configuration: VariableWidthBlock::into_arrow used Arrow's unchecked builder unless the optional validate_on_decode flag was enabled, letting out-of-bounds offsets escape into Arrow arrays (returning bytes outside the values buffer, or crashing on access). The binary mini-block and block decompressors additionally sliced buffers with unvalidated offsets and header fields, which could panic the decode task.

This change makes layout validation mandatory at the Arrow conversion boundary: offsets must be monotonic and within the data buffer, and Utf8 values must be valid UTF-8. Violations surface as a typed CorruptFile error before any batch is returned, and validate_on_decode can no longer bypass the check — a private proof type ties the remaining unchecked build to the validation pass. The decompressors keep only the structural checks needed to avoid panics (plus the first-offset-must-be-zero block contract); offset values are validated once, at the conversion boundary, rather than rescanned per layer.

The memory-safety validation is unconditional by design, so the implementation keeps it off the critical path: the hot loop is a branchless monotonicity scan (monotonic + in-bounds endpoints imply all offsets are in bounds), and the mini-block check rides the existing rebase loop instead of adding a pass. Decode microbenchmarks on this change (dictionary strings, plain high-cardinality utf8 and binary, 1M rows) show no measurable throughput regression against the pre-fix baseline.

Supersedes #8143: its offset-value checks are covered by the conversion-boundary validation, and its structural checks (exact offsets-region size, zero first offset) are folded into the block decompressor here.

[lance-gatekeeper]

Gate recommendation: request changes. The central conversion validator correctly covers the malformed Arrow layouts exercised here, but miniblock decoding still lacks a structural boundary that lets corrupted metadata become user data. Validate the value-region start before slicing and add 32/64-bit overlap cases; that preserves prefix reads while completing the corrupt-file contract.

[lance-gatekeeper]

The first offset can still point inside the offset table, so a corrupt miniblock is accepted and serialized offsets are returned as string data instead of CorruptFile. Before slicing, require first >= num_offsets * bytes_per_offset; use a lower bound rather than equality because prefix decoding can leave unrequested offsets before the values.

Reproducer

I added this regression test to the existing test module and ran cargo test -p lance-encoding gate_repro_miniblock_offset_overlaps_offset_table --lib on this head:

#[test]
fn gate_repro_miniblock_offset_overlaps_offset_table() {
    use crate::compression::MiniBlockDecompressor;

    let mut chunk = [0_u32, 21, 25, 30]
        .iter()
        .flat_map(|offset| offset.to_le_bytes())
        .collect::<Vec<_>>();
    chunk.extend_from_slice(b"alphabetagamma");
    chunk.resize(chunk.len().next_multiple_of(8), 0);

    let block = super::BinaryMiniBlockDecompressor::new(32)
        .decompress(vec![LanceBuffer::from(chunk)], 3)
        .expect("decoder currently accepts the overlapping offset");
    assert!(block.into_arrow(DataType::Utf8, false).is_err());
}

Expected the assertion to pass; observed it fail because into_arrow returned Ok. Add equivalent 32-bit and 64-bit coverage.