BlobV2Descriptor Structured Extension — Fine-Grained Random Access for Multimodal Blob

[wenxuanguan]

Background & Motivation

Blob v2 already provides efficient storage and access for large binary objects. However, the current BlobV2Descriptor only records the blob's overall position and size (kind / position / size / blob_id / blob_uri) and is unaware of the blob's internal structure.

This limits Lance's applicability in multimodal scenarios:

• Video: extracting the N-th frame necessitates a decoding utility (e.g., ffmpeg) to access both the target frame and its reference frames, incurring multiple I/O accesses and precluding batch retrieval of the target frame sequence within a single I/O operation.
• Image: cannot access a specific page of a multi-page image or a specific region of a huge image in one I/O pass.
• Audio: cannot on-demand extract a segment for a given time range within a single I/O operation.

Building on prior discussions, #7174 points out that the Blob v2 descriptor only expresses physical storage location and cannot support richer logical objects such as multiple rows sharing the same object, delta encoding, or chunked blobs. It proposes an Object Layer abstraction that decouples physical location from logical representation through a unified resolve → plan → read → decode flow, enabling domain-specific blob layouts (e.g., GOP-level video storage) and cross-row object sharing (I/O coalescing). #4320 points out that the Blob API treats video as unstructured binary data and cannot perceive video metadata (e.g., frame rate); it proposes a video encoding extension that pluggably supports "encode-on-write" and "background progressive rewrite (at GOP level)" while retaining the native strengths of columnar storage.

I'd like to suggest a minimal, forward-compatible BlobV2Descriptor extension on top of these discussions, so that Lance can perceive a blob's internal structure and support fine-grained random access, while keeping the Lance core free of any codec dependency.

Core Design: A Minimal Extension to BlobV2Descriptor

On top of the existing 5 native fields, the only addition is a nullable sub-field blob_info, which carries the metadata and internal-structure index for various multimodal blobs. Taking video as an example, the full structure is defined as follows:

BlobV2Descriptor  
├── kind : u8 (unchanged)  
├── position : u64 (unchanged)  
├── size : u64 (unchanged)  
├── blob_id : u32 (unchanged)  
├── blob_uri : string (unchanged)  
└── blob_info : struct<...> (nullable) ← new  
    ├── blob_type : u8 (1=video / 2=audio / 3=image / ...)  
    ├── stream_info : ... (metadata)  
    └── entries : ... (position index entries)  

The design supports a variety of multimodal data formats without hardcoding specific metadata schemas into the Lance core. The blob_info field provides a generic and extensible container for auxiliary metadata. Lance is responsible only for storing and retrieving this payload; it does not need to parse or understand its internal structure. The actual content of stream_info and entries is an opaque payload—its schema and semantics are fully defined and handled by the client application. For video, this might include stream‑level metadata(e.g., fps), and GOP/Frame offsets. For image, it could contain page‑level metadata such as page count, and index entries for a multi-page document.

Architecture: How It Maps Onto the Object Layer

This proposal follows the implementation path of #7174 Object Layer. It adheres to resolve → plan → read → decode, using video frame extraction as an example.

Mapping of the blob read path:

Object Layer stage	This proposal's implementation	Core value
resolve	Read blob_info and other metadata from the BlobV2Descriptor	Obtain video metadata plus GOP/frame info
plan	Extract the GOP/frame index from blob_info.entries, compute the physical read range — from the GOP's I-frame up to the target frame, including all decode dependencies	Cross-row I/O coalescing: multiple overlapping read ranges on the same GOP are merged into 1 Range GET
read	Issue the Range GET, return physical bytes	I/O volume shrinks from the full MP4 to a single GOP
decode	Call the user-supplied decoder implementation	Codec isolation: Lance core has no codec dependency

blob_info is inlined in the descriptor, so the number of I/O reads stays the same.

Data write: extend the existing write entry point to detect a video-splitting request, demux the video container, extract video metadata to populate the blob_info field of BlobV2Descriptor, and store the GOP byte stream — reusing the existing write path and committing the transaction. No video frame decoding or re-encoding is involved.

The boundary of changes is clear:

Layer	Change type	Details	Core API change?
File Format	✅ Compatible extension	BlobV2Descriptor gains 1 nullable sub-field blob_info (containing the stream_info structured sub-field and entries position index). This is a compatible extension to BlobV2Descriptor that ensures forward compatibility; not a core format API change.	No core API change
Table Format	❌ Zero change	Manifest/Fragment/DataFile are not touched at all; the existing 5 fields of BlobV2Descriptor keep their semantics; no new feature gate; no new Arrow extension type.	No core API change
Index Formats	❌ Zero change	The frame/GOP position index is descriptor sub-field data (row-level data) and does not belong to the table-level index spec; existing vector/scalar indexes are fully compatible.	No core API change
Catalog Specs	❌ Zero change	Reuse the existing _blobs/ directory and .blob files; no new directory/file type.	No API change
Namespace Client Spec	✅ New APIs	Extend write, add frame-extraction and other APIs.	API change / extension

---

Compatibility

Taking video frame‑extraction scenarios as an example:

• New readers return full original MP4 binary blobs when reading legacy blob files, and return extracted target frames when reading GOP‑organized blobs written by new writers and carrying the blob_info extension.
• Old readers must not silently interpret the newly‑written GOP‑oriented byte sequences as ordinary raw binary blobs.
• GOP‑oriented byte‑sequence data must fail explicitly on readers that lack the required feature.

The proposal decomposes MP4 files into GOP collections and stores them as GOP byte streams in the Lance Dataset. Since original MP4 container metadata is stripped during demultiplexing, legacy interfaces cannot correctly interpret these raw GOP byte sequences or reproduce a valid MP4.

Performance Gains

Taking video frame extraction as an example, the blob_info extension reduces the number of I/O operations by an order of magnitude compared to the BlobFile-based random access approach, thereby significantly lowering the I/O cost for multimodal frame-extraction workloads.

4.1 End-to-End Latency Comparison

Test setup: MP4 video stored on MinIO (133.4 MB, H.264, fps 25, 40 min duration); Ten 500 ms time windows are randomly generated, and ten corresponding keyframe extraction operations are performed.The test implementation follows the official Lance lazy video decoding example: example-decode-video-frames-lazily. In that example, the process_frame function performs RGB conversion and materializes the frame as an ndarray.

This proposal's extraction flow: The target frame sequence is located and retrieved from the blob according to the time interval, and then decoded to all keyframes using ffmpeg.

Baseline extraction flow: Using take_blobs to fetch the BlobFile, a container is built from it. The container is then seeked to the closest keyframe position based on the given start time, and all keyframes are subsequently extracted in sequential order.

Results:

Metric	This proposal	Baseline	Improvement
Avg latency	17.4 ms	67.3 ms	2.86x

4.2 Performance Breakdown

This proposal decodes from the GOP's keyframe sequentially to the target frame; the baseline computes a seek to the preceding keyframe from the frame number and fps (and related info), then decodes sequentially to the target frame. Decode times are close — the performance gap mainly comes from blob data reads.

Comparing the I/O traffic of reading blob data, the difference is significant:

Metric	This proposal	Baseline	Read reduction
Total Input Size	~1.98M	~13.91M	~6x
Number of Reads	10	183	~17x