feat: stabilize field IDs across schema evolution by Xuanwo · Pull Request #8658 · lance-format/lance

Xuanwo · 2026-08-20T09:44:58Z

Field IDs are physical bindings in Lance, but legacy allocation derives the next ID from fields that the current snapshot still references. After a field and its files disappear, a later field can reuse the same integer. A bare field ID is therefore not a durable identity across schema evolution.

This PR adds an explicit, writer-side stable field-ID contract:

New and existing datasets keep legacy behavior until a Rust caller runs migrate_to_stable_field_ids.
Migration writes both max_allocated_field_id and FLAG_STABLE_FIELD_IDS. Both values must be present for stable mode or absent for legacy mode; either mismatch is invalid. Stable field IDs do not set a reader flag.
In stable mode, each new or replacement field ID must be greater than the previous high-water mark. IDs may be sparse, which leaves room for future reservation.
Rename, reorder, metadata changes, and compatible overwrite preserve identity. Type replacement allocates a new identity.
Incoming Arrow field-ID metadata is descriptive input, not allocation authority for a new field in stable mode. Commit canonicalization assigns IDs and updates new file mappings.
Activation is one-way. Restore to a pre-activation version is rejected because that version does not record retired IDs.

The migration API is Rust-only in this PR. Python and Java enforce the contract when they operate on an activated dataset, but they do not expose activation yet.

The format documentation also defines how Blob logical fields participate in dataset field identity while writer-prepared and stored descriptor children remain file-local representation details.

Validation includes Rust stable/legacy migration, schema evolution, retry/rebase, restore, clone, detached commit, Blob identity, and Arrow canonicalization coverage, plus Java Merge, Project, and Overwrite boundary coverage.

github-actions · 2026-08-20T09:45:06Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

westonpace · 2026-08-20T16:59:43Z

Do you think this will cause transactions to conflict with each other that did not before? I don't think it does today since we treat any schema change as conflicting with any other schema change.

Do we know why we reused field ids in the first place?

codecov · 2026-08-20T19:29:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Xuanwo · 2026-08-31T18:07:57Z

Agreed that stable field IDs are fundamentally a writer-side contract. The reader bit was only intended to fence legacy writers, so I’ll keep activation explicit and align the commit-authoritative allocator with Composite Transactions’ temporary-ID model.

wjones127

I haven't gotten to the implementation yet, just read the format changes. I'm glad it's a writer flag only.

The prose is pretty difficult to read. I asked Claude to help suggest how to rewrite it, taking inspiration from the ASD-STE100 writing standard.

I also have some questions

wjones127 · 2026-09-01T16:24:01Z

+schema is merged into an activated dataset, IDs for newly introduced logical fields are cleared and
+assigned by the dataset allocator; callers cannot select or reserve IDs through Arrow metadata.
+
+The presence of `Manifest.max_allocated_field_id` is the activation marker. If it is absent, the


question(blocking): what's the difference between setting this field and setting the writer flag? What happens if you set this field but not the writer flag? What happens if you set the writer flag, but not this field?

The field stores allocator state, while the flag gates writers; both must be set together, and either mismatch is invalid.

wjones127 · 2026-09-01T16:26:03Z

 - **Reorder Columns**: Change field order in schema; IDs remain the same
- **Type Evolution**: Data type can be changed. This might require rewriting the column in the data, depending on how the type was changed.
+- **Metadata or Nullability Change**: Preserve the field ID
+- **Type Replacement**: Allocate a new field ID and retire the old identity


question(non-blocking): is this new behavior or just a description of existing behavior? I wonder if this is really desireable.

This is existing cast behavior: changing the logical type creates a new field identity, so it receives a new ID.

lance-gatekeeper

❌ Gate recommendation: request changes.

1 fixed / 1 new. The field/flag validity contract is now explicit, but this revision removes the retire-before-migration precondition based on the resolved compatibility discussion. #8580 added generic writer-feature checks only in v12.0.0-beta.4; older writers can still open this writer-only-gated dataset, and their generic schema commits did not reject or preserve the unknown stable-ID state. Such a writer can republish an activated table without its high-water contract and re-enable ID reuse. Explicit migration changes who opts in, not the need to quiesce those writers first.

Restore the requirement to retire pre-gate writers before migrate_to_stable_field_ids, both in the public API documentation and the operator-facing format/versioning documentation.

wjones127

The spec looks a lot better. Reviewed the implementation this time.

wjones127 · 2026-09-04T23:41:01Z

+The writer may temporarily add `kind`, `blob_id`, `blob_size`, and `position`. A Lance data file
+stores this descriptor shape:
+
+```python
+pa.schema([
+    pa.field(
+        "image",
+        pa.struct([
+            pa.field("kind", pa.uint8(), nullable=False),
+            pa.field("position", pa.uint64(), nullable=False),
+            pa.field("size", pa.uint64(), nullable=False),
+            pa.field("blob_id", pa.uint32(), nullable=False),
+            pa.field("blob_uri", pa.string(), nullable=False),
+        ]),
+    ),
+])
+```


question(non-blocking): when these fields are stored, how is that represented in both DataFile.fields and DataFile.column_indices? Below it says DataFile.fields = [0]. Are they really missing from the DataFile metadata? If so, how does a reader know where in the Lance file to find those columns?

wjones127 · 2026-09-04T23:42:04Z

praise: this looks a lot better! Nice work.

wjones127 · 2026-09-04T23:44:56Z

+
+#[tokio::test]
+async fn test_raw_arrow_overwrite_preserves_reordered_stable_field_identities() {
+    let source_uri = TempStrDir::default();


nitpick: For tests that aren't validating disk IO, I prefer to use the memory:// URI.

wjones127 · 2026-09-04T23:46:47Z

+    let schema = Arc::new(ArrowSchema::new(vec![
+        ArrowField::new("a", DataType::Int64, false),
+        ArrowField::new("b", DataType::Int64, false),
+    ]));
+    let batch = RecordBatch::try_new(
+        schema.clone(),
+        vec![
+            Arc::new(Int64Array::from(vec![1, 2])),
+            Arc::new(Int64Array::from(vec![3, 4])),
+        ],
+    )
+    .unwrap();


nitpick: prefer to use record_batch!() macro for simple data, ¹

Suggested change

let schema = Arc::new(ArrowSchema::new(vec![

ArrowField::new("a", DataType::Int64, false),

ArrowField::new("b", DataType::Int64, false),

]));

let batch = RecordBatch::try_new(

schema.clone(),

vec![

Arc::new(Int64Array::from(vec![1, 2])),

Arc::new(Int64Array::from(vec![3, 4])),

],

)

.unwrap();

let batch = record_batch!(

("a", Int64, [1, 2]),

("b", Int64, [3, 4])

).unwrap();

Footnotes

https://github.com/lance-format/lance/blob/main/rust/AGENTS.md#testing ↩

wjones127 · 2026-09-04T23:51:30Z

+        let err = validate_operation(Some(&manifest), &reused).unwrap_err();
+        assert!(


suggestion: could we also assert which error variant this is emitting? I think it would be InvalidInput, right?

wjones127 · 2026-09-04T23:57:02Z

+            let Some(restored_max_field_id) = manifest.max_allocated_field_id else {
+                return Err(Error::invalid_input(format!(
+                    "Cannot restore version {version}: stable field IDs were activated after that version"
+                )));
+            };


question: remind me, why can't we restore? What bad happens if we let users restore to a previous state?

wjones127 · 2026-09-05T00:00:47Z

+/// Canonicalize schema identities supplied by a transaction before validation.
+///
+/// Arrow field-ID metadata is descriptive input, not allocation authority. New
+/// datasets allocate from zero, while stable datasets preserve compatible
+/// existing identities and allocate every new identity above the persisted
+/// high-water mark. File mappings written against the incoming schema are
+/// updated in the same step; files retained by a merge are never rewritten.


suggestion: it's unclear what "canonicalize" means. Or what "schema identities" means. Could we just call this "Assign field ids given an operation". And then we can list the rules below that for how field ids are assigned?

"allocate", "identity", "canonicalize" are all new words we haven't really used. "Assign a field id" is something familiar.

wjones127 · 2026-09-05T00:05:05Z

+        let mut current_id = i64::from(schema_max_id.max(max_existing_id)) + 1;
+        let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i64;
+        if unassigned_count > 0 && current_id + unassigned_count - 1 > i64::from(i32::MAX) {
+            return Err(Error::invalid_input(
+                "No further field ID can be allocated because IDs are exhausted",
+            ));
+        }


suggestion: could we keep in the i32 domain by using saturating arithmetic?

Suggested change

let mut current_id = i64::from(schema_max_id.max(max_existing_id)) + 1;

let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i64;

if unassigned_count > 0 && current_id + unassigned_count - 1 > i64::from(i32::MAX) {

return Err(Error::invalid_input(

"No further field ID can be allocated because IDs are exhausted",

));

}

let mut current_id = schema_max_id.max(max_existing_id).saturating_add(1);

let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i32;

if unassigned_count > 0 && (current_id - 1).saturating_add(unassigned_count) > i32::MAX {

return Err(Error::invalid_input(

"No further field ID can be allocated because IDs are exhausted",

));

}

wjones127 · 2026-09-05T00:06:48Z

+            return Err(Error::invalid_input(
+                "No further field ID can be allocated because IDs are exhausted",
+            ));


suggestion: it would be nice to not have to traverse the whole schema up front to discover this. Couldn't we discover this while we are allocating ids? I suppose the downside is we would error right when we are in the middle of allocating ids, and leave the schema in an in-between state.

We actually do check it when allocating. Do we really need this then?

wjones127 · 2026-09-05T00:08:43Z

+/// Transient schema-metadata marker used by bindings for raw Arrow input.
+pub const TRANSACTION_SCHEMA_SOURCE_RAW_ARROW: &str = "lance:transaction_schema_source_raw_arrow";


question: what's the purpose of this? When is this necessary?

feat: stabilize field IDs across schema evolution

08bebbe

github-actions Bot added the enhancement New feature or request label Aug 20, 2026

Xuanwo marked this pull request as ready for review August 20, 2026 09:47

github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-format On-disk format: protos and format spec docs labels Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

fix: canonicalize stable field ids at commit boundaries

2964690

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

fix: enforce stable field ids in namespace rewrites

7e46c86

github-actions Bot added the A-namespace Namespace impls label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

wjones127 self-requested a review August 20, 2026 16:07

fix: preserve stable identities across schema operations

7d59c81

Xuanwo added 3 commits August 21, 2026 01:50

fix: reject ambiguous stable field mappings

2b45dbe

fix: centralize binding field id remapping

8851484

fix(java): preserve legacy field id remapping

bb42780

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

Xuanwo added 2 commits August 21, 2026 03:48

fix(java): validate project field identities

d199a9a

test(java): verify legacy project values

3b041d1

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 20, 2026

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

fix: fence automatic stable field ID activation

d952779

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

Xuanwo added 2 commits September 1, 2026 01:05

Merge remote-tracking branch 'origin/main' into xuanwo/stable-field-ids

4ced65b

fix: align stable field IDs in overwrite fragments

399260c

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

Xuanwo requested a review from wjones127 August 31, 2026 18:08

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 31, 2026

fix: enforce stable field ID commit boundaries

6fde30f

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 1, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 1, 2026

wjones127 reviewed Sep 1, 2026

View reviewed changes

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026

fix: align stable field ID activation contract

678e855

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026

lance-gatekeeper Bot reviewed Sep 2, 2026

View reviewed changes

lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Sep 2, 2026

wjones127 requested changes Sep 5, 2026

View reviewed changes

		let err = validate_operation(Some(&manifest), &reused).unwrap_err();
		assert!(

		/// Transient schema-metadata marker used by bindings for raw Arrow input.
		pub const TRANSACTION_SCHEMA_SOURCE_RAW_ARROW: &str = "lance:transaction_schema_source_raw_arrow";

Uh oh!

Conversation

Xuanwo commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

westonpace commented Aug 20, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov Bot commented Aug 20, 2026

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Xuanwo commented Aug 31, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Aug 20, 2026 •

edited

Loading