feat: stabilize field IDs across schema evolution

docs/src/format/table/index.md

@@ -28,6 +28,12 @@ A manifest describes a single version of the dataset.
 It contains the complete schema definition including nested fields, the list of data fragments comprising this version, 
 a monotonically increasing version number, and an optional reference to the index section that describes a list of index metadata.
 
+`max_allocated_field_id` is optional allocator state. Its presence activates the stable field-ID
+contract for this branch ancestry. It is a high-water mark: every non-negative field ID in the
+canonical manifest schema, base data-file mappings, and overlay mappings must be less than or equal
+to it, and a successor manifest must never lower it. Schema-preserving rewrites retain the value;
+new identities allocate above it with checked arithmetic. See [Field IDs](schema.md#field-ids).
+
 <details>
 <summary>Manifest protobuf message</summary>
 

docs/src/format/table/schema.md

@@ -221,16 +221,51 @@ Assigned IDs with parent relationships:
 Note: A `parent_id` of -1 indicates a top-level field. For nested fields, `parent_id` references the ID of the parent field. Child fields reference their parent via `parent_id` rather than being stored as separate "children" arrays in the protobuf message (though the Rust in-memory representation maintains a children vector for convenience).
 
 **New field assignment (incremental):**
-When fields are added later (e.g., through schema evolution), they receive the next available ID
-incrementally. This preserves the history of field additions.
+When fields are added later (e.g., through schema evolution), they receive incrementally assigned
+IDs. On a dataset with the stable field-ID contract activated, allocation starts immediately after
+the manifest's persistent `max_allocated_field_id`; IDs retired by drop or replacement are never
+reused.
+
+Field-ID metadata supplied on an incoming Arrow schema is not an allocation authority. When that
+schema is merged into an activated dataset, IDs for newly introduced logical fields are cleared and
+assigned by the dataset allocator; callers cannot select or reserve IDs through Arrow metadata.
+
+The presence of `Manifest.max_allocated_field_id` is the activation marker. If it is absent, the
+dataset has legacy allocation semantics and an implementation may derive the next ID from fields
+still referenced by the current snapshot. Activation initializes the high-water mark from the
+maximum non-negative ID referenced by the canonical manifest schema, base data files, and overlay
+files. Activation provides a forward guarantee only; it cannot reconstruct identities that were
+dropped or reused in older snapshots.
+
+New datasets activate this contract in their initial manifest and set the corresponding reader and
+writer feature bits. The reader bit is a compatibility fence for released runtimes whose generic
+commit path did not enforce unknown writer bits; stable field IDs do not otherwise change how fields
+are read. Existing legacy datasets remain unchanged until an explicit migration commit. Explicit
+migration sets only the writer bit and requires operators to retire older writers before activation.
+The reader fence, once set by automatic activation, is retained by every later commit, clone, and
+restore in that branch ancestry.
+
+Activation is one-way within a branch ancestry. After migration, restore cannot target a version
+from before activation because that version does not carry the high-water mark needed to preserve
+retired identities.
 
 ### Field ID Properties
 
-- **Immutable**: Once assigned, a field's ID never changes
+- **Immutable after activation**: An identity keeps its ID throughout the activated branch ancestry
 - **Unique**: Each field within a table has a unique ID
-- **Stable**: IDs are preserved across schema evolution operations
+- **Never reused after activation**: Dropped and replaced identities permanently retire their IDs
+- **Monotonic**: New identities are allocated densely above the persistent high-water mark
 - **Sparse**: Field IDs may not form a contiguous sequence after schema evolution
 
+The guarantee is scoped to one dataset branch ancestry. Field IDs are not globally unique across
+datasets or independently evolving branches. A persistent cross-dataset or cross-branch reference
+must carry the corresponding dataset and ancestry identity.
+
+Two branches may allocate the same integer after their common ancestor. A future branch-merge
+operation must reject the merge when both sides created post-ancestor identities whose bindings
+differ; it must not silently choose one binding, renumber already-persisted fields, or merge them by
+name. Branch-local collision avoidance requires a separate allocator design.
+
 ### Using Field IDs
 
 When referencing fields internally within the format, use the field ids rather than field names or positions.
@@ -296,13 +331,29 @@ The complete schema is represented as a collection of top-level fields plus meta
 Field IDs enable efficient schema evolution:
 
 - **Add Column**: Assign a new field ID and add to schema
-- **Drop Column**: Remove field from schema; its ID may be reused in some systems
+- **Drop Column**: Remove the field and permanently retire its ID after activation
 - **Rename Column**: Change field name; ID remains the same
 - **Reorder Columns**: Change field order in schema; IDs remain the same
-- **Type Evolution**: Data type can be changed. This might require rewriting the column in the data, depending on how the type was changed.
+- **Metadata or Nullability Change**: Preserve the field ID
+- **Type Replacement**: Allocate a new field ID and retire the old identity
+- **Overwrite**: Preserve compatible logical identities; allocate new IDs for added fields and type
+  replacements
 
 The use of field IDs ensures that data files can be correctly interpreted even as the schema changes over time.
 
+### Blob Identity Namespace
+
+A Blob column's canonical logical fields in the manifest schema participate in stable field-ID
+allocation. The top-level Blob field is the public binding identity; logical children already
+present in the manifest schema are also allocated and retired normally.
+
+Blob writer-prepared fields and stored descriptor fields are representation details. Synthetic
+children such as `kind`, `blob_id`, `blob_size`, `position`, `size`, and `blob_uri` do not enter the
+dataset field-ID namespace unless they are part of the canonical manifest schema or a
+`DataFile.fields` mapping. Their IDs may remain `-1` or use a file-local namespace, and they do not
+advance `max_allocated_field_id`. The `blob_id` value identifies a sidecar object and is unrelated
+to schema field IDs.
+
 ## Example Schemas
 
 The examples below use a simplified representation of the field structure. In the actual protobuf format, `type` refers to the field type enum (PARENT/REPEATED/LEAF) and `logical_type` contains the data type string representation.

docs/src/format/table/versioning.md

@@ -31,7 +31,9 @@ they should return an "unsupported" error on any read or write operation.
 | 32       | `FLAG_DISABLE_TRANSACTION_FILE` | No              | Yes             | Transactions are recorded in the manifest rather than in a separate transaction file.                       |
 | 64       | `FLAG_UNSTABLE_DATA_OVERLAY_FILES` | Yes          | Yes             | Fragments may carry data overlay files. Unstable: release builds reject it unless explicitly opted in.      |
 | 128      | `FLAG_COVERED_INDEX_METADATA`   | Yes             | Yes             | Some index declares covering columns (`IndexMetadata.covering_fields`), so `fields` means keyed columns followed by carried ones. An implementation without this flag selects an index by membership of `fields` and would answer a query on a merely-carried column with an index keyed on a different one. |
+| 256      | `FLAG_MIXED_DATA_FILE_VERSIONS` | Yes             | Yes             | Reserved for datasets that may reference recognized V2 data files with different exact versions.           |
+| 512      | `FLAG_STABLE_FIELD_IDS`         | Automatic activation | Yes         | The manifest carries a persistent field-ID high-water mark that writers must use when allocating field IDs. Automatic activation also fences released runtimes whose write paths did not consistently enforce writer flags; explicit migration is writer-only. See [Field IDs](schema.md#field-ids). |
 
 </div>
 
-Flags with bit values 256 and above are unknown and will cause implementations to reject the dataset with an "unsupported" error.
+Flags with bit values 1024 and above are unknown and will cause implementations to reject the dataset with an "unsupported" error. The reserved mixed-data-file-version bit remains unsupported until its storage contract lands.