Skip to content

Spark 4.2: Implement SupportsSchemaEvolution for automatic schema evolution - #17957

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:spark-4.2-schema-evolution-capability
Open

Spark 4.2: Implement SupportsSchemaEvolution for automatic schema evolution#17957
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:spark-4.2-schema-evolution-capability

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Implements Spark 4.2's SupportsSchemaEvolution mix-in on Iceberg's SparkTable, so Iceberg participates in DSv2 automatic schema evolution during writes.

During automatic schema evolution Spark passes each candidate column change to supportsColumnChange(TableChange.ColumnChange) and skips the ones the source reports as unsupported. SparkTable now reports support consistent with what Iceberg's schema update actually applies:

  • add column: only when nullable, without a default, and with a Spark type Iceberg can convert;
  • update type: only when the new type is a primitive and an allowed Iceberg type promotion;
  • update nullability: only when relaxing to nullable;
  • delete column: rejected for identifier fields and for any field whose subtree contains an identifier field (matching SchemaUpdate), and for unknown fields;
  • rename, comment, and position changes are supported.

SparkTable already advertises TableCapability.AUTOMATIC_SCHEMA_EVOLUTION (gated by the write.spark.auto-schema-evolution.enabled table property), which is what makes Spark consult this hook. A unit test is added in TestSparkTable.

…lution

Implement Spark 4.2's SupportsSchemaEvolution mix-in on SparkTable so Iceberg participates in DSv2 automatic schema evolution during writes. supportsColumnChange reports support consistent with what Iceberg's schema update applies: nullable, no-default column additions of a convertible type; primitive type promotions allowed by Iceberg; relaxing nullability; column deletes except identifier fields and any field whose subtree contains one; and rename, comment, and position changes. Adds a TestSparkTable unit test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant