direct: Fix spurious recreate of schemas and volumes with mixed-case names#5531
Merged
Conversation
The direct engine saves local config to state, so when the UC API normalizes an identifier (lowercases catalog/schema/volume names) or strips a trailing slash from a storage URL, the next plan compares the original local value against the normalized remote value and triggers a spurious recreate/update. On every second deploy with no config changes, schemas and volumes were recreated/renamed. Previously this was suppressed with per-resource OverrideChangeDesc methods. Replace those with two declarative resources.yml keys, normalize_case and normalize_slash, parsed at runtime like the other lifecycle rules and applied in addPerFieldActions before recreate/update classification. Adding a new UC resource is now a few YAML lines instead of a Go method. - config.go: NormalizeCase / NormalizeSlash on ResourceLifecycleConfig - bundle_plan.go: shouldSkipNormalized wired into the skip-ladder - schema.go, volume.go: delete OverrideChangeDesc; declare fields in resources.yml - testserver: lowercase schema/volume identifier names to mimic UC - schema_uppercase_name invariant config is schema-only (the migrate invariant deploys via Terraform first, and the TF provider rejects an uppercase volume schema_name); volume coverage lives in resources/volumes/uppercase-name Co-authored-by: Isaac
Co-authored-by: Isaac
volume_uppercase_name covers the uppercase no-drift case on direct via the no_drift invariant; excluded from migrate (Terraform deploys first and rejects an uppercase volume schema_name). The dedicated schemas/uppercase-name test is redundant with the no_drift invariant, so remove it. Co-authored-by: Isaac
denik
changed the title
Co-authored-by: Isaac
pietern
approved these changes
Jun 10, 2026
|
|
||
| # migrate deploys via Terraform first, and the TF provider rejects an uppercase | ||
| # volume schema_name ("inconsistent final plan"). Covered by no_drift on direct. | ||
| EnvMatrixExclude.no_volume_uppercase = ["INPUT_CONFIG=volume_uppercase_name.yml.tmpl"] |
Contributor
There was a problem hiding this comment.
Interesting; this error doesn't happen for schemas?
shouldSkipNormalized returned a hardcoded reason and ignored the per-field
reason from resources.yml, unlike every other rule. Route the config reason
through (like recreate_on_changes etc.) and drop the now-unused
Reason{UC,URL}Normalization constants.
Record the relevant plan changes (field action + reason) in the
volumes/uppercase-name test so the skip classification is visible.
Co-authored-by: Isaac
Collaborator
|
Commit: 0169c0c
22 interesting tests: 15 SKIP, 7 KNOWN
Top 28 slowest tests (at least 2 minutes):
|
&>> is bash 4+ syntax; macOS CI runners use the stock bash 3.2, which fails
to parse the whole script ("syntax error near unexpected token '>'") and
produces no output files. Use >> ... 2>&1 instead.
Co-authored-by: Isaac
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Jun 10, 2026
## Release v1.3.0 ### Notable Changes * The `direct` deployment engine is now Generally Available and the default for new deployments. To opt out, set `engine: terraform` under `bundle` in your `databricks.yml` or set `DATABRICKS_BUNDLE_ENGINE=terraform`. Existing deployments keep their current engine; see https://docs.databricks.com/aws/en/dev-tools/bundles/direct to migrate. ### CLI * Added the `databricks quickstart` command, a short introduction to the CLI that prints a human-friendly guide interactively and an agent-oriented version when run non-interactively ([#5464](#5464)). * Add `databricks version --check` to report whether a newer CLI version is available and print the upgrade command for the detected install method ([#5469](#5469)). * `databricks auth describe` now verifies credentials against both the workspace and account endpoints before reporting a failure, fixing false "Unable to authenticate" errors for account console profiles ([#5479](#5479)). * `databricks auth login` no longer prompts for workspace selection when logging in to an account console host (`https://accounts.*`). Pass `--workspace-id` explicitly to store a workspace ID on such a profile ([#5504](#5504)). * `databricks auth profiles --skip-validate` no longer makes any network calls; the host metadata fetch is skipped along with validation ([#5530](#5530)). ### Bundles * Set the default `data_security_mode` to `DATA_SECURITY_MODE_AUTO` in bundle templates ([#5452](#5452)). * Mark vector search index index_subtype as backend_default to prevent drift after deployment ([#5454](#5454)). * `bundle deployment migrate`: handle resources added to or removed from `databricks.yml` since the last Terraform deploy ([#5463](#5463)). * Add the `genie_spaces` bundle resource for managing Databricks Genie spaces as code, plus `bundle generate genie-space` to import an existing space. Direct deployment engine only ([#5282](#5282)). * Fix spurious recreate of schemas and volumes whose names use mixed case ([#5531](#5531)).
janniklasrose
added a commit
that referenced
this pull request
Jun 11, 2026
The GitHub web conflict editor left the <<<<<<< / ======= / >>>>>>> markers in resources.yml, breaking YAML parsing (lint, direct tests, and validate-generated). Keep both sides: backend_defaults from this branch plus normalize_case/normalize_slash from main (#5531), ordered to match the volumes section. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The direct engine saves local config to state, so when Unity Catalog lowercases a schema/volume identifier, the next deploy sees the original mixed-case value drift against the normalized remote value and recreates/renames the resource — on every second deploy with no config changes. This PR fixes that.
The fix is expressed declaratively via a new
resources.ymlkey,normalize_case. The same mechanism also absorbs the existing trailing-slashstorage_locationsuppression (#5145) as a second key,normalize_slash— a refactor of behavior that already lived in per-resourceOverrideChangeDescmethods, not a new behavior.