Asset Bundle Deployment always triggers Cluster restart

[brain246]

Describe the issue

When defining (all purpose) clusters in asset bundles, the clusters will be restarted after each deployment, even if nothing has changed in the spec.

In the cluster event logs i can see a "Compute edited by " after each deployment. That log entry has a json payload that is probably supposed to show the changes, but oh well, there aren't any in my case...

Configuration

Cluster definition:

my_cluster:
  data_security_mode: DATA_SECURITY_MODE_DEDICATED
  cluster_name: 'All Purpose for me'
  single_user_name: ${workspace.current_user.userName}
  kind: CLASSIC_PREVIEW
  runtime_engine: STANDARD
  spark_version: 16.2.x-scala2.12
  node_type_id: Standard_D4ads_v5
  driver_node_type_id: Standard_D4ads_v5
  autotermination_minutes: 60
  is_single_node: true
  enable_local_disk_encryption: false
  num_workers: 1

Steps to reproduce the behavior

1. Create a cluster definition (see example above)
2. Run databricks bundle deploy ...
3. Wait for the cluster to be created and spun up
4. Run databricks bundle deploy ... again
5. Watch the cluster being restarted or check event log (Compute edited by...)

Expected Behavior

The expected behavior is, that in case of no changes to the cluster configuration, the cluster should be untouched and available all the time (no restarts).

Actual Behavior

Cluster is restarting on each deployment, regardless of changes or not.

OS and CLI version

OS: Windows 11
CLI version: v0.260.0

Is this a regression?

Never used the feature before.

Debug Logs

cluster_eventlog.json

[andrewnester]

Hi @brain246 ! This is indeed unexpected. There's likely a permanent drift in your configuration due to the way you have single cluster configuration setup (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster#is_single_node-1): you have the following settings missed spark_conf and custom_tags and num_workers should be 0 if that's what you're trying to achieve (https://stackoverflow.com/questions/67324097/databricks-job-api-create-job-with-single-node-cluster)

Can you try to change it to

my_cluster:
  data_security_mode: SINGLE_USER
  cluster_name: 'All Purpose for me'
  single_user_name: ${workspace.current_user.userName}
  kind: CLASSIC_PREVIEW
  runtime_engine: STANDARD
  spark_version: 16.2.x-scala2.12
  node_type_id: Standard_D4ads_v5
  driver_node_type_id: Standard_D4ads_v5
  autotermination_minutes: 60
  is_single_node: true
  enable_local_disk_encryption: false
  num_workers: 0
  spark_conf:
    "spark.databricks.cluster.profile": "singleNode"
    "spark.master": "local[*, 4]"
  custom_tags:
    "ResourceClass": "SingleNode"

Also note I changed DATA_SECURITY_MODE_DEDICATED to SINGLE_USER, DATA_SECURITY_MODE_DEDICATED is an alias to SINGLE_USER and might cause a permanent drift too.

Can you give it a try and see if it works?

[brain246]

@andrewnester

First of all: Thanks for the quick response!

And you are totally right; the posted cluster spec makes no sense (singleNode and num_workers=1 don't go together). I must have missed this detail after changing these settings back and forth many times.

I don't know which of your recommendations actually solved my issue, but according to your feedback i just tested the deployment (and re-deployment) successfully (no more restarts) with these settings:

my_cluster:
  cluster_name: "All Purpose for me"
  spark_version: 16.2.x-scala2.12
  spark_conf:
    "spark.databricks.cluster.profile": "singleNode"
    "spark.master": "local[*, 4]"
  node_type_id: Standard_D4ads_v5
  driver_node_type_id: Standard_D4ads_v5
  custom_tags:
    "ResourceClass": "SingleNode"
  autotermination_minutes: 60
  single_user_name: ${workspace.current_user.userName}
  data_security_mode: SINGLE_USER
  runtime_engine: STANDARD
  kind: CLASSIC_PREVIEW
  is_single_node: true
  enable_local_disk_encryption: false
  num_workers: 0

So, a typical layer-8-issue on my part 😆

Having said that, a warning when validating/deploying such a spec would go a long way.

[kyle-kent]

@andrewnester could you please elaborate on what and why "permanent drift" is? What configs might cause it?

Do you know if this is a DABs/terraform issue or a databricks cluster config issue?

[leekennett]

@andrewnester
This still occurs despite following the above steps.

A hidden configuration is getting added that cannot be specified -
"release_version": "17.3.8"

When I view the Event Log that's the only property that differs.

[andrewnester]

@leekennett please open a new issue for the release_version field issue you see. You can also try a new direct deployment mode which shouldn't have this problem https://docs.databricks.com/aws/en/dev-tools/bundles/direct

@kyle-kent this is a DABs/TF issue where we incorrectly detect a change in configuration when nothing actually changes in dabs configuration.