bundle deploy fails with azure-devops-oidc auth - missing env var SYSTEM_TEAMFOUNDATIONCOLLECTIONURI

[ylashin]

Describe the issue

When using azure-devops-oidc authentication in Azure DevOps pipelines, CLI commands like databricks current-user me and databricks bundle validate work correctly. However, databricks bundle deploy fails during the Terraform apply phase. The error message is as follows:

Error: terraform apply: exit status 1

Error: cannot create job: failed during request visitor: azure-devops-oidc auth: not calling from Azure DevOps Pipeline: missing env var SYSTEM_TEAMFOUNDATIONCOLLECTIONURI. Config: host=https://adb-xxxx.azuredatabricks.net, client_id=<CLIENT_ID>. Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID

  with databricks_job.gnaf_dlt_orchestrator_job,
  on bundle.tf.json line 54, in resource.databricks_job.gnaf_dlt_orchestrator_job:
  54:       },

I was actually facing issue #4047 yesterday and once it has been fixed, the current issue started to appear.

Steps to reproduce the behavior

1. Configure a Databricks service principal with Azure DevOps OIDC federation policy
2. Set up an Azure DevOps pipeline like the following :

steps:
    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: |
          databricks bundle deploy -t dev
      displayName: 'Deploy Bundle'
      env:
        DATABRICKS_HOST: $(DATABRICKS_HOST)
        DATABRICKS_CLIENT_ID: $(DATABRICKS_CLIENT_ID)
        DATABRICKS_AUTH_TYPE: azure-devops-oidc
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)

3. Run the pipeline

Expected Behavior

databricks bundle deploy should deploy the bundle successfully.

Actual Behavior

Bundle deployment fails and the DevOps task fails with error ##[error]Bash exited with code '1'

OS and CLI version

• Databricks CLI v0.281.0
• Ubuntu (Azure DevOps hosted agent vmImage: ubuntu-latest)

Is this a regression?

• Unknown

Debug Logs

Error: terraform apply: exit status 1

Error: cannot create job: failed during request visitor: azure-devops-oidc auth: not calling from Azure DevOps Pipeline: missing env var SYSTEM_TEAMFOUNDATIONCOLLECTIONURI. Config: host=https://adb-xxxx.azuredatabricks.net, client_id=<CLIENT_ID>. Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID

  with databricks_job.gnaf_dlt_orchestrator_job,
  on bundle.tf.json line 54, in resource.databricks_job.gnaf_dlt_orchestrator_job:
  54:       },

[SonOfLope]

In the meantime, I'm sharing my workaround until this is resolved :

    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: |
          # Workaround: Databricks CLI doesn't pass env vars to Terraform subprocess
          # See: https://github.com/databricks/cli/issues/4047

          # Download terraform 1.5.5 (version used by Databricks CLI)
          curl -fsSL https://releases.hashicorp.com/terraform/1.5.5/terraform_1.5.5_linux_amd64.zip -o /tmp/tf.zip
          unzip -qo /tmp/tf.zip -d /tmp

          # Create wrapper that injects SYSTEM_* env vars
          mkdir -p /tmp/tfwrap
          {
            echo '#!/bin/bash'
            env | grep -E '^(SYSTEM_|DATABRICKS_)' | while read line; do
              echo "export ${line%%=*}=\"${line#*=}\""
            done
            echo 'exec /tmp/terraform "$@"'
          } > /tmp/tfwrap/terraform
          chmod +x /tmp/tfwrap/terraform

          export DATABRICKS_TF_EXEC_PATH="/tmp/tfwrap/terraform"
          databricks bundle deploy --target ${{ parameters.environment }} --auto-approve
        workingDirectory: $(Build.SourcesDirectory)/${{ parameters.bundlePath }}
      displayName: 'Deploy Bundle'
      env:
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)

There are a few system environment variables needed from https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#system-variables. Didn't have time to test it so I just grep all the ones starting with SYSTEM_. I'm sharing this because you will need to include more than just 'SYSTEM_TEAMFOUNDATIONCOLLECTIONURI'.

[andrewnester]

The fix will be released in the upcoming CLI release this week