bundle deploy command fails with azure-devops-oidc auth - missing env var SYSTEM_JOBID

[ylashin]

Describe the issue

When using azure-devops-oidc authentication in Azure DevOps pipelines, CLI commands like databricks current-user me and databricks bundle validate work correctly. However, databricks bundle deploy fails during the Terraform apply phase. The error message is as follows:

Error: cannot read job: failed during request visitor: azure-devops-oidc auth: not calling from Azure DevOps Pipeline: missing env var SYSTEM_JOBID. Config: host=https://adb-xxxx.azuredatabricks.net, client_id=xxxx Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID

  with databricks_job.abn_lookup_job,
  on bundle.tf.json line 46, in resource.databricks_job.abn_lookup_job:
  46:       }

This issue relates to a couple recent issues:

• bundle deploy fails with azure-devops-oidc auth - missing env var SYSTEM_TEAMFOUNDATIONCOLLECTIONURI #4161
• Bundle Deploy fails when using azure-devops-oidc #4226

The workaround proposed by @SoneOfLope worked pretty well which means there are still one (or more) environment variables to be relayed to Terraform and without them the deploy command fails. So hopefully we can get all of them done once and for all.

Steps to reproduce the behavior

1. Configure a Databricks service principal with Azure DevOps OIDC federation policy
2. Set up an Azure DevOps pipeline like the following :

steps:
    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: |
          databricks bundle deploy -t dev
      displayName: 'Deploy Bundle'
      env:
        DATABRICKS_HOST: $(DATABRICKS_HOST)
        DATABRICKS_CLIENT_ID: $(DATABRICKS_CLIENT_ID)
        DATABRICKS_AUTH_TYPE: azure-devops-oidc
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)

3. Run the pipeline

Expected Behavior

databricks bundle deploy should deploy the bundle successfully.

Actual Behavior

Bundle deployment fails and the DevOps task fails with error ##[error]Bash exited with code '1'

OS and CLI version

• Databricks CLI v0.283.0
• Ubuntu (Azure DevOps hosted agent vmImage: ubuntu-latest)

Is this a regression?

• Unknown

Debug Logs

Error: terraform apply: exit status 1

Error: cannot read job: failed during request visitor: azure-devops-oidc auth: not calling from Azure DevOps Pipeline: missing env var SYSTEM_JOBID. Config: host=https://adb-xxxx.azuredatabricks.net, client_id=xxxx Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID

  with databricks_job.abn_lookup_job,
  on bundle.tf.json line 46, in resource.databricks_job.abn_lookup_job:
  46:       }