Use Azure managed identities with Azure Databricks

This page shows you how to set up and use Azure managed identities authentication to automate your Azure Databricks accounts and workspaces.

Azure automatically manages identities in Microsoft Entra ID for applications to authenticate with resources that support Microsoft Entra ID authentication, including Azure Databricks accounts and workspaces. This authentication method obtains Microsoft Entra ID tokens without requiring you to manage credentials.

This page walks you through creating a user-assigned managed identity and assigning it to your Azure Databricks account, workspace, and Azure virtual machine (Azure VM). You then install and configure the Databricks CLI on your Azure VM to use Azure managed identities authentication and run commands to automate your Azure Databricks account and workspace.

• For more information about managed identities, see What are managed identities for Azure resources?.
• For more information about Azure managed identities authentication for Azure Databricks, see Authenticate with Azure managed identities.

Requirements

• Azure RBAC permissions to create and assign managed identities.
• Account or workspace admin role to assign managed identities to Azure Databricks. See Assign account admin roles to a user and Assign the workspace admin role to a user.
• Virtual Machine Contributor and Managed Identity Operator roles to create an Azure VM and assign the managed identity to it.

Step 1: Create a user-assigned managed identity

Create a user-assigned managed identity for Azure resources. Azure supports both system-assigned and user-assigned managed identities. Databricks recommends using user-assigned managed identities for Azure managed identities authentication with Azure Databricks.

To create a user-assigned managed identity, follow the instructions in Manage user-assigned managed identities using the Azure portal.

After creating the managed identity, copy the Client ID value from the managed identity's overview page. You'll need this value in Steps 2, 3, and 7.

Step 2: Assign the managed identity to your account

Assign your managed identity to your Azure Databricks account. Databricks treats managed identities as service principals. If you don't need account-level access, skip to Step 3.

Follow the instructions in Add service principals to your account. Choose Microsoft Entra ID managed and paste the Client ID from Step 1 as the Microsoft Entra application ID.

Step 3: Assign the managed identity to your workspace

Assign the managed identity to your Azure Databricks workspace. Databricks treats managed identities as service principals. Follow the instructions in Assign a service principal to a workspace.

When adding the service principal:

• If your workspace is enabled for identity federation: Select the service principal you created in Step 2.
• If your workspace isn't enabled for identity federation: Use the Client ID from Step 1 as the ApplicationId.

Step 4: Get the Azure resource ID for your workspace

Get the resource ID that Azure assigns to your Azure Databricks workspace. You'll need this value in Step 7.

1. In your Azure Databricks workspace, click your username in the top bar and click Azure Portal.

2. On the side pane, in the Settings section, click Properties.

3. In the Essentials section, copy the Id value. It should look similar to the following:

   /subscriptions/<subscription-id>/resourceGroups/<resource-group-id>/providers/Microsoft.Databricks/workspaces/<workspace-id>
   

Step 5: Create and log in to an Azure VM

Azure VMs are one of the resource types that support managed identities. You'll use this VM to run the Databricks CLI with managed identities authentication.

To create and connect to an Ubuntu Server VM using SSH authentication, follow the instructions in Quickstart: Create a Linux virtual machine in the Azure portal.

When creating the VM:

• Use Ubuntu Server 22.04 LTS as the image.
• Select SSH public key as the authentication type.
• Note the location of your downloaded private key file (.pem) and the VM's public IP address, as you'll need them to connect to the VM.

Step 6: Assign the managed identity to the Azure VM

Associate your managed identity with your Azure VM so that Azure can use it for authentication. See Assign a user-assigned managed identity to an existing VM.

1. In the Azure portal, navigate to your Azure VM's settings page and click Identity in the Settings section.
2. On the User assigned tab, click + Add.
3. Select the managed identity you created in Step 1 and click Add.

Step 7: Configure authentication

Install and configure the Databricks CLI on your Azure VM to use Azure managed identities authentication.

Install the CLI

From your SSH session on the Azure VM, install the Databricks CLI:

sudo apt install unzip
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh

Verify the installation:

databricks -v

Add configuration profiles

Create or edit the .databrickscfg file in your home directory (~/.databrickscfg) with the following content. See Azure Databricks configuration profiles.

Replace the following values:

• <account-console-url> with your Azure Databricks account console URL.
• <account-id> with your Azure Databricks account ID. See Locate your account ID.
• <azure-managed-identity-application-id> with the Client ID value for your managed identity from Step 1.
• <workspace-url> with your per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
• <azure-workspace-resource-id> with the Azure resource ID from Step 4.
• Optionally, replace the suggested configuration profile names AZURE_MI_ACCOUNT and AZURE_MI_WORKSPACE with different names.

If you don't need account-level operations, omit the [AZURE_MI_ACCOUNT] section.

[AZURE_MI_ACCOUNT]
host            = <account-console-url>
account_id      = <account-id>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

[AZURE_MI_WORKSPACE]
host                        = <workspace-url>
azure_workspace_resource_id = <azure-workspace-resource-id>
azure_client_id             = <azure-managed-identity-application-id>
azure_use_msi               = true

Step 8: Test the configuration

Test the configuration by running Databricks CLI commands from your SSH session on the Azure VM.

To test account-level access (if you configured it in Step 7):

databricks account users list -p AZURE_MI_ACCOUNT

To test workspace-level access:

databricks users list -p AZURE_MI_WORKSPACE

If you renamed the configuration profiles in Step 7, replace AZURE_MI_ACCOUNT or AZURE_MI_WORKSPACE with your custom names.