Create and manage Azure HDInsight cluster with Entra ID authentication

This article provides comprehensive information on creating and managing Azure HDInsight clusters with Microsoft Entra ID authentication. Users can securely authenticate and manage access to HDInsight clusters, ensuring enterprise-grade security and centralized identity governance by using Entra ID.

With this capability, organizations can enforce role-based access, streamline user onboarding and offboarding, and enhance compliance by using existing Entra ID policies. It simplifies cluster security management while providing a seamless login experience for data engineers, analysts, and administrators.

Prerequisites

Before you begin, ensure the following requirements are met:

• Azure Subscription

  • An active Azure subscription with sufficient permissions to create HDInsight clusters.

• Microsoft Entra ID Tenant

  • Access to an Entra ID tenant linked to your Azure subscription.

  • Permissions to create and assign Entra ID groups and roles.

• Resource Group

  • A resource group in Azure where the HDInsight cluster can be deployed.

• HDInsight Cluster Requirements

  • HDInsight cluster type (for example, Hadoop, Spark, HBase, or Kafka) selected for deployment.

  • Correct region chosen that supports Entra ID integration.

Overview

Users have to perform the following for setting up Entra ID authentication when creating a HDInsight cluster:

• Select the desired authentication method: Entra ID

• Add one (or more) admin Entra ID user during cluster creation(adding atleast one admin is mandatory).

  

User Profiles in Ambari

Entra ID enabled users are assigned one of two profiles:

• Cluster Admin: Admin permission.

• Cluster User: View only permission.

Login Options

Users can log in via Multifactor Authentication (MFA) once they input their Entra ID.

Adding users with API

Admin can add multiple users at once via an API, ideal for managing large clusters.

This operation allows users to change the cluster gateway HTTP credentials.

		{ 
		"restAuthEntraUsers": [ 
			{ 
				"objectId": "0d7c4bd6-d042-45ec-9ae5-1ed7871c38e0", 
						"displayName": "Hemant Gupta", 
						"upn": "john@contoso.com" 
					} 
				] 
			} 

Response

HTTP 202 (Accepted) on successful completion of the operation.

Authentication Process

The authentication process varies based on the chosen method during cluster creation:

If Entra ID is selected:

• The cluster creator provides an ID for the default cluster administrator user in Ambari.

• The default admin can add Ambari users after cluster creation. Users may have either Cluster Administrator or Cluster User permissions, set via the Ambari UI or REST API. The cluster admin also has to add Object ID, and the display name and click on "Save".

  

  

• A multifactor authentication prompt appears when the user logs in with their Entra ID.

Basic Authentication

Users can use the legacy basic authentication way of authenticating users as well

If Basic Authentication is selected:

• User provides a User ID and password for the default admin user.

• New users can be created with various roles, similar to current functionality.

• Users are prompted to enter their User ID and password upon login.

Steps to add object ID in Ambari UI

1. Log in to the Ambari portal.

   

2. Navigate to "Manage Ambari" option.

   

3. Click on user tab to see all present users in Ambari UI.

   

4. Click on "Add User" tab to add more users in the cluster.

   

5. Input Object ID, display name, and select the user access (Cluster Administrator or Cluster User). Select "Save".