Azure Databricks PERMISSION_DENIED on ADLS container, seems to require account-level access

Regarding Databricks Runtime version, Serverless will use latest (managed by Databricks). When I ran using a non-serverless cluster, I used 16.4 LTS (includes Apache Spark 3.5.2, Scala 2.12).

Yes, when I assign Storage Blob Data Reader at the account level, I am able to successfully access and read the external file location from compute.
Here is the External Location config:

Thanks again for the detailed follow-up — this helps a lot. I see you’re using Databricks Runtime 16.4 LTS (non-serverless) and that assigning Storage Blob Data Reader at the account level does resolve the issue, which confirms the root cause is permission related.

Since your external location uses a managed identity via an access connector (as shown in your screenshot), and you’ve confirmed RBAC works at the account level, the likely culprit is ACLs on the specific ADLS Gen2 path — not the storage account or container-level RBAC. Even if the managed identity has Storage Blob Data Reader role, ADLS Gen2’s hierarchical namespace enforces POSIX-style ACLs that can override RBAC at the directory or file level.

Here’s what I’d suggest: Go into Azure Storage Explorer or use Azure CLI to check the ACLs on the exact folder or file path you’re trying to access — especially the parent directories leading up to it. The managed identity needs “r-x” (read + execute) on all parent folders and “r--” on the files themselves. If any folder in the path has restrictive ACLs (like no execute permission), you’ll get PERMISSION_DENIED even with correct RBAC.

You can verify this using: az storage fs access show --path <your-path> --file-system <container-name> --account-name <storage-account>

Also, since you’re using a managed identity via access connector, double-check that the connector is properly registered and assigned to the workspace — sometimes a misconfigured connector ID can silently fail even if RBAC looks right.

One quick question to help further:

• Are you able to list the contents of the external location using dbutils.fs.ls("abfss://...")? Or does the error occur only when reading actual files?

For reference, Microsoft’s guidance on ACLs in ADLS Gen2 is up to date here: https://free.blessedness.top/en-us/azure/storage/blobs/data-lake-storage-access-control

And the integration steps for Databricks access connectors with managed identities: https://free.blessedness.top/en-us/azure/databricks/data/data-sources/azure/adls-gen2/azure-datalake-gen2-sp-access

Thanks

Pratyush

Since I am able to access the folders/files once account-level permissions are added, it is not ACL related. I did not need to change any ACL to allow it to work. The real question is why the compute calls require escalated (account-level) permission when being called. If the container has sufficient permissions, then the account-level permissions should not matter.

However, in answer to your question, I get an error when using dbutils.fs.ls to list External Location contents (using any compute type) without the account-level role. Once the account-level role is added, it works as expected.

So, the question is why the exact same code attempting to read an external location pointing to a container ABFSS location requires account-level / escalated permissions. My understanding is that we should be able to grant access to the container level directly without requiring account-level permissions. For example, we currently use Azure Data Factory to read from the same container without issue - it only requires read access at the container level. Furthermore, Databricks can also browse the content in Unity Catalog Explorer, so it has permissions.

I suspect this may be a bug in how Azure Databricks is implementing the read from External Location. This is repeatable, so an engineer should be able to reproduce the exact same scenario.

Hi Scott Hietpas,

Thank you for your response.

Please allow us some time to reproduce the issue on our end, we’ll get back to you with an update

Thank you for your patience!

Kalyani

ACLs should not be required. Despite that, I did attempt to add ACLs, but this had no effect - as expected since they would be ignored based on already having course-grained RBAC access. See storage access control model: https://free.blessedness.top/en-us/azure/storage/blobs/data-lake-storage-access-control-model#:~:text=you%20cannot%20use%20an%20ACL%20to%20restrict%20access%20that%20has%20already%20been%20granted%20by%20a%20role%20assignment%20and%20its%20conditions.%20That%27s%20because%20the%20system%20evaluates%20Azure%20role%20assignments%20and%20conditions%20first%2C%20and%20if%20the%20assignment%20grants%20sufficient%20access%20permission%2C%20ACLs%20are%20ignored.

Do you re-start after your implementation?

Hi Scott Hietpas,

Thanks for your feedback.

For this kind of an issue, I will suggest you keep your current container‑scoped RBAC (Storage Blob Data Reader on the specific container) for data operations and then add a second, minimal account‑scope assignment of the same role but constrained with Azure ABAC conditions so it only authorizes requests for that one container (and optional subpath) this satisfies any account‑level checks compute performs while preserving least privilege. After that, restart the SQL warehouse/cluster or sign out/in to refresh tokens and allow up to 30 minutes for RBAC propagation, if your storage uses firewalls and you’re on Serverless SQL, ensure the serverless compute subnets are allowed. Validate the Unity Catalog objects by describing the external location and showing grants to confirm workspace and principal permissions. - https://free.blessedness.top/en-us/azure/role-based-access-control/troubleshooting.

You can also validate UC permissions (run in a Databricks SQL editor or notebook):

DESCRIBE EXTERNAL LOCATION my_ext_loc;
SHOW GRANTS ON EXTERNAL LOCATION my_ext_loc;

https://free.blessedness.top/en-us/azure/databricks/connect/unity-catalog/cloud-storage/external-locations

Also, for an example ABAC condition (attach to a Storage account–scope, Storage Blob Data

Reader assignment, limiting access to one container and prefix):

{
  "Version": "1.0",
  "Condition": {
    "StringEquals": {
      "storage:containerName": "my-container"
    },
    "StringLike": {
      "storage:blobPath": "restricted-prefix/*"
    }
  }
}

https://free.blessedness.top/en-us/azure/storage/blobs/storage-auth-abac

Then, re-test from compute:

# Python in a notebook (fresh cluster/warehouse session)
dbutils.fs.ls("abfss://******@myaccount.dfs.core.windows.net/restricted-prefix/")

If you’re using Serverless SQL with storage firewalls, also follow this link for more details: https://free.blessedness.top/en-us/azure/databricks/admin/sql/serverless.

Success