Share via

Azure blob NFS protocol kubelet Deadline exceeded error

BatmanNinja 0 Reputation points
2025-10-04T05:55:31.02+00:00

We have deployed azure blob with Nfs protocol . Below is storageClass code. It was working fine when we had public with limited access. Not when we had private endpoint few pods are coming up. But we are having error like #### kubelet RPC deadline exceeded aks disk contention monitorm . Operation for volume failed. No retried permitted. Error unmount volume. Tear down failed for volume data. Kubernetes.io/csi: unmounter. Tear down at failed.rpc error. Memory cgrou0 out of memory ##### Is there any change I need to make to work with private endpoint? Is there any way to tackle these errors. As we have around 40 pods using the same container with nfs protocol. Does blob storage with NFS work in private endpoint ? I guess PVC is unable to unmount kubelet. Any way to tackle these issues.

apiVersion: storage.k8s.io/v1apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

name: azurefile-nfs

parameters:

skuName: Premium_LRS

protocol: nfs. provisioner: blob.csi.azure.com

server: blobendpoint resourceGroup: <resource-group-name>

storageAccount: <storage-account-name>

reclaimPolicy: Delete

volumeBindingMode: Immediate

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Harish Peddapally 1,405 Reputation points Microsoft External Staff Moderator
    2025-10-07T05:20:34.5166667+00:00

    Hi BatmanNinja,

    Welcome to Microsoft Q&A, i hope you are doing well.

    Thank you for sharing the details and error messages about mounting Azure Blob storage with NFS protocol on AKS after switching to private endpoints. Based on the scenario and your symptoms, here are the targeted steps to resolve this:

    1. Confirm Private Endpoint and DNS Configuration:

    Azure Blob NFS works with private endpoints, but AKS nodes must resolve the storage account’s private DNS name correctly. Ensure you have configured a custom DNS zone or Azure Private DNS Zone for privatelink.blob.core.windows.net, and AKS nodes can resolve <storage-account>.privatelink.blob.core.windows.net to the private endpoint IP.

    See the official documentation for details: https://free.blessedness.top/en-us/azure/storage/common/storage-private-endpointslearn.microsoft

    • On an AKS node or test pod, run: nslookup <storage-account>.privatelink.blob.core.windows.net Confirm this resolves to the private endpoint IP.

    2. Network Security Group and Firewall Rules:

    Verify your Network Security Groups (NSGs) and firewall rules allow AKS nodes to communicate with the storage account’s private endpoint on port 2049 (NFS). Blocked traffic will trigger mount/unmount errors and RPC deadline exceeded messages.

    Refer to: https://free.blessedness.top/en-us/azure/storage/blobs/network-file-system-protocol-support-how-tolearn.microsoft

    3. AKS Pod and Node Resources:

    "Memory cgroup out of memory" and mount/unmount errors suggest either resource contention or excessive parallel mounts. Test by scaling down the number of pods mounting the NFS volume and monitoring node resource consumption:

    Restart affected AKS nodes to clear any stale mounts.

    Ensure node memory is adequate for your pod workload. Troubleshooting guide: https://free.blessedness.top/en-us/troubleshoot/azure/azure-kubernetes/storage/mounting-azure-blob-storage-container-faillearn.microsoft

    4. Known Limitations and Additional Guidance:

    NFS protocol on Blob storage does not support all legacy NFS features; check for known limitations here: https://free.blessedness.top/en-us/azure/storage/blobs/network-file-system-protocol-supportlearn.microsoft

    For persistent issues, verify you are using the latest CSI driver and StorageClass YAML per Microsoft examples: https://free.blessedness.top/en-us/azure/storage/blobs/network-file-system-protocol-support-how-tolearn.microsoft

    If these steps are implemented and verified, AKS pods should mount and unmount Azure Blob NFS volumes via private endpoints reliably. For full testing and setup validation, please follow all steps in Microsoft's official guides linked above.

    Let me know if you need further troubleshooting or clarification.

    If the provided information answers your query, do click "Accept Answer", it will help others who might be facing similar challenges.

    Thanks,

    Harish.

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.