Why can't nvidia-smi communicate with the driver on my Ubuntu 22.04 Azure VM, even after a clean install, disabling Secure Boot, and confirming the DKMS driver is built for the active kernel?

Question

Why can't nvidia-smi communicate with the driver on my Ubuntu 22.04 Azure VM, even after a clean install, disabling Secure Boot, and confirming the DKMS driver is built for the active kernel?

Deeksha Dixit 20

nvidia-smi Fails on Ubuntu 22.04 Azure VM Despite Correct Setup

Goal: Get the NVIDIA driver for an A10 GPU working on an Azure VM.

Problem: nvidia-smi consistently fails with couldn't communicate with the NVIDIA driver, even after extensive troubleshooting and multiple reboots.

System Details

OS: Ubuntu 22.04.5 LTS

GPU: NVIDIA A10

Kernel: 6.8.0-1036-azure

Driver: nvidia-driver-580

Troubleshooting Performed

The installation appears correct, but the driver fails to load at runtime. Key steps taken include:

Clean Installation: Overcame initial package conflicts by completely purging all nvidia-* packages and reinstalling the recommended driver.

Secure Boot Disabled: The VM was fully recreated from its disk with Standard Security to ensure Secure Boot was disabled.

✅ Verified: mokutil --sb-state confirms SecureBoot disabled.

Correct Kernel Module Built: The NVIDIA driver module has been successfully built for the active kernel via DKMS.

  ✅ __Verified:__ `dkms status` shows `nvidia/580.95.05, 6.8.0-1036-azure, x86_64: installed`.

Current Status

Despite all indicators showing a correct installation, the driver still fails to load. This suggests a persistent, unrecoverable issue with the OS state on the disk, making a fresh VM from a pre-built image the most logical next step.

Jilakara Hemalatha 3,115 Reputation points Microsoft External Staff Moderator

2025-10-21T06:08:11.1533333+00:00

Hi Deeksha Dixit,

Could you please confirm, if provided response was helpful to you. Please let me know if you have any queries.

Answer accepted by question author

0 additional answers

Your answer

Jilakara Hemalatha 3,115 Reputation points Microsoft External Staff Moderator

2025-10-21T06:08:11.1533333+00:00

Hi Deeksha Dixit,

Could you please confirm, if provided response was helpful to you. Please let me know if you have any queries.

Answer 1

Jilakara Hemalatha 3,115 Microsoft External Staff Moderator

Hi Deeksha Dixit,

Thank you for providing detailed information on your NVIDIA driver issue on the Azure Ubuntu 22.04 VM with the A10 GPU.

Could you please follow below steps:

If you have any existing drives uninstall all existing NVIDIA drivers to start from a clean state or alternatively, you can create a new VM to ensure a clean environment.
After creating the VM. Enable the NVIDIA GPU Driver Extension for Linux using this CLI command:

az vm extension set --publisher Microsoft.HpcCompute --name NvidiaGpuDriverLinux --vm-name <your-vm-name> --resource-group <your-rg>

This installs and configures the validated NVIDIA driver automatically, ensuring compatibility with the Azure kernel and persistence across reboots. User's image

Login to the VM and verify the drive details:

nvidia-smi

User's image

Reference: https://free.blessedness.top/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux

https://free.blessedness.top/en-us/azure/virtual-machines/linux/n-series-driver-setup#supported-distributions-and-drivers

Hope this helps! You should now be able to view the NVIDIA driver details successfully. Please let me know if you have any further questions or if the issue persists after these steps.

Jilakara Hemalatha 3,115 Reputation points Microsoft External Staff Moderator

2025-10-25T10:20:41.27+00:00
Hi Deeksha Dixit,

As I mentioned above, could you please create a new VM of the same series and perform the provided steps.

You can refer to the known issues documented here:

https://free.blessedness.top/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux#:~:text=execution%20output%20log.-,Known%20issues,-GRID%20driver%2016

NOTE: The NVIDIA GPU Driver Extension (NvidiaGpuDriverLinux) currently installs the latest GRID drivers (17.5), which has known issues with CUDA on A10 series GPUs. NVIDIA is working on resolving this issue. Meanwhile, you should use GRID driver 16.5 by passing the runtime setting to the extension:

az vm extension set --resource-group <rg-name> --vm-name <vm-name> --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute --settings "{'driverVersion':'535.161'}"

Azure’s default GRID drivers were originally built for vGPU-enabled VMs, mainly the NV-series (NC/ND) and T4-series (NCasT4v3).

A10 VMs on Azure are part of the latest GPU series designed for AI/ML workloads, not traditional vGPU sharing. The default GRID driver (e.g., 17.x) often fails to initialize the A10 hardware, causing errors.

3.Even if DKMS kernel modules appear installed on Ubuntu 22.04, they are not fully compatible with Azure’s custom kernel and A10 firmware.

4.Using the Azure GPU Driver Extension with a validated driver version (e.g., 535.161) ensures:

Correct kernel module for the Azure 6.8.x-azure kernel

Proper NVML initialization

CUDA runtime compatibility

Persistence across reboots

Please refer to the wiki for the more details. Please let me know the status on this.

Share via

Why can't nvidia-smi communicate with the driver on my Ubuntu 22.04 Azure VM, even after a clean install, disabling Secure Boot, and confirming the DKMS driver is built for the active kernel?

0 additional answers

Your answer