I have tried: az network bastion delete --resource-group RandomResources-rg --name MyBastionHost  
Still I have Azure resource which is Type Bastion  
If I run: az network bastion list --resource-group RandomResources-rg  
I get:  
[  
"id": "/subscriptions//resourceGroups/RandomResources-rg/providers/Microsoft.Network/bastionHosts/RandomResources-rg-vnet-bastion",  
"ipConfigurations": [  
{  
"etag": "W/\"4b45cd72-3488-408a-8ca7-27364108ef95\"",  
"id": "/subscriptions//resourceGroups/RandomResources-rg/providers/Microsoft.Network/bastionHosts/RandomResources-rg-vnet-bastion/bastionHostIpConfigurations/IpConf",  
"name": "IpConf",  
"privateIpAllocationMethod": "Dynamic",  
"provisioningState": "Succeeded",  
"publicIpAddress": {  
"id": "/subscriptions//resourceGroups/RandomResources-rg/providers/Microsoft.Network/publicIPAddresses/RandomResources-rg-vnet-ip",  
"resourceGroup": "RandomResources-rg"  
},  
"resourceGroup": "RandomResources-rg",  
"subnet": {  
"id": "/subscriptions//resourceGroups/RandomResources-rg/providers/Microsoft.Network/virtualNetworks/RandomResources-rg-vnet/subnets/AzureBastionSubnet",  
"resourceGroup": "RandomResources-rg"  
},  
"type": "Microsoft.Network/bastionHosts/bastionHostIpConfigurations"  
}  
],
So there are still 3 resources left:  
Bastion - RandomResources-rg-vnet-bastion  
VNET: RandomResources-rg-vnet  
Public IP address: RandomResources-rg-vnet-ip