Edit

Share via

Troubleshoot BMM issues using the az networkcloud baremetalmachine run-read-command

There might be situations where a user needs to investigate and resolve issues with an on-premises bare metal machine (BMM). Operator Nexus provides the az networkcloud baremetalmachine run-read-command so users can run a curated list of read only commands to get information from a BMM.

The command produces an output file containing the results of the run-read command execution. By default, the data is sent to the Cluster Manager storage account. There's also a preview method where users can configure the Cluster resource with a storage account and identity that has access to the storage account to receive the output.

Prerequisites

  1. Install the latest version of the appropriate CLI extensions
  2. Ensure that the target BMM must have its poweredState set to On and have its readyState set to True
  3. Get the Managed Resource group name (cluster_MRG) that you created for Cluster resource

Send command output to a user specified Storage Account

To configure the Storage Account and container to which command output is sent, see Azure Operator Nexus Cluster support for managed identities and user provided resources.

To access the output of a command, users need the appropriate access to the storage blob, including both having the necessary Azure role assignments and ensuring that any networking restrictions are properly configured.

For role assignments, a user must have the following role assignments on the blob container or its Storage Account:

  • A data access role, such as Storage Blob Data Reader or Storage Blob Data Contributor
  • The Azure Resource Manager Reader role, at a minimum

For information on assigning roles to storage accounts, see Assign an Azure role for access to blob data.

For networking restrictions, if the Storage Account allows public endpoint access via a firewall, the firewall must be configured with a networking rule to allow that user's IP address through. If it allows only private endpoint access, a user must be part of a network that has access to the private endpoint.

For information on allowing access through the storage account firewall using networking rules or private endpoints, see the respective documentation.

Verify access to the specified Storage Account

Before running commands, you might wish to verify you have access to the specified Storage Account:

  1. From the Azure portal, navigate to the Storage Account.
  2. In the Storage Account details, select Storage browser from the navigation menu on the left side.
  3. In the Storage browser details, select Blob containers.
  4. Find the container to which command output is to be sent and select it.
  5. If you encounter errors while accessing the Storage Account or container, the user you're using might need a role assignment for the Storage Account or container. Alternatively, the Storage Account’s firewall settings might need to be updated to include your IP address.

Execute a run-read command

The run-read command lets you run a command on the BMM that doesn't change anything. Some commands have more than one word, or need an argument to work. These commands are made like this to separate them from the ones that can change things. For example, run-read-command can use kubectl get but not kubectl apply. When you use these commands, you have to put all the words in the "command" field. For example, {command:'kubectl get',arguments:[nodes]} is right; {command:kubectl,arguments:[get,nodes]} is wrong.

Also note that some commands begin with nc-toolbox nc-toolbox-runread and must be entered as shown. nc-toolbox-runread is a special container image that includes more tools that aren't installed on the bare metal host, such as ipmitool and racadm.

Some of the run-read commands require specific arguments be supplied to enforce read-only capabilities of the commands. An example of run-read commands that require specific arguments is the allowed Mellanox command mstconfig, which requires the query argument be provided to enforce read-only.

Warning

Microsoft doesn't provide or support any Operator Nexus API calls that expect plaintext username and/or password to be supplied. Note any values sent are logged and are considered exposed secrets, which should be rotated and revoked. The Microsoft documented method for securely using secrets is to store them in an Azure Key Vault. If you have specific questions or concerns, submit a request via the Azure portal.

This list shows the commands you can use. Commands in *italics* can't have arguments; the rest can.

  • arp
  • brctl show
  • dmidecode
  • fdisk -l
  • host
  • hostname
  • ifconfig -a
  • ifconfig -s
  • ip address show
  • ip link show
  • ip maddress show
  • ip route show
  • journalctl
  • kubectl api-resources
  • kubectl api-versions
  • kubectl describe
  • kubectl get
  • kubectl logs
  • mount
  • ping
  • ss
  • tcpdump
  • traceroute
  • uname
  • ulimit -a
  • uptime
  • timedatectl status
  • hostnamectl status
  • nc-toolbox nc-toolbox-runread ipmitool channel authcap
  • nc-toolbox nc-toolbox-runread ipmitool channel info
  • nc-toolbox nc-toolbox-runread ipmitool chassis status
  • nc-toolbox nc-toolbox-runread ipmitool chassis power status
  • nc-toolbox nc-toolbox-runread ipmitool chassis restart cause
  • nc-toolbox nc-toolbox-runread ipmitool chassis poh
  • nc-toolbox nc-toolbox-runread ipmitool dcmi power get_limit
  • nc-toolbox nc-toolbox-runread ipmitool dcmi sensors
  • nc-toolbox nc-toolbox-runread ipmitool dcmi asset_tag
  • nc-toolbox nc-toolbox-runread ipmitool dcmi get_mc_id_string
  • nc-toolbox nc-toolbox-runread ipmitool dcmi thermalpolicy get
  • nc-toolbox nc-toolbox-runread ipmitool dcmi get_temp_reading
  • nc-toolbox nc-toolbox-runread ipmitool dcmi get_conf_param
  • nc-toolbox nc-toolbox-runread ipmitool delloem lcd info
  • nc-toolbox nc-toolbox-runread ipmitool delloem lcd status
  • nc-toolbox nc-toolbox-runread ipmitool delloem mac list
  • nc-toolbox nc-toolbox-runread ipmitool delloem mac get
  • nc-toolbox nc-toolbox-runread ipmitool delloem lan get
  • nc-toolbox nc-toolbox-runread ipmitool delloem powermonitor powerconsumption
  • nc-toolbox nc-toolbox-runread ipmitool delloem powermonitor powerconsumptionhistory
  • nc-toolbox nc-toolbox-runread ipmitool delloem powermonitor getpowerbudget
  • nc-toolbox nc-toolbox-runread ipmitool delloem vflash info card
  • nc-toolbox nc-toolbox-runread ipmitool echo
  • nc-toolbox nc-toolbox-runread ipmitool ekanalyzer print
  • nc-toolbox nc-toolbox-runread ipmitool ekanalyzer summary
  • nc-toolbox nc-toolbox-runread ipmitool fru print
  • nc-toolbox nc-toolbox-runread ipmitool fwum info
  • nc-toolbox nc-toolbox-runread ipmitool fwum status
  • nc-toolbox nc-toolbox-runread ipmitool fwum tracelog
  • nc-toolbox nc-toolbox-runread ipmitool gendev list
  • nc-toolbox nc-toolbox-runread ipmitool hpm rollbackstatus
  • nc-toolbox nc-toolbox-runread ipmitool hpm selftestresult
  • nc-toolbox nc-toolbox-runread ipmitool ime help
  • nc-toolbox nc-toolbox-runread ipmitool ime info
  • nc-toolbox nc-toolbox-runread ipmitool isol info
  • nc-toolbox nc-toolbox-runread ipmitool lan print
  • nc-toolbox nc-toolbox-runread ipmitool lan alert print
  • nc-toolbox nc-toolbox-runread ipmitool lan stats get
  • nc-toolbox nc-toolbox-runread ipmitool mc bootparam get
  • nc-toolbox nc-toolbox-runread ipmitool mc chassis poh
  • nc-toolbox nc-toolbox-runread ipmitool mc chassis policy list
  • nc-toolbox nc-toolbox-runread ipmitool mc chassis power status
  • nc-toolbox nc-toolbox-runread ipmitool mc chassis status
  • nc-toolbox nc-toolbox-runread ipmitool mc getenables
  • nc-toolbox nc-toolbox-runread ipmitool mc getsysinfo
  • nc-toolbox nc-toolbox-runread ipmitool mc guid
  • nc-toolbox nc-toolbox-runread ipmitool mc info
  • nc-toolbox nc-toolbox-runread ipmitool mc restart cause
  • nc-toolbox nc-toolbox-runread ipmitool mc watchdog get
  • nc-toolbox nc-toolbox-runread ipmitool bmc bootparam get
  • nc-toolbox nc-toolbox-runread ipmitool bmc chassis poh
  • nc-toolbox nc-toolbox-runread ipmitool bmc chassis policy list
  • nc-toolbox nc-toolbox-runread ipmitool bmc chassis power status
  • nc-toolbox nc-toolbox-runread ipmitool bmc chassis status
  • nc-toolbox nc-toolbox-runread ipmitool bmc getenables
  • nc-toolbox nc-toolbox-runread ipmitool bmc getsysinfo
  • nc-toolbox nc-toolbox-runread ipmitool bmc guid
  • nc-toolbox nc-toolbox-runread ipmitool bmc info
  • nc-toolbox nc-toolbox-runread ipmitool bmc restart cause
  • nc-toolbox nc-toolbox-runread ipmitool bmc watchdog get
  • nc-toolbox nc-toolbox-runread ipmitool nm alert get
  • nc-toolbox nc-toolbox-runread ipmitool nm capability
  • nc-toolbox nc-toolbox-runread ipmitool nm discover
  • nc-toolbox nc-toolbox-runread ipmitool nm policy get policy_id
  • nc-toolbox nc-toolbox-runread ipmitool nm policy limiting
  • nc-toolbox nc-toolbox-runread ipmitool nm statistics
  • nc-toolbox nc-toolbox-runread ipmitool nm suspend get
  • nc-toolbox nc-toolbox-runread ipmitool nm threshold get
  • nc-toolbox nc-toolbox-runread ipmitool pef
  • nc-toolbox nc-toolbox-runread ipmitool picmg addrinfo
  • nc-toolbox nc-toolbox-runread ipmitool picmg policy get
  • nc-toolbox nc-toolbox-runread ipmitool power status
  • nc-toolbox nc-toolbox-runread ipmitool sdr elist
  • nc-toolbox nc-toolbox-runread ipmitool sdr get
  • nc-toolbox nc-toolbox-runread ipmitool sdr info
  • nc-toolbox nc-toolbox-runread ipmitool sdr list
  • nc-toolbox nc-toolbox-runread ipmitool sdr type
  • nc-toolbox nc-toolbox-runread ipmitool sel elist
  • nc-toolbox nc-toolbox-runread ipmitool sel get
  • nc-toolbox nc-toolbox-runread ipmitool sel info
  • nc-toolbox nc-toolbox-runread ipmitool sel list
  • nc-toolbox nc-toolbox-runread ipmitool sel time get
  • nc-toolbox nc-toolbox-runread ipmitool sensor get
  • nc-toolbox nc-toolbox-runread ipmitool sensor list
  • nc-toolbox nc-toolbox-runread ipmitool session info
  • nc-toolbox nc-toolbox-runread ipmitool sol info
  • nc-toolbox nc-toolbox-runread ipmitool sol payload status
  • nc-toolbox nc-toolbox-runread ipmitool user list
  • nc-toolbox nc-toolbox-runread ipmitool user summary
  • nc-toolbox nc-toolbox-runread racadm arp
  • nc-toolbox nc-toolbox-runread racadm coredump
  • nc-toolbox nc-toolbox-runread racadm diagnostics
  • nc-toolbox nc-toolbox-runread racadm eventfilters get
  • nc-toolbox nc-toolbox-runread racadm fcstatistics
  • nc-toolbox nc-toolbox-runread racadm get
  • nc-toolbox nc-toolbox-runread racadm getconfig
  • nc-toolbox nc-toolbox-runread racadm gethostnetworkinterfaces
  • nc-toolbox nc-toolbox-runread racadm getled
  • nc-toolbox nc-toolbox-runread racadm getniccfg
  • nc-toolbox nc-toolbox-runread racadm getraclog
  • nc-toolbox nc-toolbox-runread racadm getractime
  • nc-toolbox nc-toolbox-runread racadm getsel
  • nc-toolbox nc-toolbox-runread racadm getsensorinfo
  • nc-toolbox nc-toolbox-runread racadm getssninfo
  • nc-toolbox nc-toolbox-runread racadm getsvctag
  • nc-toolbox nc-toolbox-runread racadm getsysinfo
  • nc-toolbox nc-toolbox-runread racadm gettracelog
  • nc-toolbox nc-toolbox-runread racadm getversion
  • nc-toolbox nc-toolbox-runread racadm hwinventory
  • nc-toolbox nc-toolbox-runread racadm ifconfig
  • nc-toolbox nc-toolbox-runread racadm inlettemphistory get
  • nc-toolbox nc-toolbox-runread racadm jobqueue view
  • nc-toolbox nc-toolbox-runread racadm lclog view
  • nc-toolbox nc-toolbox-runread racadm lclog viewconfigresult
  • nc-toolbox nc-toolbox-runread racadm license view
  • nc-toolbox nc-toolbox-runread racadm netstat
  • nc-toolbox nc-toolbox-runread racadm nicstatistics
  • nc-toolbox nc-toolbox-runread racadm ping
  • nc-toolbox nc-toolbox-runread racadm ping6
  • nc-toolbox nc-toolbox-runread racadm racdump
  • nc-toolbox nc-toolbox-runread racadm sslcertview
  • nc-toolbox nc-toolbox-runread racadm swinventory
  • nc-toolbox nc-toolbox-runread racadm systemconfig getbackupscheduler
  • nc-toolbox nc-toolbox-runread racadm systemperfstatistics (PeakReset argument NOT allowed)
  • nc-toolbox nc-toolbox-runread racadm techsupreport getupdatetime
  • nc-toolbox nc-toolbox-runread racadm traceroute
  • nc-toolbox nc-toolbox-runread racadm traceroute6
  • nc-toolbox nc-toolbox-runread racadm usercertview
  • nc-toolbox nc-toolbox-runread racadm vflashsd status
  • nc-toolbox nc-toolbox-runread racadm vflashpartition list
  • nc-toolbox nc-toolbox-runread racadm vflashpartition status -a
  • nc-toolbox nc-toolbox-runread mstregdump
  • nc-toolbox nc-toolbox-runread mstconfig (requires query arg)
  • nc-toolbox nc-toolbox-runread mstflint (requires query arg)
  • nc-toolbox nc-toolbox-runread mstlink (requires query arg)
  • nc-toolbox nc-toolbox-runread mstfwmanager (requires query arg)
  • nc-toolbox nc-toolbox-runread mlx_temp

The command syntax for a single command with no arguments is as follows, using hostname as an example:

az networkcloud baremetalmachine run-read-command --name "<bareMetalMachineName>"
    --limit-time-seconds "<timeout>" \
    --commands "[{command:hostname}]" \
    --resource-group "<cluster_MRG>" \
    --subscription "<subscription>"
  • --name is the name of the BMM resource on which to execute the command.
  • The --commands parameter always takes a list of commands, even if there's only one command.
  • Multiple commands can be provided in json format using Azure CLI Shorthand notation.
  • Any whitespace must be enclosed in single quotes.
  • Any arguments for each command must also be provided as a list, as shown in the following examples.
  • Not all commands can run on any BMM. For example, kubectl commands can only be run from a BMM with the control-plane role.
--commands "[{command:hostname},{command:'nc-toolbox nc-toolbox-runread racadm ifconfig'}]"
--commands "[{command:hostname},{command:'nc-toolbox nc-toolbox-runread racadm getsysinfo',arguments:[-c]}]"
--commands "[{command:ping,arguments:[198.51.102.1,-c,3]}]"

These commands can be long running so the recommendation is to set --limit-time-seconds to at least 600 seconds (10 minutes). Running multiple commands might take longer than 10 minutes.

This command runs synchronously. If you wish to skip waiting for the command to complete, specify the --no-wait --debug options. For more information, see how to track asynchronous operations.

When an optional argument --output-directory is provided, the output result is downloaded and extracted to the local directory, provided the user running the command has appropriate access to the Storage Account.

Warning

Using the --output-directory argument overwrites any files in the local directory that have the same name as the new files being created.

This example executes a 'kubectl get pods'

az networkcloud baremetalmachine run-read-command --name "<bareMetalMachineName>" \
   --limit-time-seconds 60 \
   --commands "[{command:'kubectl get',arguments:[pods,-n,nc-system]}]" \
   --resource-group "<cluster_MRG>" \
   --subscription "<subscription>"

This example executes the hostname command and a ping command

az networkcloud baremetalmachine run-read-command --name "<bareMetalMachineName>" \
    --limit-time-seconds 60 \
    --commands "[{command:hostname},{command:ping,arguments:[198.51.102.1,-c,3]}]" \
    --resource-group "<cluster_MRG>" \
    --subscription "<subscription>"

This example executes the racadm getsysinfo -c command

az networkcloud baremetalmachine run-read-command --name "<bareMetalMachineName>" \
    --limit-time-seconds 60 \
    --commands "[{command:'nc-toolbox nc-toolbox-runread racadm getsysinfo',arguments:[-c]}]" \
    --resource-group "<cluster_MRG>" \
    --subscription "<subscription>"

Check the command status

Sample output is shown. It prints the top 4,000 characters of the result to the screen for convenience and provides a short-lived link to the storage blob containing the command execution result.

  ====Action Command Output====
  + hostname
  rack1compute01
  + ping 198.51.102.1 -c 3
  PING 198.51.102.1 (198.51.102.1) 56(84) bytes of data.

  --- 198.51.102.1 ping statistics ---
  3 packets transmitted, 0 received, 100% packet loss, time 2049ms

  ================================
  Script execution result can be found in storage account:
  https://<storage_account_name>.blob.core.windows.net/bmm-run-command-output/a8e0a5fe-3279-46a8-b995-51f2f98a18dd-action-bmmrunreadcmd.tar.gz?se=2023-04-14T06%3A37%3A00Z&sig=XXX&sp=r&spr=https&sr=b&st=2023-04-14T02%3A37%3A00Z&sv=2019-12-12

How to view the full output of a command in the associated Storage Account

To access the output of a command, users need the appropriate access to the storage blob, including both having the necessary Azure role assignments and ensuring that any networking restrictions are properly configured.

For role assignments, a user must have the following role assignments on the blob container or its Storage Account:

  • A data access role, such as Storage Blob Data Reader or Storage Blob Data Contributor
  • The Azure Resource Manager Reader role, at a minimum

For information on assigning roles to storage accounts, see Assign an Azure role for access to blob data.

For networking restrictions, if the Storage Account allows public endpoint access via a firewall, the firewall must be configured with a networking rule to allow that user's IP address through. If it allows only private endpoint access, a user must be part of a network that has access to the private endpoint.

For information on allowing access through the storage account firewall using networking rules or private endpoints, see the respective documentation.

With the necessary permissions and access configured, you can then use the link or command from the output summary to download the zipped output file (tar.gz).

You can also download it via the Azure portal:

  1. From the Azure portal, navigate to the Storage Account.
  2. In the Storage account details, select Storage browser from the navigation menu on the left side.
  3. In the Storage browser details, select on Blob containers.
  4. Select the blob container.
  5. Select the output file from the command. The file name can be identified from the output summary. Additionally, the Last modified timestamp aligns with when the command was executed.
  6. You can manage & download the output file from the Overview pop-out.