Troubleshooting silent testing

The Microsoft eCDN Silent Testing framework allows running simulations on multiple devices easily to emulate and examine how a given network behaves under the load of a live video event. All findings are logically presented in Microsoft eCDN's analytics dashboards.

Potential problems:

• Unable to create a silent test
• Silent runners blocked from coming online by policy
• No runners (or peers) participate in a silent test
• Unable to validate the instantiation of a silent runner
• Lack of analytics
• Low peering efficiency
• Large discrepancy between concurrent viewers and assigned devices

Solution for inability to create a silent test

If missing the plus (+) button for creating a silent test, ensure your Microsoft Entra user has an appropriate role assigned which grants Silent Test Modify permissions. For more, see the applicable roles in the Manage access documentation.

Solution for policy blocking silent runners

Some Microsoft Edge policies are known to interfere with silent runners.

After changing a policy, restart the endpoint or sign out/in and rerun a silent test. Confirm the runner instantiates by checking for indications as described in the Solution for inability to validate the instantiation of a silent runner section.

BrowserSignin policy

On some managed Windows endpoints, the Microsoft Edge BrowserSignin policy can prevent the silent runner from properly instancing. This policy is stored in the registry at:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge\BrowserSignin

When the value is set to 2 it has been observed to block the ability to load a website that the silent runner relies on, causing the runner to never come online.

To check the policy value (run PowerShell as Administrator):

Get-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name BrowserSignin -ErrorAction SilentlyContinue

If the output shows BrowserSignin : 2, remediate as follows:

• Preferred (recommended for production): Update the policy centrally using Group Policy or Microsoft Intune to change the Edge "BrowserSignin" setting to an option that allows browser sign-in (for example, Not configured) and then force a policy sync on the endpoints. After policy changes, sign out/sign in or reboot the device.

• For temporary local testing (requires admin rights): set the registry value to 0 or 1 (or remove it) to validate whether the policy is the cause:

# Set BrowserSignin to 1 (creates or updates the value)
New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name BrowserSignin -PropertyType DWord -Value 1 -Force

# Or remove the policy value (if present)
Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name BrowserSignin -ErrorAction SilentlyContinue

HeadlessModeEnabled policy

On managed Windows endpoints, the Microsoft Edge HeadlessModeEnabled policy prevents the silent runner's browser instance from launching in a headless or background mode. When the browser can't start in the expected (headless) mode, the runner fails to come online.

This policy is stored in the registry at:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge\HeadlessModeEnabled

To check the policy value (run PowerShell as Administrator):

Get-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name HeadlessModeEnabled -ErrorAction SilentlyContinue

If the output shows the value that disables headless operation HeadlessModeEnabled : 0, remediate as follows:

• Preferred (recommended for production): Update the policy centrally using Group Policy or Microsoft Intune to change the Edge "HeadlessModeEnabled" setting to an option that allows headless operation (for example, Not configured or Enabled) and then force a policy sync on the endpoints. After policy changes, sign out/sign in or reboot the device.

• For temporary local testing (requires admin rights): set the registry value to 1 (or remove it) to validate whether the policy is the cause:

# Set HeadlessModeEnabled to 1 (creates or updates the value)
New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name HeadlessModeEnabled -PropertyType DWord -Value 1 -Force

# Or remove the policy value (if present)
Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Edge' -Name HeadlessModeEnabled -ErrorAction SilentlyContinue

Solution for lack of runner participation in silent tests

To ensure compliance, review the silent testing process overview. Continue with this troubleshooting guide for more.

Solution for inability to validate the instantiation of a silent runner

The silent runner is designed to be inconspicuous to the user so validating the instantiation of one isn't a straight forward endeavor. Here's what to look for when seeking to validate whether a silent runner is active or not.

• The Connected Clients count in your Silent Testing dashboard.
• The presence of log files as "$env:TEMP\p5_log_" + $TestID + ".txt" where $env:TEMP is a system-set path and the $TestID value is set in the script. For example, "C:\Users\MYUSERNAME\AppData\Local\Temp\p5_log_123.txt"
• The presence of each of these background processes.
  • powershell or pwsh - A hidden PowerShell instance that runs the silent runner script; the choice of which is typically controlled by your endpoint management system.
  • msedge or chrome - Hidden Chromium browser instance, which takes on the role of your simulated viewer.
  • cmd or powershell - A hidden watchdog process that ends the PowerShell and chromium processes after the time-out elapses.

Solution for lack of analytics

If missing analytics, it's presumed that there's a blockage between individual clients and Microsoft eCDN's backend. Begin troubleshooting by opening the Management Console then navigating to Advanced > Silent Tester. At the top-right of that page, you'll find a question mark (?) link that takes you to the helper page where additional information on steps 1 and 3 can be found. Alternatively, you can also refer to How To Perform Silent Testing.

1. Open a direct runner URL taking care to use your tenant ID instead of the TENANT_ID_HERE placeholder. Construct it from the template URL provided here or you can use the prebuilt URL found in your helper page. The direct runner page simulates a single viewer, with which we'll investigate the state of the required connections.

   https://st-sdk.ecdn.teams.microsoft.com/?customerId=TENANT_ID_HERE&adapterId=Direct
   

   

2. Reveal the browser's DevTools. On Microsoft Edge, right-click on the page, then select "Inspect."

   Important

   The DevTools must be opened before starting the silent test in the following step, or crucial session start-up information won't be captured for your analysis.

3. Back on the Silent Tester page, start a Scheduled (Silent) Test, leaving all settings on their defaults aside from Video Quality. Set video quality to a static value, such as 480p - 1Mbps to remove variability. Once started, a gray DIV bar appears on the direct runner page with the text "Ongoing Test" to confirm the stream has begun, as seen in this example figure.

   

   You can also confirm the session is ongoing by observing video chunks downloading in the DevTool's Network tab.

   

4. Inspect the DevTool's Network and Console tabs for errors. Pay particular attention to the beginning of the session.

   Note any errors in red, excluding the "font" errors, which are known and unimpactful. Any errors that may indicate something blocking Microsoft eCDN is a fair lead in the troubleshooting effort.

   For example, in the Console tab, you may witness a connection rejection error similar to the following figure.

   

   This may be indicating that you have domains listed in the Websites Allow List in your Third Party Platforms page, and that the Microsoft eCDN domain hasn't been added to the allowlist.

   

   Or alternatively, your IP may not be included in the End-user IP Allow List of your Security page.

   

   Note

   Neither filter requires entries for Microsoft eCDN to function. That is, leaving either of these filters unpopulated disables the filter.

   If you’re not getting HTTP video data chunks, it could mean that something is blocking you from connecting to our backend; be it a firewall, proxy, etc. If you can identify the URL or protocol that is being blocked, check with your networking and/or security team to see if connections of that sort are being allowed. Review the following documents to ensure proper exceptions have been created: Network Requirements, Cloud and Security doc.

Solution for low peering efficiency

Low peering efficiency typically occurs when you're conducting a silent test with too few clients (under 20). By increasing the number of clients, you're enlarging the peering groups thus increasing the peering efficiency. See low efficiency troubleshooting for a list of other potential causes and how to troubleshoot each.

Solution for a large discrepancy between concurrent viewers and assigned devices

This situation typically happens when:

• local or corporate security software is blocking Microsoft eCDN
• the silent test is being conducted during transition periods (employees leaving the office and shutting down their workstations)

Here's how to troubleshoot:

• Avoid starting the silent test in transition periods (where employees are leaving the office) since then that might skew the number of participants.

• Allow between 1 - 3 minutes for real-time analytics to reflect in the Microsoft eCDN Analytics dashboard.

• Ensure the runner URL isn't being blocked. To see more, refer to the steps in the Solution for lack of analytics section.