Share via

Test an agent

Important

Some information in this article relates to a prereleased product which may be substantially modified before it's commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

This article guides you on how to test the agent you created or its tools in Security Copilot.

You must publish your agent to a user or a workspace scope in order to test your agent. The testing phase is an iterate and improve loop: configure the agent, invoke it with sample inputs, or use the autotune instruction optimization, observe its behavior or output, and refine the YAML manifest as needed based on the observed results.

Prerequisites

Plugin is a collection of related tools that users can enable in Security Copilot that extend what an agent can do by giving it access to resources outside of an agent and LLM.

  • Ensure that the Required plugins are enabled. You can enable them from the Test tab > Required plugins section.

  • Select Enable plugin if the plugin is not enabled. For example, for the Microsoft Threat Intelligence plugin, select Enable plugin for Microsoft Threat Intelligence.

    Image of the testing tab to enable the required plugins in Security Copilot

Steps to test an agent

The steps for testing the agent is as follows:

  1. Publish the agent to a user or workspace scope.

  2. In the top navigation pane, go to Test tab.

    Image of the testing tab to test agents in Security Copilot

  3. On the Test your agent's tools page:

    • Select the agent or the tool from the dropdown.

    • Provide inputs to the agent or tool if configured.

  4. On the Instruction optimization section:

    • If enabled (optional), autotune instruction optimization recommends optimized agent instructions based on the expected output. It automates the iterative process of updating, testing, and refining agent instructions through an evaluation loop. Autotune reviews the agent instructions and optimizations to boost the AI quality score, so that the agent performs as expected.

    • Depending on the selected tool type, you can provide prompt and Natural Language (NL) Expected output instructions for testing. Provide a detailed output of the agent as you would expect the agent to behave.

      Image of Autotune enabled for test optimization in Security Copilot

  5. Select Run test to generate test results.

    • Output of the agent is the result of the test.

    • If autotune instruction optimization is enabled, it provides scores and recommended instructions based on the expected output.

  6. After the Autotune instruction optimization is run, if enabled:

    • The Autotune instruction optimization analytics is provided that includes the quality scores.

    • The results include recommended instructions and score differentiation between current instructions and Autotune instruction optimization recommended instructions along with reasoning.

    • You can copy and paste the Autotune instruction optimization provided instructions in their agent.

    • Scroll down on the Test panel to see the Autotune optimized results. Paste this into the instructions in the tool configuration form by navigating to the Build tab.

      Image of the test results optimized using Autotune in Security Copilot

  7. Select Activity map if you would like to view the agentic run workflow. Select the specific node to view the details.

  8. Once you are satisfied with the response and functionality of the agent after testing, publish the agent to workspace scope. The agent is published to Active agents.

    Note

    Autotune is currently available in two regions - US and EU. It will be expanded to other regions soon.

Versioning agent

  1. If you publish the agent with the same name, it overrides the previous agent.

  2. If you publish the agent with a new name, it creates a new agent.

Delete agent

  1. You can navigate to Sources in the prompt bar > Manage sources in Security Copilot and delete the agent.

  2. Third-party partners: You must go to Security Store to delete.

Edit agent

You can edit your custom agent choosing either option:

  1. Navigate to menu Build.

  2. Under My agents, view the agents that you authored and published at user scope (Just you) or workspace scope.

  3. Toggle Visible for just me to distinguish between agents that you published in user scope from agents that you published to everyone in the workspace.

  4. Select or search for your agent to edit.

    Image of my agents view in Security Copilot

  5. The agent details are populated in the agent overview page.

  6. Select Publish after making the changes to your agent.

    Note

    You can also go to the top navigation bar and select the dropdown to edit the agent.

    Image of the path to edit agent in Security Copilot