Automate web and desktop apps with computer use (preview)

[This article is prerelease documentation and is subject to change.]

Computer use is a tool in Copilot Studio that lets your agent interact with and automate tasks on a Windows computer. It works with websites and desktop apps by selecting buttons, choosing menus, and entering text into fields on the screen. Describe in natural language what you want computer use to do, and it performs the task on a computer you set up by using a virtual mouse and keyboard. With computer use, agents can complete tasks even when there's no API to connect directly to the system. If a person can use an app or website, computer use can too. You can use computer use for tasks like automated data entry, invoice processing, and data extraction.

Computer use is powered by Computer-Using Agents (CUA), an AI model that combines vision capabilities with advanced reasoning to interact with graphical user interfaces (GUIs). Because it's AI-powered, it adapts to interface changes. For example, when buttons or screens change, the tool continues working without breaking your flow. It's simple to use. Just describe what you want in natural language. You don't need to write code.

Watch this video to learn how the computer use tool lets an agent interact with a web application.

Requirements

• The feature is available for environments where the region is set to United States.

• Make sure your agent has the generative orchestrator enabled to use computer use.

Add computer use to your agent

On a new agent or an existing agent in Copilot Studio, add computer use as a tool by following these steps:

1. Go to the Tools section in your agent and select Add tool.

2. In the Add tool dialog, select New tool.

3. Select Computer use.

4. Provide the instructions that describe the task the tool should perform on the computer. You see some instruction templates to get started with. To learn more about how to best write instructions for computer use, check best practices for instructions for computer use.

5. Choose the machine where computer use runs. You can either use the hosted browser, a ready-to-use machine for automating tasks on public-facing websites, or select a Windows machine that you can configure to use for computer use. Learn more in Configure where computer use runs.

6. Select Add and configure.

7. On the configuration page, configure these three fields:

   • Name: Enter the display name for the computer use tool. This name helps you differentiate it from other tools you add to your agent.

   • Description: Enter a short description of what this tool does and when to use it. This text lets your agent know when to use this tool.

   • Instructions: List the steps the tool should perform, including URLs and application names. For tips, see Best practices for instructions for computer use.

8. Review these other fields and settings on the configuration page that might be relevant:

   • Inputs: Use Inputs to define dynamic values that change each time computer use runs. For example, if you want to fill out a form with a different value on every run, create an input for that field. At execution time, computer use combines your instructions with the input values to complete the task.

   • Stored credentials: Define the credentials that computer use uses to sign in to websites and applications. During execution, if a sign-in prompt appears, computer use securely uses any credentials you defined in this section for that site or application. Password values of these credentials are stored in an Azure Key Vault that you provide. Learn how to create an Azure Key Vault in Create a key vault using the Azure portal.

     To configure credentials in the tool, first enter the subscription ID, resource group name, and Key Vault name. All this information is available from the Overview page of your Key Vault.

     To use Azure Key Vault secrets with Power Platform:

     • The Azure subscription that has the vault must have the "PowerPlatform resource provider" registered.
     • The user who creates the environment variable must have appropriate permissions to the Azure Key Vault resource.

     If you haven't already, follow the steps in Configure Azure Key Vault. Then provide the following details:

     • Name: the display name for the credential.

     • Username: the username used to sign in to the target website or application.

     • Login domain: the domain where the credentials should be entered (for example, login.microsoft.com). Make sure to verify this domain, as it might differ from the main site URL.

     • Azure secret name: the name of the secret in the Key Vault that stores the password for the website or application.

       Note

       Password fields are supported on all websites and most Windows applications (WinForms, WPF, UWP, WinUI, Win32), which covers most customer scenarios. Some app types, such as Electron, Java, Unity, games, command-line interfaces, Citrix, or other virtualized environments, might not be supported.

   • Access control: By default, computer use can operate on any website or application. If you want to restrict this access, enable access control to define the specific URLs and desktop applications that computer use should be limited to. You can configure both websites and applications:

     • Websites: Enter the main website address (for example, example.com). All pages on that website are included automatically. You can also use wildcards (*) for subdomains.
       • Examples: www.contoso.com, *.contoso.com, contoso.com
     • Desktop applications: Enter the application product name or process name. To find it, press Ctrl + Shift + Esc to open Task Manager. Then check the Processes tab.
       • Examples: Microsoft Edge, msedge, Notepad.

     Note

     Access control only prevents the model from taking actions on websites or applications that aren't in the allowlist. It doesn't stop the model from opening them. For example, if only microsoft.com and Microsoft Edge are in the allowlist, the model can still use the Edge search bar to open Bing. However, once Bing is opened, any attempt to interact with it fails because it isn’t in the allowlist.

   • Machine: This value is the name of the machine where computer use runs. Select Manage machine to view more machine settings.

   • Connection: This value is the connection for this tool. Update the connection to change the credentials and machine for computer use.

   • Authentication: Specify how computer use authenticates during execution:

     • Copilot author authentication (default): This option uses the maker’s credentials and is suitable for autonomous agents.

       Warning

       If you share an agent with this setting, anyone using it can act with the original author’s access on the configured machine.

     • User authentication: This option uses the credentials of the person interacting with the agent. Each user must have access credentials to the machine.

9. Select Save.

Configure where computer use runs

To interact with websites and desktop apps for task automation, computer use runs on a Windows machine. When you add computer use to an agent, you have the following options:

Hosted browser (powered by Windows 365)

The hosted browser lets you get started quickly without any machine setup. It provides both web automation using Microsoft Edge and access to built-in Windows applications.

The hosted browser runs in a Microsoft-managed environment. It isn't Microsoft Entra joined to your tenant or managed by your Intune policies. It's designed for quick web automation and early experimentation, but it doesn't support enterprise resource access, custom desktop apps, or organization-specific device management.

Machine

This option lets you use Windows machines that you own and manage. You register and manage machines in Power Automate.

Here are the key requirements to use a machine for computer use:

• Install Power Automate for desktop version 2.59.169.25241 or later. Include the Power Automate web extension installation for web browser interactions.

• After registering the machine, turn on computer use in the machine settings.

Learn more in prerequisites and limitations.

Register a new machine

Because computer use runs on the machine, dedicate a machine to computer use to avoid interruptions.

To register a new machine:

1. Install the latest version of Power Automate on your machine. During installation, make sure you check the Install the machine-runtime app to connect to the Power Automate cloud portal option.

2. Open the Power Automate machine runtime app and sign in.

3. Register the machine to the environment you want to use with computer use.

Learn more about registering machines in register a new machine.

Enable computer use

After you register your machine, enable it for computer use.

1. Sign in to Power Automate.
2. Go to Machines.
3. Select the machine you registered.
4. On the machine detail page, select Settings.
5. Turn on Enable for computer use.
6. Select Save.

Run scheduling and queuing

When a Machine is in use, run requests targeting the same machine are queued and executed sequentially:

1. The first run executes on the target machine.
2. Subsequent runs are added to the queue and marked as Queued.
3. When a run completes, the next run starts and is marked as Next to run.

To monitor the run queue:

1. Sign in to Power Automate.
2. Go to Machines.
3. Select the machine.
4. On the machine detail page, select Run queue.

Test computer use

Testing computer use is a key step in the authoring journey. After you enter a name, description, and instructions, and save the tool, select Test to start the test experience.

After a short loading period, the test experience appears:

• The left panel shows your instructions and a step-by-step log of the tool’s reasoning and actions.
• The right panel shows a preview of the actions on the machine you set up for computer use.

When the task finishes, you see a Test completed message. While the test is in progress, you can select Stop testing to immediately stop all actions on the machine.

If the result isn't what you expect, go to the configuration page and refine your instructions. Add more detail to improve accuracy. For guidance, see best practices for writing effective instructions.

Publish an agent with computer use

Set up computer use, then publish your agent. How your agent runs depends on the scenario. It can be autonomous or conversational:

• Autonomous agents run automatically and perform tasks in the background.

• Conversational agents let users interact through channels like Microsoft Teams.

Computer use works best for autonomous agents, performing tasks in the background without user interaction.

You can also apply computer use in conversational experiences, but keep these considerations in mind:

1. If you select User authentication as the authentication setting, each user interacting with the agent in a conversation needs valid credentials for the machine used by computer use.
2. When the tool runs, it shares reasoning messages and screenshots of the machine's activity in the chat.

Monitor computer use runs

To check what computer use did during a run:

1. Go to the Activity section of your agent and select a run.
2. On the run details page, switch between the Activity map and Transcript views.

The Transcript view shows how computer use follows your instructions, providing a step-by-step log with reasoning messages and screenshots for each action.

Advanced computer use activity

Storing computer use logs in Dataverse provides enhanced capabilities for reporting, auditing, and troubleshooting. When these logs are stored in Dataverse (default setting), you can view detailed activity information in the side panel by selecting a computer use action from the activity map.

Dataverse storage

When you turn on the Computer Use feature in the Power Platform admin center, the Store logs in Dataverse option is turned on by default. If you turn off this setting, only default activity logs appear. When enabled, logs consume a combination of Dataverse capacity:

• Database capacity (flowsession table and flowsessionbinary metadata)
• Log capacity (flowlog table)
• File storage capacity (flowsessionbinary file attachments)

Ensure you have sufficient capacity and that your retention period and verbosity settings align with your requirements.

Configure advanced computer use logging

Follow these steps to configure or verify advanced computer use logging:

1. Go to Power Platform admin center, select your environment, and open the Features area.

2. Scroll down to the Computer Use settings and ensure Store logs in Dataverse is turned on (default setting).

   Note

   The following advanced computer use log settings don't affect how Microsoft Copilot agent transcripts (including any computer use logs) are stored. Transcripts continue to include basic computer use logs, screenshots, and all other agent and tool logs, regardless of the configuration here.

3. Review or change the Computer use logs verbosity level:

   • All data (default) - Stores the complete set of advanced computer use logs.
   • Data without screenshots - Stores the same content as with All data, except the screenshots.
   • Minimal - Currently behaves the same as Data without screenshots, but this behavior might change in a future release.

4. Review or change the Log retention time to match your data retention timeframes:

   • Default: 7 days (10,080 minutes)
   • Maximum: 33,554,432 minutes
   • To keep logs forever, enter 0 or -1 as a custom value and ignore the unit dropdown.

5. (Optional) Turn on Send audit logs to Microsoft Purview if you want computer use run logs to be available in Microsoft Purview:

   • Once enabled, logs appear under the activity term CUAOperation.
   • This setting is independent of the previous two settings.

View computer use activity details

When you select a computer use action in the activity map, a side panel opens with comprehensive session information:

Best practices

To stay productive and safe in today's digital environments, follow best practices for computer use, especially when writing clear instructions and securing machines.

Best practices for securing machines

When setting up machines for computer use that lets AI perform tasks by using natural language, consider these security recommendations:

Best practices for instructions for computer use

The instructions you write determine how well computer use works. Specific, detailed instructions help computer use complete tasks accurately. Think of it as explaining a task to a colleague. Clear, step-by-step guidance helps ensure success.

Tips for writing effective instructions:

• Be specific about websites and applications. Always include the full URL of any website and the exact name of any application the tool should use. Example: "Open https://www.microsoft.com and go to 'Company news'."

• Clearly state relevant actions. If you want something done, say it explicitly - especially actions like submitting a form or sending an email. Example: "Once you fill in the form, select Submit. No need to ask for permission."

• Break down complex interactions. For areas where the UI might be more complex to navigate, explain each step in detail. Example: "Select the More icon in the top right corner. A dropdown appears. Once it opens, select the last item in the list."

• Use step-by-step formatting for longer tasks. Long instructions are easier to follow when you format them as a list.

Sample instructions

Explore these sample instructions to try out computer use or use them as a reference for writing your own.

Administrative controls

Administrators can control the availability and usage of computer use features. These controls help your organization manage how computer use is deployed and ensure it aligns with your security and compliance requirements.

Disable computer use

You can disable computer use in the admin center or using the Power Platform CLI.

Use the admin center

To disable computer use in an environment using the admin center:

1. Go to the Power Platform admin center.
2. Select Manage on the navigation pane, then select Environments. A list of environments appears.
3. Select the environment to update. The environment's details page opens.
4. On the top menu bar, select Settings.
5. Expand Product, and then select Features.
6. Scroll to the Computer use and turn off the toggle.

Use the Power Platform CLI

To disable computer use in an environment using the Power Platform CLI:

1. If you haven't already, install the Power Platform CLI.

2. Open a command line and run the following command:

     pac auth create
   

   Follow the prompts to connect to your tenant.

3. Run the following command to disable the setting:

   pac env update-settings --environment <envid> --name iscomputeruseinmcsenabled --value false
   

   Replace <envid> with the environment ID, which you can find in the URL of the environment settings page.

Disable hosted browser

To disable the hosted browser in a tenant:

1. Go to the Power Platform admin center.
2. Select Manage on the navigation pane, then select Tenant settings. A list of settings appears.
3. Select Hosted browser in computer use. A pane opens.
4. Turn off the toggle.
5. Select Save.

Licensing

While computer use is in preview, it's billed using the Agent action feature with a billing rate of five Copilot Credits. For more information, see Microsoft Copilot Studio billing rate and management.

Each computer use run can perform several actions, and each action costs five Copilot Credits.

For example, if you set up computer use to fill out a web-based timesheet form, it performs these actions each time it's triggered:

1. Launch browser.

2. Navigate to the timesheet web portal.

3. Select Create new timesheet.

4. Fill the Start Time form field.

5. Fill the End Time form field.

6. Fill the Project Code form field.

7. Select the Submit button.

In this example, computer use executes seven actions, consuming a total of 35 messages.

Limitations

The following limitations affect computer use:

• Hosted machine groups aren't supported.
• Multi-screen desktops aren't supported.
• In the agent's Activity tab, the agent session status might show as "In progress", although the session might be completed. Select a specific agent session and switch to transcript mode to validate that the computer use session is completed.

Hosted browser limitations

The service might throttle hosted browser usage based on demand. Therefore, it's not recommended for production use.

The following scenarios might cause throttling:

Troubleshooting

If you encounter issues while using computer use, review these common problems and their solutions:

Share your feedback

Do you have feedback about computer use? Let us know at computeruse-feedback@microsoft.com.

Related information

FAQ for the computer use tool