Claude's Computer Use: A Field Guide for Forward Deployed Engineers

Most large organizations rely on at least one system that has no API, only an old web interface and a long manual workflow. Computer use lets Claude operate that kind of system the way a person would, by looking at the screen and using the mouse and keyboard.

How it works

Computer use is a tool. You run a desktop environment, usually in a virtual machine or container. Claude receives a screenshot and replies with an action, such as moving the mouse, clicking, typing or pressing a key. Your code performs the action, takes a new screenshot and sends it back. This repeats until the task is finished.

Claude never controls the machine directly. Your code does, which is where you put your safety controls.

Step 1: Check whether you need it

Working through a screen is slower and more expensive per task than calling an API, and it breaks when the interface changes. Look first for an API, a database export or an existing integration. Use computer use only when there's no other way in.

Step 2: Build the sandbox first

• Run the desktop in an isolated VM or container, never on someone's own computer.
• Create a dedicated account with minimal permissions in the target system.
• Limit network access to the sites the task requires.
• Keep real credentials out of the environment where possible, and never put secrets in prompts.

Anthropic publishes a reference implementation with a containerized desktop that you can start from.

Step 3: Write specific instructions

Describe the goal, the exact application, what a finished task looks like and when to stop. For example: "Update the delivery date on the order screen. If the order is locked or a dialog mentions billing, stop and report back."

Step 4: Add approval points

Require a person to approve irreversible actions, such as submitting forms, sending messages or deleting records. Save every screenshot and action so you can review what happened.

Step 5: Guard against prompt injection

Everything on the screen is input to the model. A web page or document can contain text written to redirect the agent. Keep it away from untrusted websites, and treat instructions that appear inside the content as suspicious.

Common problems

• Screen resolution affects both accuracy and cost. Use a moderate resolution and keep it consistent.
• Pop-ups, cookie banners and session timeouts cause many failures. Cover them in the instructions.
• Early success rates can be misleading. Measure across at least 20 real runs before committing to a timeline.

Use computer use for the systems with no other option, and give it the same controls you'd give any automation with access to production data.