How to Build a Gemini Computer Use Agent in Python

Gemini can now look at a browser, click through it and type into forms, which is great until a hidden line on a web page tells it to do something else. By the end of this guide you will have a working Gemini computer use Python agent: the request loop, an action executor, screenshot feedback, safety checks and a sandbox to run it in.

Computer Use went GA in the Gemini API on October 30, according to the Gemini API changelog. You will use the Interactions API, the gemini-3.8-flash model and google-genai 2.7.0 or later, as in the Computer Use docs.

Step 1: Install the SDK and Playwright

pip install "google-genai>=2.7.0" playwright
playwright install chromium
export GEMINI_API_KEY=<your-api-key>

Playwright drives the browser. The model never touches it: it only returns actions, and your code performs them.

Step 2: Send the first Gemini Computer Use tool request

import base64, json, sys, time
from urllib.parse import urlparse

from google import genai
from playwright.sync_api import sync_playwright

MODEL = "gemini-3.8-flash"
WIDTH, HEIGHT = 1440, 900
ALLOWED_HOSTS = {"ai.google.dev"}  # add every host the task needs
SELECT_ALL = "Meta+A" if sys.platform == "darwin" else "Control+A"

client = genai.Client()  # reads GEMINI_API_KEY

TOOLS = [{
    "type": "computer_use",
    "environment": "browser",
    "enable_prompt_injection_detection": True,
}]

def b64(png: bytes) -> str:
    return base64.b64encode(png).decode("utf-8")

The tools entry is the whole integration. You never send a screen size, because the model replies in normalized coordinates. The first call (step 6) carries your goal plus a screenshot.

Step 3: Execute the returned actions and scale the coordinates

Each response has steps. The ones with type == "function_call" are actions, with a name and arguments. Coordinates arrive on a 1000-wide grid (0 to 999), so multiply by your real viewport.

def px(value: int, size: int) -> int:
    return int(value / 1000 * size)

def run_action(page, name: str, args: dict) -> None:
    if name in ("click", "double_click", "right_click", "middle_click", "move"):
        x, y = px(args["x"], WIDTH), px(args["y"], HEIGHT)
        if name == "click":
            page.mouse.click(x, y)
        elif name == "double_click":
            page.mouse.dblclick(x, y)
        elif name == "right_click":
            page.mouse.click(x, y, button="right")
        elif name == "middle_click":
            page.mouse.click(x, y, button="middle")
        else:
            page.mouse.move(x, y)
    elif name == "type":
        if "x" in args and "y" in args:
            page.mouse.click(px(args["x"], WIDTH), px(args["y"], HEIGHT))
        page.keyboard.press(SELECT_ALL)
        page.keyboard.press("Backspace")
        page.keyboard.type(args["text"])
        if args.get("press_enter"):
            page.keyboard.press("Enter")
    elif name == "navigate":
        page.goto(args["url"])
    elif name == "go_back":
        page.go_back()
    elif name == "go_forward":
        page.go_forward()
    elif name == "wait":
        time.sleep(args.get("seconds", 1))
    else:
        raise NotImplementedError(f"No handler for action: {name}")

This covers the common actions. The docs' action table also lists scroll, press_key, hotkey and drag_and_drop, so add handlers for those before you aim this at real sites. An unhandled action raises, and the error goes back to the model.

Step 4: Return screenshots with previous_interaction_id

After the actions run, answer every call with a function_result holding the current URL and a fresh screenshot.

def function_results(page, done):
    shot = b64(page.screenshot(type="png"))
    return [
        {
            "type": "function_result",
            "name": name,
            "call_id": call_id,
            "result": [
                {"type": "text", "text": json.dumps({"url": page.url, **result})},
                {"type": "image", "data": shot, "mime_type": "image/png"},
            ],
        }
        for name, call_id, result in done
    ]

Send one result for every call in the turn, passed as input with previous_interaction_id=interaction.id, so the server keeps the history and you don't resend it.

Step 5: Turn on prompt injection detection and handle safety_decision

The enable_prompt_injection_detection flag in step 2 is opt-in and defaults to false. When on, it scans the screenshot for hidden adversarial instructions such as "Ignore previous commands" and blocks execution if it finds them. The docs say it works with Gemini 3.5 Flash-Lite or later.

Separately, a function call can carry a safety_decision inside its arguments. When the decision is require_confirmation, ask a human.

class Halt(Exception):
    pass

def confirm(decision: dict) -> None:
    if decision.get("decision") != "require_confirmation":
        raise Halt(f"Safety system stopped the run: {decision}")
    answer = input(f"Confirm? {decision.get('explanation', '')} [y/N] ")
    if answer.strip().lower() != "y":
        raise Halt("User declined the action")

This fails closed: any decision other than require_confirmation ends the run. If the user agrees, the docs' sample sets safety_acknowledgement in the action result, which step 4 serializes into the text item.

Step 6: Assemble the loop with a navigation allowlist

def guard(route):
    request = route.request
    host = urlparse(request.url).hostname
    if request.is_navigation_request() and host not in ALLOWED_HOSTS:
        route.abort()
    else:
        route.continue_()

def run(goal: str, start_url: str, max_turns: int = 20) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(viewport={"width": WIDTH, "height": HEIGHT})
        context.route("**/*", guard)
        page = context.new_page()
        page.goto(start_url)

        interaction = client.interactions.create(
            model=MODEL,
            input=[
                {"type": "text", "text": goal},
                {"type": "image", "data": b64(page.screenshot(type="png")), "mime_type": "image/png"},
            ],
            tools=TOOLS,
        )

        for _ in range(max_turns):
            calls = [s for s in interaction.steps if s.type == "function_call"]
            if not calls:
                text = " ".join(
                    block.text
                    for s in interaction.steps if s.type == "model_output"
                    for block in s.content if block.type == "text"
                )
                return text or "No final answer; check the logs."

            done = []
            for call in calls:
                args = call.arguments
                print(f"-> {call.name}: {args.get('intent', '')}")
                result = {}
                if "safety_decision" in args:
                    confirm(args["safety_decision"])
                    result["safety_acknowledgement"] = True
                try:
                    run_action(page, call.name, args)
                except Exception as exc:
                    result["error"] = str(exc)
                page.wait_for_timeout(1500)
                done.append((call.name, call.id, result))

            interaction = client.interactions.create(
                model=MODEL,
                previous_interaction_id=interaction.id,
                input=function_results(page, done),
                tools=TOOLS,
            )
        raise Halt("Turn limit reached")

if __name__ == "__main__":
    print(run(
        "Go to ai.google.dev/gemini-api/docs and search for pricing.",
        "https://ai.google.dev/gemini-api/docs",
    ))

The guard function blocks page and iframe navigations to unlisted hosts while images and scripts still load. The max_turns cap stops a confused agent looping on your bill.

Step 7: Run the Gemini computer use agent in a sandbox

The docs say to run the agent in a sandboxed VM or container. Playwright's Docker image works as a base, but its Docker docs warn that it runs browsers as root by default, which disables the Chromium sandbox. They recommend a non-root user with a seccomp profile, and say the image is for testing and development, not untrusted websites. So treat the container as one layer and the allowlist as another.

FROM mcr.microsoft.com/playwright/python:v1.64.1-noble
ENV PIP_BREAK_SYSTEM_PACKAGES=1
RUN pip install --no-cache-dir "google-genai>=2.7.0" playwright==1.64.1
WORKDIR /app
COPY agent.py .
USER pwuser
CMD ["python", "agent.py"]

docker build -t gemini-cu-agent .
docker run -it --rm --init --ipc=host \
  --security-opt seccomp=seccomp_profile.json \
  -e GEMINI_API_KEY gemini-cu-agent

Pin the playwright package to the image tag's version, and copy seccomp_profile.json from the Playwright Docker docs. PIP_BREAK_SYSTEM_PACKAGES is there because Ubuntu 24.04's system Python refuses global pip installs.

Gotchas

• Old tutorials break. The Interactions API schema moved from outputs to steps, with the legacy schema removed on June 8. Code that reads outputs will not match.
• Select-all is platform specific. The docs' sample uses Meta+A, which is macOS. In a Linux container you need Control+A, as the code above picks.
• Overrides don't replace handling. disabled_safety_policies states a preference, and the model may still return require_confirmation. Keep confirm() either way.
• Cost grows per turn. Every turn sends a full screenshot, so check pricing and rate limits and keep max_turns low.
• Keep credentials out. Use a fresh browser context with no logged-in sessions.

A computer-use agent is a model with a mouse, so the allowlist, the confirmation gate and the container are what let you ship it to a customer.