Computer-Use Agents

Agents that see a screen and click. Universal compatibility, unreliable execution, and a security model that demands isolation.

Most agents call APIs. Computer-use agents operate a graphical interface: take a screenshot, decide where to click, click, take another screenshot.

The appeal is universality. Any software with a screen becomes automatable, including legacy systems with no API and no prospect of one.

The cost is that everything is harder — perception, precision, and security all degrade compared with calling an API.

The loop

Structurally the same agent loop, with a screen as the environment:

1. Screenshot the current state
2. A vision model interprets it and decides an action
3. Execute — click at coordinates, type text, scroll, press a key
4. Screenshot again, observe the result
5. Repeat

Tools are primitives: click(x, y), type(text), scroll(direction), key(combo).

The hard part is step 2 producing correct coordinates. The model must locate a target in an image precisely enough to hit it, and precise spatial reasoning is a known weakness — patches do not individuate small UI elements well.

Why it is unreliable

Coordinate precision. Off by twenty pixels means clicking the wrong control. Small targets, dense toolbars, and adjacent buttons are where this fails.

Screenshot cost. Each one consumes substantial context — often more than several pages of text. A twenty-step task accumulates twenty screenshots, which is expensive and fills the window fast. Most systems keep only recent images and summarize older steps.

Timing. Pages load asynchronously. Acting on a stale screenshot is a constant hazard, and knowing when a UI has settled is genuinely difficult.

State ambiguity. Did the click register? Did a dialog appear behind the window? The agent sees only pixels, with no access to application state.

Compounding failure. Reliability compounds downward, and per-step reliability here is lower than for API calls. Long GUI tasks fail often.

Use an API when one exists

Worth stating plainly, because it is the most common mistake: if the software has an API, use it. It is faster, cheaper, more reliable, and testable.

Computer use is for when there is no alternative — legacy internal tools, desktop applications, workflows spanning several applications with no integration between them.

For web work specifically, driving the browser through its automation interface — reading the DOM, using selectors — is substantially more reliable than screenshots and clicks. Semantic structure beats pixels. Reach for vision only when the DOM is unusable, such as canvas-rendered interfaces.

The security model

This is the part that needs care, because the exposure is larger than it appears.

An agent with screen control has broad access. It can see anything displayed and act on anything clickable. Scoping it is not straightforward, unlike an API key with limited permissions.

Every screen is untrusted input. A web page, an email, or a document rendered on screen can contain text shaped like instructions. The agent reads pixels and cannot distinguish your instruction from text in the content it is viewing. This is prompt injection with a wide surface and no clean defense.

Consequences are frequently irreversible. Submitting a form, sending a message, confirming a payment. There is no git revert for a GUI action.

Which produces requirements rather than suggestions:

• Run in an isolated environment — a dedicated VM or container, not a machine with logged-in production access
• Scope credentials narrowly; assume anything reachable on that screen is reachable by the agent
• Require confirmation before consequential actions — submissions, payments, deletions, sends
• Log every screenshot and action for audit; this is your only record
• Treat any agent that browses the open web as compromised for planning purposes

Where it works today

Form filling across systems with no integration.

Data extraction from interfaces with no export function.

UI testing — navigate an application and verify what appears, where reversibility is inherent.

Legacy bridging — a narrow, well-specified path through an old internal tool.

The common thread is short, well-specified, reversible tasks. Open-ended computer use remains largely a demonstration.

What to remember

• Screenshot → decide → click → screenshot. Universal compatibility, low per-step reliability.
• Coordinate precision, screenshot context cost, and timing are the recurring failures.
• Use an API if one exists; for web work, DOM automation beats pixels.
• Every screen is untrusted input, and prompt injection has no clean defense here.
• Requirements: isolated environment, narrow credentials, confirmation before consequential actions, full logging.
• Works for short, specified, reversible tasks.

Next: Human-in-the-Loop Design