A GUI agent is an LLM that operates a computer the way you do: it looks at the screen, moves a cursor, clicks, types, looks again. No API, no integration, no per-app connector. Point it at anything with a screen and it should work.
That is the pitch, and the chart above is the reality. Every bar is a published number from the vendor’s own announcement, which means every bar is the friendliest number available. The best of them, on the benchmark that most resembles a real desktop, is roughly half of what a person scores.
I spent a while reading everything that shipped and then building against it, and the thing I came away with is not “these need to be smarter.” It is that they are missing a set of habits that human operators have and nobody has bothered to implement.
What actually ships
OpenAI’s Computer-Using Agent
CUA is the model behind Operator. It works from raw pixel screenshots and acts through a virtual mouse and keyboard, in a perceive–think–act loop: screenshot in, chain-of-thought over it, one action out, screenshot again. Deliberately no special web or OS APIs, which is what gives it the run-anywhere property.
It runs in a cloud-contained virtual browser on OpenAI’s infrastructure, so it cannot reach your filesystem, and it stops to ask before high-stakes actions like entering a password or submitting a purchase. Numbers: 38.1% on OSWorld, 58.1% on WebArena, 87% on WebVoyager.
The 87% is the one people quote. It is also the most constrained environment on the chart, and the 38.1% is the one that resembles the job.
Anthropic’s computer use
Anthropic went the other way: an API that lets Claude drive an entire desktop, not a sandboxed browser tab. Spreadsheets, IDEs, terminals, anything with a window. The interesting detail from the announcement is that it was trained on a handful of simple pieces of software and generalised to programs it had never seen, which suggests it learned something about GUI conventions rather than about those specific apps.
The detail I keep thinking about is that they had to teach it to count pixels. Without accurate pixel geometry the model cannot express a click at all: it knows what it wants to press and cannot say where it is. That is a strange and very physical failure mode for a language model.
22.0% on OSWorld with extended steps, 14.9% with screenshots only, 56% on WebVoyager. Lower than CUA, in a much harder setting. Anthropic shipped it calling it experimental and error-prone, which was accurate and rarer than it should be.
The open-source layer
You do not need either of those to start. The building blocks are MIT-licensed and on your machine tonight:
- browser-use: Playwright underneath, exposes the page to an LLM as both DOM and screenshot, and turns “click the login button” into an actual click. Most grassroots browser agents are built on it.
- Taxy: a Chrome extension that flattens the DOM to a list of interactive elements with IDs, hands that to GPT-4, and executes whatever
click(id)comes back. Its own README is honest that many workflows confuse it. - Agent S: the academic entry, with hierarchical task decomposition and memory of past attempts. Open-sourced, and the part worth stealing is the interface design rather than the planner.
Earlier than all of these, Adept’s ACT-1 did the same thing in 2022 with a browser extension and a custom action space. It stayed proprietary and quiet, but it is visibly the ancestor of everything above.
Two ways to see a screen
Every one of these systems makes the same first decision, and it determines nearly everything downstream.
Pixels. CUA and Claude take a screenshot and read it. Anything that renders can be understood: a canvas app, a remote desktop, a video call, a game. The cost is that you are doing OCR and layout inference on every single step, and you cannot see what is not drawn: the disabled state, the value behind the mask, the element that exists but is scrolled out of view.
Structure. Taxy, browser-use and the accessibility-tree projects read the DOM or the ARIA tree and hand the model something like button[id=btn12] “Sign In”. No OCR errors, no coordinate arithmetic, dramatically cheaper. The cost is that the structure and the screen disagree more often than you would like: elements present in the DOM but invisible, menus that do not exist until hovered, whole interfaces painted onto a canvas with nothing underneath.
The accessibility tree is the underrated middle. It is a text representation of what is rendered, roles and labels included, which is closer to what a person perceives than the raw HTML is. GPT-ARIA tried this in 2023 and mostly discovered how many sites ship broken ARIA, which is its own finding.
My read: structure first because it is cheap and exact, pixels as the fallback for everything structure cannot see. Neither alone is enough, and the systems that use both are not doing it for redundancy; they are doing it because each one is blind to a different thing.
Two ways to act
Virtual input. Synthesise a real mouse move and a real click into the OS or the VM. Works on anything, because from the application’s side it is indistinguishable from a person. This is what the desktop-general agents do, and they have to.
Programmatic events. Tell the browser to dispatch a click on an element, the way Selenium does. Fast, exact, no coordinates, and only available where such an API exists. It also skips every behaviour that requires a real pointer, which is a longer list than people expect.
Here is what nobody does. Both approaches teleport. The cursor is at rest, and then it is on the target, and then it has clicked. No agent I looked at plots a trajectory, and the papers do not treat it as a variable worth having.
The habits nobody implemented
Watch a person use software they have never seen. They do three things that current agents do not, and all three are cheap.
They look coarsely, then finely
A person finds the region first (the toolbar, the sidebar, the modal) and only then the control. Agents process the whole screenshot and emit a target in one shot. You can see something like coarse-to-fine emerge in chain-of-thought (“I need the File menu, then Save”), but it is a happy accident of prompting rather than a property of the system. Making it explicit gives the model a checkpoint where it can still be wrong cheaply.
They hover before they commit
This is the one I would build first. Move the pointer to the target, pause, and then take a screenshot, before clicking. You get two things for one extra frame.
The first is a free correctness check: most UIs highlight what is under the cursor, so if the model expected “Delete” and the highlighted control says “Delete All”, you find out while it is still recoverable. Coordinate drift of a few pixels onto the adjacent button is a common failure and this catches it.
The second is that hovering is functionally required on a lot of the web. Dropdowns that open on hover, tooltips carrying the only disambiguating text, toolbars that appear on mouse-over. An agent that teleports and clicks never sees any of it, and its failure looks like a reasoning failure when it is a physics failure.
# what everything does today
click(x, y)
# what a person does
move_to(x, y) # along a path, not instantly
dwell(120) # let hover states fire
frame = screenshot() # is the highlighted control the one I meant?
if not matches(frame, expected_label):
replan()
click()That is one extra screenshot and about 150ms per action. Against a base rate where the best system on a real desktop finishes 38% of tasks, I would spend it.
They check that it worked
After clicking Send, a person looks for “Message sent.” Agents do have the next screenshot in context, so they can notice, but noticing is left to the model’s judgment rather than being a step. Making it a step is ordinary test automation: state the expected post-condition before acting, assert it after. The agent then stops claiming success it did not achieve, which is most of what makes these things exhausting to supervise.
Slower is also more trustworthy
There is a second-order argument for all of this that I did not expect to find convincing.
An agent that executes a hundred actions in two seconds is unsupervisable. By the time you have understood what it is doing it has done forty more things. An agent that moves a visible cursor at human pace, pauses over what it is about to press, and narrates its post-conditions can be watched and, critically, interrupted. You can say “no, not that” while it still matters.
So the human-like design buys reliability and legibility from the same change, and pays for both in latency. On a task where the alternative is doing it yourself, that is not a hard trade.
Making the agent behave more like a person is not anthropomorphism for its own sake. Every one of these habits exists because it caught a mistake often enough to become automatic.
What I built
The hover-then-verify idea is the part I actually wanted, so I pulled it out on its own: mouse-point resolves a natural-language description of a UI element to screen coordinates, so “the blue Submit button” becomes an (x, y) you can move to and inspect before committing. It is the targeting layer, deliberately separate from any planner, because the planner is the part that keeps changing.
The wider harness (decomposing an instruction into typed actions and executing them) is in ai_command_automation, which I wrote about separately.
If you want to try the dwell idea without building anything, the shortest path is Taxy: it already tells you where it is about to click, so inserting a delay and a DOM re-read before the click is a small diff against someone else’s working agent.
What I am not claiming
- I have not run a controlled comparison. I am arguing from failure modes I hit and from what the published evaluations do and do not measure, not from an ablation. Someone should run it; the experiment is not expensive.
- The benchmark numbers are vendor-reported and the environments are not the same, so CUA’s 38.1% and Claude’s 22.0% are not directly comparable: one is a managed browser sandbox, the other a full desktop. The chart puts them side by side because the gap to the human line is the point, not the gap to each other.
- None of this addresses the security problem. An agent with a mouse has every permission you have, and everything it reads on screen is untrusted input. That is a different post.
References
- Computer-Using Agent. OpenAI
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. Anthropic
- ACT-1: Transformer for Actions. Adept
- TaxyAI/browser-extension
- Agent S. An open agentic framework that uses computers like a human
- OSWorld and WebArena. The benchmarks, including their human baselines