For thirty years the dominant pattern of computing was a person, a form, and a button. You filled in fields; you clicked submit; software did something. The vast majority of software ever written assumed that the entity on the other side of the screen was a human reading pixels and pressing keys. In late 2024 a new kind of entity started reading the same pixels and pressing the same keys, in volume, on its own initiative. The form did not change. The user did. This chapter is about what that means.
The form was the interface
Almost every digital workflow we have built rests on a graphical interface designed for a person. Bank transfers, expense reports, insurance claims, hotel bookings, government filings, internal SaaS dashboards — they all assume someone is sitting at a screen, parsing the layout, clicking the right element, typing into the right box. APIs exist for some of this work, but the long tail of corporate and consumer software has no useful API; the form is the only entry point.
For thirty years this was fine, because automating a UI was a thankless engineering project. Brittle screen-scraping libraries broke whenever the layout changed; robotic process automation vendors made a serviceable but expensive business gluing together fragile rule-based bots. The economics of automation against UIs was tilted heavily toward leaving the human in the loop.
Computer-use as a primitive
What changed is that a sufficiently capable model can now read a screenshot of a UI and produce a useful answer to "what should I click next, and where on the screen is it?" Anthropic's Claude with computer use, OpenAI's Operator, and Google's parallel offerings all expose roughly the same primitive: a controlled browser or virtual machine, a screenshot pipeline, an action API (click at coordinates, type, scroll, drag, take screenshot), and a model with vision good enough to navigate.
The architecture is unglamorous. The agent receives a goal ("book a flight to Berlin on the 12th"); it takes a screenshot; the model decides on an action (click the search field, type "Berlin"); the action is dispatched to a sandboxed browser; the next screenshot is read; the loop continues until the task is done or the agent gives up. Performance is bounded by the model's vision (can it find the right button?), its planning (does it know the right sequence?), and the latency of each round-trip (each action takes 5-15 seconds end-to-end).
The change relative to RPA is that the model generalizes. The same agent that booked a flight yesterday can fill out a tax form today, because both involve reading a layout and clicking the right element. There is no per-site engineering. The model is the engineering.
How it actually works
In production, computer-use agents combine three modalities of perception, not just pixels. They read the rendered screenshot for spatial understanding ("the submit button is in the lower-right"), the DOM tree for structural information ("this element is an input of type=date"), and the accessibility tree as a backup when the DOM is obfuscated. The most reliable agents fuse all three; pixel-only agents fail more often on complex sites because vision is still imperfect at recognizing low-contrast, oddly-styled, or modal-occluded interface elements.
Action dispatch happens through standard browser-control protocols: WebDriver, Chrome DevTools Protocol, or operating-system-level synthetic input on virtual machines. The agent does not ask the page for permission; it acts as if it were a user, which means it inherits whatever permissions the user it is acting as has. This is convenient for capability and dangerous for security; we'll come back to it.
What breaks
The realistic state of computer-use agents in 2026 is that they work well on common workflows on stable sites and break on long tails. Specifically: anything multi-factor authentication that requires a phone tap, anything CAPTCHA-protected (and CAPTCHAs have been redesigned in the last two years specifically to defeat AI agents), anything with anti-bot detection that fingerprints browser behaviour, anything with rapidly changing UIs, anything requiring drag-and-drop with precise targets, and anything where the cost of a wrong click is high enough that the operator has placed friction in the way.
On the WebArena benchmark, the best agents in late 2025 complete 50-70% of tasks; the median time per task is several minutes. A human can do the same tasks in seconds with a much higher success rate. The economic case for the agent is not speed; it is unit cost and parallelism — one agent can run a thousand tasks in parallel for less than the cost of the human who would do them in series.
Three failure modes are worth naming because they will recur. Drift: the agent slowly diverges from the goal across many steps, ending up on a related but wrong page. Loops: the agent retries the same failing action repeatedly because no observation falsifies its plan. Confabulated success: the agent claims to have completed the task but did not actually click the final submit button. Production systems mitigate each with verifier sub-agents, hard step caps, and human approval gates on consequential actions.
The end of the form-and-button era
The medium-term implication, assuming the trajectory continues, is the slow obsolescence of the GUI as the primary interface for routine business operations. The forms will still exist, because legacy systems do not vanish, but the user behind them will increasingly be software acting on behalf of a person, not the person directly. This has already happened for some workloads — automated trading, programmatic ad bidding, search-engine indexing — but it is now happening for the long tail of expense reports and insurance claims and government filings.
The realistic consequences will be uneven. Companies whose moat was a hard-to-use UI that customers tolerated will discover that an agent does not tolerate, it just leaves. Companies whose value was in the underlying service rather than the interface will be largely unaffected. The systems that held up well will be the ones with stable APIs (which agents can call directly without screenshotting) and the ones with such strong network effects that they survive any interface change.
What is genuinely lost is some of the ambient telemetry that came from human interaction — the subtle signals about user intent that a form could elicit through layout, that an agent reads as just another field to fill. What is genuinely gained is access: workflows that were too tedious for humans to bother with at scale become routine. The economic surplus this releases is real and large, and the question of who captures it is the subject of the next chapter.
Before answering it, we need to look at what is happening to the supply side of intelligence itself.