Do agents need their own computers?

Hypothesis

Some work only exists behind a real screen — web apps with no API, legacy tools, anything that needs a browser session. Cloud "agent desktop" services sell this by the seat. Could we self-host a fleet of agent-driven Linux desktops at a fraction of the price, and give every agent its own computer?

What we built

Reverse-engineered how the commercial offerings work (containerised Linux desktops driven by screenshot + mouse/keyboard over a websocket), then built our own: Docker containers running a full desktop, viewable in the browser via noVNC, driven by a low-cost vision model that receives screenshots and returns clicks. The text-only orchestrator agent never sees pixels — it delegates desktop tasks to the vision driver and gets results back. Current cost per completed desktop task: one to two cents.

Learnings

• The driver loop needs unglamorous guardrails more than a smarter model: a loop-breaker that detects repeated near-identical clicks (the model will nudge coordinates one pixel to evade exact-match detection!), coordinate normalisation, and pruning old screenshots from context.
• Cheap open-weight vision models are good enough for UI driving — the economics work at roughly a cent per task, which changes what's worth automating.
• Isolation tiers matter commercially: plain containers are fine for our own agents, but untrusted client workloads need VM-grade isolation — and not every server can provide it. Check for virtualisation support before you architect.

Log

• 2026-07-11 — Fleet pattern feeding the client "agent pod" product; hardening applied.
• 2026-07-09 — First desktop live, vision driver working end-to-end, auto-start on container boot.