agent-desktop: computer use for agents via the accessibility tree
Added Copyable prompt below
Published Last verified Released
Our verdict
Worth trying if you want your coding agent to open, click through and check native apps, not just web pages. Instead of reading screenshots and guessing pixels, agent-desktop returns a JSON snapshot of the app's real UI with a ref for every element, the agent acts on a ref, then snapshots again to verify. Actions run headless by default, without moving your pointer or taking focus. It hit 1.0 on 28 Oct 2026, moved under Vercel Labs, and was gaining about 2,300 GitHub stars a day on 2026-11-01 (about 5,000 total), but stars are not a quality signal. The limit is the app itself: it only sees what an app exposes to accessibility, and snapshots can contain whatever is on your screen.
- Who it's for
- Builders who want coding agents to drive or test native desktop apps on macOS, Windows or Linux
- Leverage
- Your agent can run build, open, click through and verify loops on native apps from the terminal, with structured output instead of screenshot guessing
- Effort
- Low
- Date added
What it does
agent-desktop (vercel-labs/agent-desktop, Apache-2.0, npm agent-desktop, first released 2026-02-23, v1.0.2 on 2026-10-30) is a CLI and C library for macOS, Windows and Linux. snapshot returns a JSON tree of an app's interactive elements, each with a ref tied to that snapshot; click, type, set-value, scroll and about 50 other commands act on a ref, and a stale or ambiguous ref returns STALE_REF or AMBIGUOUS_TARGET so the agent re-snapshots instead of clicking the wrong thing. Every command prints one JSON envelope, and errors say whether a retry is safe. Dense apps can be mapped with --skeleton and drilled into by region. It also covers windows, clipboard, notifications and screenshots, and ships its own agent docs (agent-desktop skills get desktop). Platform notes: macOS needs Accessibility permission; Linux needs the graphical session with at-spi2-core, and mouse input and window control are X11 only. Risks: snapshots, clipboard reads and screenshots can hold private data, so keep them out of logs; the optional JEV helper sends screen descriptions to a model provider you configure; apps with poor accessibility labels are hard or impossible to drive.
How builders use it
- Install with npm install -g agent-desktop and run agent-desktop version; on macOS run agent-desktop permissions --guide to grant Accessibility.
- Add the skill to your agent with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop, or add the two-line AGENTS.md rule below.
- Have the agent build your desktop app, launch it, snapshot it with -i, click through the flow by ref, and re-snapshot to check each step.
- For busy apps such as Slack, start with snapshot --skeleton --compact and drill into one region with --root.
What to expect
Your agent can test and operate native apps in a structured observe, act, verify loop, in the background, without you handing over your mouse.
Copy this: Install and wire into your agent
# Official source: github.com/vercel-labs/agent-desktop (Apache-2.0)
npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide # macOS only
# Option 1: install the skill
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
# Option 2: add to AGENTS.md
Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.
# The loop: observe, act on a ref, observe again
agent-desktop snapshot --app Finder -i
agent-desktop click @<snapshot_id>:e8
agent-desktop snapshot --app Finder -iRelated discoveries
See the newest picks →- ToolResearch & Automation
n8n
Workflow automation with A.I. agent nodes, self-hostable.
Effort: Medium#Self-hostable
- WorkflowResearch & Automation
Start from n8n community workflow templates
Thousands of community templates: copy one, swap credentials, ship.
Effort: Low
- WorkflowResearch & Automation
Weekly competitor digest with n8n + Firecrawl
Scrape competitor pages weekly, summarise changes with an LLM, email yourself.
Effort: MediumPrompt
- ToolResearch & Automation
Zapier Agents
A.I. teammates that act across thousands of apps, on command or on a schedule.
Effort: Low
Get the best new A.I. picks in your inbox
The best new agent workflows and tools we’ve vetted, with a clear verdict on each. Short, occasional, useful.