Skip to content
ML2

agent-desktop: computer use for agents via the accessibility tree

Added Copyable prompt below

Published Last verified Released

Our verdict

Worth trying if you want your coding agent to open, click through and check native apps, not just web pages. Instead of reading screenshots and guessing pixels, agent-desktop returns a JSON snapshot of the app's real UI with a ref for every element, the agent acts on a ref, then snapshots again to verify. Actions run headless by default, without moving your pointer or taking focus. It hit 1.0 on 28 Oct 2026, moved under Vercel Labs, and was gaining about 2,300 GitHub stars a day on 2026-11-01 (about 5,000 total), but stars are not a quality signal. The limit is the app itself: it only sees what an app exposes to accessibility, and snapshots can contain whatever is on your screen.

Who it's for
Builders who want coding agents to drive or test native desktop apps on macOS, Windows or Linux
Leverage
Your agent can run build, open, click through and verify loops on native apps from the terminal, with structured output instead of screenshot guessing
Effort
Low
Date added

What it does

agent-desktop (vercel-labs/agent-desktop, Apache-2.0, npm agent-desktop, first released 2026-02-23, v1.0.2 on 2026-10-30) is a CLI and C library for macOS, Windows and Linux. snapshot returns a JSON tree of an app's interactive elements, each with a ref tied to that snapshot; click, type, set-value, scroll and about 50 other commands act on a ref, and a stale or ambiguous ref returns STALE_REF or AMBIGUOUS_TARGET so the agent re-snapshots instead of clicking the wrong thing. Every command prints one JSON envelope, and errors say whether a retry is safe. Dense apps can be mapped with --skeleton and drilled into by region. It also covers windows, clipboard, notifications and screenshots, and ships its own agent docs (agent-desktop skills get desktop). Platform notes: macOS needs Accessibility permission; Linux needs the graphical session with at-spi2-core, and mouse input and window control are X11 only. Risks: snapshots, clipboard reads and screenshots can hold private data, so keep them out of logs; the optional JEV helper sends screen descriptions to a model provider you configure; apps with poor accessibility labels are hard or impossible to drive.

How builders use it

  • Install with npm install -g agent-desktop and run agent-desktop version; on macOS run agent-desktop permissions --guide to grant Accessibility.
  • Add the skill to your agent with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop, or add the two-line AGENTS.md rule below.
  • Have the agent build your desktop app, launch it, snapshot it with -i, click through the flow by ref, and re-snapshot to check each step.
  • For busy apps such as Slack, start with snapshot --skeleton --compact and drill into one region with --root.

What to expect

Your agent can test and operate native apps in a structured observe, act, verify loop, in the background, without you handing over your mouse.

Copy this: Install and wire into your agent

# Official source: github.com/vercel-labs/agent-desktop (Apache-2.0)
npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide   # macOS only

# Option 1: install the skill
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

# Option 2: add to AGENTS.md
Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.

# The loop: observe, act on a ref, observe again
agent-desktop snapshot --app Finder -i
agent-desktop click @<snapshot_id>:e8
agent-desktop snapshot --app Finder -i

Related discoveries

See the newest picks →
  • ToolResearch & Automation

    n8n

    Workflow automation with A.I. agent nodes, self-hostable.

    Effort: Medium#Self-hostable

  • ToolResearch & Automation

    Zapier Agents

    A.I. teammates that act across thousands of apps, on command or on a schedule.

    Effort: Low

Get the best new A.I. picks in your inbox

The best new agent workflows and tools we’ve vetted, with a clear verdict on each. Short, occasional, useful.

More in Research & Automation →

Free. No spam. Unsubscribe anytime.