Vercel labs launches open-source tool for AI agent desktop interaction

Vercel Labs has launched agent-desktop, an open-source command-line interface (CLI) designed for AI agents to interact with desktop applications on macOS, Windows, and Linux. The tool, released on October 30, 2026, under the Apache-2.0 license, allows developers to control native apps without relying on vision models. It reads applications through the operating system’s accessibility tree, providing JSON snapshots with unique element references. Agent-desktop is a Rust-based CLI and C library that enables agents to perform actions through a structured observe-act-verify loop. It avoids using machine learning models and instead relies on strict ref resolution and error handling. The tool can reduce token usage by up to 96% compared to full-tree dumps, with significant savings in apps like Notion and Apple Maps. The tool supports multiple programming languages and integrates with Electron web contents via DevTools. Developers can install it using npm, Bun, Yarn, or pnpm, with native binaries for x64 and ARM64. Vercel Labs also provides bundled documentation and skills for agents to learn the tool effectively.