Agent Desktop CLI enables AI agents to control apps via OS accessibility trees
TL;DR. A new native desktop automation CLI, Agent Desktop, allows AI agents to interact with applications through OS accessibility trees without pixel matching. - Built with Rust, the CLI provides structured JSON output and deterministic element references for AI-optimized workflows. - It supports 58 commands for observation, interaction, keyboard, mouse, and window management across any macOS application. - The tool offers progressive skeleton traversal for up to 96% token reduction and headless interactions via accessibility APIs.
- Agent Desktop CLI provides native desktop automation for AI agents using OS accessibility trees.
- It avoids screenshot/pixel matching, offering structured JSON output and deterministic element references.
- The Rust-based tool supports various interactions across macOS applications and reduces token usage via progressive traversal.
- A C-ABI cdylib is available for integration with Python, Swift, Go, Ruby, and Node.
Sources
- useokf.com — useokf.com
- Show HN: I spent 3 months making desktop automation stop lying to AI agents — github.com
- bleepingcomputer.com — bleepingcomputer.com