Macbeth — Open-source Computer Use for macOS

Give the agent you already use hands and eyes on your Mac.

See and operate Mac applications through MCP or TypeScript, including native and Electron interfaces.

Structured UI access when possible. Screenshots and OCR when necessary.
Think Playwright for your entire Mac.

Simulated Macbeth walkthrough showing System Settings, Electron interaction, and screenshot OCR

_{Simulated walkthrough of Macbeth's current MCP tool surface: System Settings, Electron interaction, and screenshot/OCR.}

Quickstart · Examples · Why Macbeth? · TypeScript · Security · Skills

Quickstart

Give Claude Code access to Macbeth:

claude mcp add macbeth -- npx -y macbeth

Then ask it to verify the connection:

Using Macbeth, list the running applications, connect to Finder, inspect its front window, and summarize the first few controls you can see. Do not perform any other actions.

Macbeth ships with a prebuilt, signed and notarized universal daemon. No Swift toolchain or manual service installation is required. If setup fails, run npx macbeth doctor; it prints a self-contained diagnostic prompt you can paste back into your agent.

Any MCP client that can launch a stdio server can use npx -y macbeth. See MCP setup for generic configuration, updates, and the complete tool list.

Across your Mac

Work inside Unity, Logic Pro, and other complex tools

“In Logic Pro, set the project tempo to 128 BPM and start playback.”

Macbeth combines structured controls, menus, keyboard input, screenshots, OCR, and application-specific guidance. Logic Pro has a dedicated skill. Applications with thinner accessibility trees, including parts of Unity, can combine structured controls with keyboard, menu, screenshot, and OCR fallbacks.

Automate macOS itself

“Open System Settings, find the firewall controls, and report their current state without changing anything.”

Macbeth can inspect named controls and values, select native menu items, and use keyboard input when a semantic action is unavailable.

Use applications even when they have no API

“Open HEY, find the newest visible thread from the design team, and summarize it without sending or changing anything.”

Macbeth can work through exposed Electron interfaces even when the service does not provide the API your agent needs. HEY works through Macbeth's generic Electron support; results vary with the application's accessibility tree and version.

See what the accessibility tree misses

“Capture the frontmost Notes window and extract all visible text.”

ScreenCaptureKit screenshots and local Vision OCR provide a fallback for content that macOS does not expose as useful accessibility data.

Build repeatable scripts and tests

The same engine is available as a TypeScript API with lazy locators, auto-waiting actions, explicit state waits, screenshots, and application connections.

Why Macbeth?

• Use your own agent. Macbeth is a control layer, not a model or agent runtime.
• Structured when possible. Query roles, labels, values, enabled state, hierarchy, and stable handles instead of relying only on coordinates.
• Visual when necessary. Fall back to window capture and OCR when accessibility metadata is thin.
• Application-aware. Versioned skills preserve workflows, shortcuts, and known failure modes.
• Scriptable. Use the same primitives interactively over MCP or directly from TypeScript.

Structured before pixels

Accessibility information can itself be incomplete or misleading. Macbeth is a hybrid interface: it uses stronger structured primitives when macOS exposes them and keeps visual and system-level fallbacks available when it does not.

Security and limitations

• Macbeth requires macOS 14 or newer and Node.js 20 or newer.
• Accessibility permission is required for UI inspection and interaction. Screen Recording is required only for screenshots and window OCR.
• Application accessibility quality varies. Custom-rendered canvases, games, and some creative tools may require menus, screenshots, OCR, keyboard input, or application-specific guidance.
• Interface changes between application versions can break specialized workflows.
• The interaction glow makes Macbeth activity visible, but it is not an approval system or a complete audit log.
• Agents can make mistakes. Use human review for destructive, financial, publishing, or privacy-sensitive actions.

Macbeth uses a local stdio MCP server and a Unix-domain socket; it does not open a TCP listener. Screenshots requested through MCP are written to a temporary directory and are not automatically deleted. Read the security model for data flow, permission revocation, local-process risks, and safer operating practices.

TypeScript API

npm install macbeth

import { connect } from "macbeth";

const app = await connect("TextEdit");
await app.window("Untitled").textField().fill("Hello from Macbeth");
await app.pressKey("s", ["cmd"]);

See Developing with Macbeth for locators, screenshots, Electron behavior, lifecycle, and tests.

Teach Macbeth an application

A Macbeth skill is a SKILL.md file, optionally accompanied by runnable scripts, that teaches an agent reliable workflows and pitfalls. Call load_skill with no arguments for the core Macbeth guide. Bundled app skills cover Calendar, Contacts, Logic Pro, Mail, Maps, Messages, Music, Notes, Reminders, Safari, System Settings, and generic Electron applications.

You can add a skill without changing the Swift daemon. Start with the contribution guide and include a reproducible prompt, tested application version, expected result, and honest failure modes.

Documentation

Requirements

• macOS 14 (Sonoma) or later
• Node.js 20+
• Swift 6.0+ only when building the daemon from source

License

MIT