This MCP server provides “accessibility-first desktop control” for an agent, including background input and an agent pointer, plus features for zoom and browser/Chrome-related control. It is intended for desktop and GUI automation use cases across multiple operating systems.
🛠️ Key Features
Accessibility-first desktop control for agents
Background input
Agent pointer
Zoom control
Chrome support
🚀 Use Cases
Desktop automation and GUI automation
Browser automation, including Chrome
Agent-driven interaction with operating systems and applications
⚡ Developer Benefits
Model Context Protocol compatibility (listed under MCP topics)
Supports development workflows involving AI agents (e.g., computer-use-agent)
Cross-platform targeting indicated by OS topics: macOS, Windows, Linux
⚠️ Limitations
README excerpt provided is incomplete and does not describe constraints, permissions, or supported environments beyond the listed topics.
Computer Use is an open-source MCP server — a computer-use agent (CUA) backend — that lets any coding agent use your computer the way a person does. It reads the screen through accessibility trees, clicks and types in the background so your mouse stays yours, shows an agent pointer where it is working, zooms in on small text, and drives tabs in your signed-in Chrome — on macOS, Windows and Linux.
Works with Claude Code, Codex, Cursor and MT Code, or any other MCP client, with any model — no vision model is required for interaction.
Built by Munim Technologies as the Computer Use engine of MT Code, and published here on its own.
Download the latest binary for your platform from Releases (munim-computer-use-macos-universal.zip, munim-computer-use-windows-x64.zip) or build from source.
Put it somewhere on your PATH (/usr/local/bin/munim-computer-use, or %LOCALAPPDATA%\Programs\munim-computer-use\munim-computer-use.exe).
macOS only: run munim-computer-use request-permissions once to be prompted for Accessibility and Screen Recording.
Register it with your agent:
sh
# Claude Code — fastest: the npm launcher fetches the signed binary on first run.# Add `--scope user` to register it for every project instead of just this one.
claude mcp add munim-computer-use -- npx -y munim-computer-use
# or point at a downloaded binary
claude mcp add munim-computer-use -- /usr/local/bin/munim-computer-use
sh
# Codex — writes the entry below into ~/.codex/config.toml for you (needs a Codex CLI# with `codex mcp`; check with `codex mcp --help`)
codex mcp add munim-computer-use -- npx -y munim-computer-use
toml
# Codex — ~/.codex/config.toml, if you would rather edit it yourself[mcp_servers.munim-computer-use]command = "npx"args = ["-y", "munim-computer-use"]
sh
# Cursor — no MCP subcommand in its CLI, so write the config. ~/.cursor/mcp.json applies# to every project; .cursor/mcp.json in a repo applies to that one.mkdir -p ~/.cursor && [ -s ~/.cursor/mcp.json ] || echo'{}' > ~/.cursor/mcp.json
jq '.mcpServers["munim-computer-use"] = {"command":"npx","args":["-y","munim-computer-use"]}' \
~/.cursor/mcp.json > ~/.cursor/mcp.json.tmp && mv ~/.cursor/mcp.json.tmp ~/.cursor/mcp.json
json
// Cursor — the entry that produces, in .cursor/mcp.json{"mcpServers":{"munim-computer-use":{"command":"npx","args":["-y","munim-computer-use"]}}}
Then ask: "Open Safari, find the cheapest flight to Denver on Tuesday and put it in a note." The agent reads the UI with get_app_state, acts by element id, and verifies with screenshot.
Capability matrix
Munim Computer Use is the highlighted first column; the others are the computer-use servers people reach for. each cell comes from that project's own README or docs in September and October 2026 (sources under Credits and license). ✅ present · ❌ absent or not documented · ⚠️ partial.
Capability
Munim Computer Use
Cua Driver
Codex Computer Use
OpenAI Agents API
Anthropic reference demo
Windows-MCP
MacOS-MCP
open-computer-use
computer-use-mcp (zavora)
Notes
macOS
✅
✅
✅
n/a
❌
❌
✅
✅
✅
The Anthropic demo drives a Linux desktop inside Docker, not your machine. The OpenAI Agents API (September 2026) runs a browser OpenAI hosts, so it never drives your machine either.
Windows
✅
✅
✅
n/a
❌
✅
❌
✅
✅
Windows-MCP is Windows only; MacOS-MCP is macOS only.
Linux
✅
✅
❌
n/a
✅ (sandbox)
❌
❌
✅
✅
Munim Computer Use uses AT-SPI + X11; native Wayland apps get element actions but not coordinate clicks.
Accessibility tree with element ids
✅
✅
⚠️
❌
❌
✅
✅
✅
✅
Codex reads the accessibility tree first on macOS and works from screenshots on Windows; the Agents API and the Anthropic demo are screenshot-driven. Ids let the agent press the button instead of a pixel.
Background input (your mouse never moves)
✅
✅
⚠️
n/a
n/a
❌
❌
❌
❌
Munim Computer Use addresses events to the target window: always on macOS (SkyLight and per-process events), for UI Automation patterns and classic Win32 controls on Windows; other Windows UI and all Linux pointer input (XTEST) still move the real pointer. Codex works in the background on macOS with cursors of its own, and takes over the pointer on Windows. Cua Driver delivers in the background by default and falls back to the foreground when that does not land. The other servers in this table drive the real cursor.
Agent pointer overlay
✅
✅
✅
❌
❌
⚠️
❌
❌
❌
Windows-MCP flashes a border around captures; Codex and Cua Driver draw their own cursor on your screen.
Zoom into a region at full resolution
✅
✅
❌
❌
✅
❌
❌
❌
❌
Anthropic's toolset has zoom; here it is a tool on every platform.
Screenshots carry screen-coordinate mapping
✅
⚠️
n/a
n/a
n/a
❌
❌
❌
❌
Origin and pixels-per-point in every capture, so clicks from Retina or downscaled images land.
Hover, wait, label query
✅
⚠️
⚠️
❌
⚠️
✅
⚠️
❌
⚠️
Windows-MCP has Wait/WaitFor; MacOS-MCP has Wait; Anthropic has wait/mouse_move.
Your signed-in Chrome, own tab group or yours
✅
⚠️
⚠️
❌
❌
⚠️
❌
❌
❌
Codex uses its in-app browser; Cua Driver attaches to a browser over CDP; Windows-MCP reads the DOM of open browsers. Munim Computer Use opens its own labelled tab group in your real Chrome, and can also take over a tab you already have open when you ask it to.
Act and observe in one call
✅
✅
n/a
n/a
✅
❌
❌
❌
❌
return_state on any action appends the fresh accessibility tree or page snapshot, so one call acts and checks. Cua Driver's run_steps runs several actions and reports what changed. The Anthropic demo returns a screenshot after each action; Codex and the Agents API run the loop themselves.
Several actions in one call
✅
✅
n/a
n/a
❌
⚠️
❌
❌
✅
Cua Driver and computer-use-mcp (zavora) run a list of steps in one call with run_steps; Windows-MCP's MultiSelect and MultiEdit act on several elements at once. Munim Computer Use's run_steps does the same, checks the whole list before running any of it, and stops at the first failure.
Read a page's text in a signed-in tab
✅
⚠️
❌
❌
❌
⚠️
❌
❌
❌
browser_read returns the page as text with headings, in chunks, with a query filter, from a background tab. Windows-MCP scrapes pages.
Sign-in without the model seeing the password
✅
❌
❌
✅
❌
❌
❌
❌
❌
browser_request_credentials opens a Chrome window naming the site's real origin; what the user types goes into the page and never into the conversation. The Agents API hands sign-in to the developer's app the same way. Desktop password fields refuse typing by default.
Per-app and per-site allow / ask / block rules
✅
❌
⚠️
⚠️
❌
❌
❌
❌
❌
A policy file the user controls; ask shows an approval prompt once per task. Codex asks before each new app and keeps an "Always allow" list; the Agents API asks the developer's app before each new website.
Works with any MCP client
✅
✅
❌
❌
❌
✅
✅
✅
✅
Codex Computer Use is Codex only; the Anthropic demo is Claude only.
Identical tool surface on every platform
✅
⚠️
n/a
n/a
n/a
n/a
n/a
⚠️
✅
38 tools with byte-identical schemas across the Swift and Rust servers. Cua Driver keeps shared parameters in step with a CI check, with a few tools on one platform only.
Prebuilt signed binaries + npm launcher
✅
✅ (script)
✅
n/a
❌
❌
❌
✅ (npm)
✅ (npm)
macOS universal (Developer ID signed) and Windows x64 on Releases.
The chart counts the table above (n/a cells are left out), so it shares the table's bias: we picked the rows.
Benchmarks
Reading a window is the step an agent repeats most, so every server we could run reads the same file manager window on each system as the folder grows from 100 to 20,000 files: File Explorer on Windows, Finder on macOS and Thunar on Linux. Each one ran its latest release on 4 November 2026. munim-computer-use is the fastest at every folder size on all three, and returns the fewest tokens on all three.
Windows 11 · File Explorer
Server
Read tool
100 files
1,000
5,000
20,000
Tokens per read (≈)
Files listed
munim-computer-use 0.10.0
get_app_state
0.12 s
0.12 s
0.13 s
0.12 s
2.0K
16
munim-computer-use 0.9.3
get_app_state
0.12 s
0.12 s
0.13 s
0.13 s
2.0K
16
Cua Driver 0.38.0
get_window_state, include_screenshot: false
0.15 s
0.16 s
0.15 s
0.16 s
3.9K
17
Windows-MCP 0.8.8
Snapshot
15.37 s
15.71 s
15.24 s
15.59 s
1.2–3.9K
0
computer-use-mcp 7.8.0 (zavora)
get_ui_tree (COMPUTER_USE_PROFILE=full)
0.44 s
0.40 s
0.43 s
0.40 s
9.3–9.4K
17
open-computer-use 1.1.3
get_app_state, include_screenshot: false
3.09 s
2.96 s
3.23 s
3.22 s
4.9K
17
Windows-MCP 0.8.8 takes about 15 s per Snapshot on this PC, where 0.8.5 took 0.7 s on the same windows the same day (and in our 27 October run), so the slowdown is in that release. At 1,000 files its last read left the File Explorer window out (1.2K tokens, no files). munim lists 16 rows and the others 17: munim's on-screen check leaves out the row at the bottom edge of the list.
macOS 27 · Finder
Server
Read tool
100 files
1,000
5,000
20,000
Tokens per read (≈)
Files listed
munim-computer-use 0.10.0
get_app_state, window: 0
0.04 s
0.05 s
0.06 s
0.10 s
1.0K
18
munim-computer-use 0.9.3
get_app_state, window: 0
0.04 s
0.05 s
0.06 s
0.10 s
2.3–2.4K
18
Cua Driver 0.38.0
get_window_state, include_screenshot: false
0.29 s
0.29 s
0.30 s
0.35 s
2.4K
13
computer-use-mcp 7.8.0 (zavora)
get_ui_tree (COMPUTER_USE_PROFILE=full)
1.16 s
1.18 s
1.16 s
1.17 s
22.4K
38
open-computer-use 1.1.3
get_app_state, include_screenshot: false
0.13 s
0.30 s
0.98 s
3.59 s
1.1K
20
MacOS-MCP 0.4.7
Snapshot, use_vision: false
0.36 s
2.26 s
8.63 s
73.91 s
2.6–17.3K
18
A MacBook Air (M1) on macOS 27.2. munim reads one window here (window: 0), as the others do; without it, get_app_state lists every Finder window. Cua Driver ran with mcp --direct and open-computer-use with OPEN_COMPUTER_USE_DISABLE_APP_AGENT_PROXY=1, their documented switches for running inside the process that holds the permissions, since their own apps had none on this Mac. Cua Driver's read stops at 99 elements, which leaves out part of the file list; zavora lists rows scrolled out of view too. MacOS-MCP's reads swing between about 0.3 s and 17 s at 5,000 files and take over a minute at 20,000.
Linux · Thunar
Server
Read tool
100 files
1,000
5,000
20,000
Tokens per read (≈)
Files listed
munim-computer-use 0.10.0
get_app_state
0.19 s
0.20 s
0.24 s
0.23 s
1.2K
33
munim-computer-use 0.9.3
get_app_state
0.51 s
0.52 s
0.55 s
0.20 s
0.4–1.9K
33
Cua Driver 0.38.0
get_window_state, include_screenshot: false
0.66 s
0.30 s
0.29 s
0.29 s
2.3–5.7K
0
computer-use-mcp 7.8.0 (zavora)
get_ui_tree (COMPUTER_USE_PROFILE=full)
0.85 s
0.91 s
0.88 s
0.86 s
28.6K
0
open-computer-use 1.1.3
get_app_state, include_screenshot: false
1.58 s
2.89 s
4.93 s
12.90 s
26.0–42.7K
175
Ubuntu in WSL 2 on the same laptop, X11 (Xvfb, 1920×1080) with bspwm and AT-SPI. Thunar's default icon view tells assistive technology nothing about its files, so every server read its details view. Only munim lists exactly the 33 rows in view: Cua Driver's read listed files only at 100 files, zavora's tree lists none of them, and open-computer-use lists every row it has, in or out of view, and timed out at 20,000 files on some calls. munim 0.9.3 read nothing of the 20,000-file folder, because AT-SPI refuses to hand over more than 65,536 children at once; 0.10.0 walks such a list by index.
How it was measured
For each folder size a fresh file manager window was opened in front and every server was started once, called over stdio with its own read tool (text only where the tool has a switch for it), one warm-up call dropped, then 5 timed calls; the whole run was done twice, so each time is the median of 10. Tokens are estimates at 4 characters per token. "Files listed" counts the folder's file names in the 1,000-file read. Terminator was left out: its get_window_tree picks the first explorer process, which is the taskbar, and it takes no pid. The scripts are in a scratch kit, not this repository; ask in an issue if you want them.
Acting
On the same Windows PC: the five clicks of 1 + 2 = on Calculator (clear, 1, +, 2, =), each a separate call by element id, after one read. Median of 3 runs.
Server
Five clicks
Per click
munim-computer-use 0.10.0
1.70 s
0.34 s
Cua Driver 0.38.0
2.31 s
0.46 s
Most of a click is the agent pointer flying to the target so you can see where the agent acts. run_steps sends all five in one call.
Every Windows release
The same window read by every Windows release, from the release binaries, in two rounds of all 17 with Windows-MCP in each round, plus 0.8.0 measured against 0.7.3 the same way, on 27 October 2026 (an earlier run than the ones above, so the times differ a little; Windows reads have not changed since 0.8.0, and 0.9.3 and 0.10.0 measured alike above). 0.7.0 is where get_app_state started reading only what is on screen and fetching each element's attributes in one request; 0.8.0 fetches the whole window in one request instead of one per element, with the same output.
The numbers
Release
Read time (median)
Characters
Tokens (≈)
0.2.0
0.40 s
9,860
2.5K
0.3.0
0.43 s
9,860
2.5K
0.3.1
0.38 s
9,860
2.5K
0.4.0
0.38 s
9,860
2.5K
0.4.1
0.38 s
9,860
2.5K
0.4.2
0.39 s
9,860
2.5K
0.4.3
0.40 s
9,860
2.5K
0.4.4
0.38 s
9,860
2.5K
0.5.0
0.38 s
9,860
2.5K
0.5.1
0.37 s
9,860
2.5K
0.5.2
0.43 s
9,860
2.5K
0.6.0
0.37 s
9,860
2.5K
0.6.1
0.41 s
9,860
2.5K
0.7.0
0.23 s
7,499
1.9K
0.7.1
0.28 s
7,499
1.9K
0.7.2
0.25 s
7,520
1.9K
0.7.3
0.25 s
7,520
1.9K
0.8.0
0.16 s
7,515
1.9K
Windows-MCP 0.8.5 Snapshot
0.75 s
14,929
3.7K
node scripts/readme-charts.mjs redraws the charts; the coverage chart is counted from the capability matrix.
Why it works well
Accessibility first, pixels second.get_app_state returns the app's accessibility tree with stable element ids, so the agent presses the button instead of guessing at a coordinate. It costs a fraction of the tokens of a screenshot and it is what scores highest on OSWorld-style tasks. Screenshots are for verifying and for content the tree cannot describe.
What is on screen, fast.get_app_state lists what is in view: a list or table gives its visible rows, and anything scrolled away is skipped and counted (offscreen: true lists it too). Attributes are read in one request per element, and on Windows the whole window comes back in a single request. Rows of a list print one to a line. A Finder window reads in about 0.05 s, File Explorer in about 0.12 s and Thunar in about 0.2 s, faster than any other server we measured on each system (see Benchmarks).
Dialogs first. While a sheet, alert or popover blocks a window, only its controls are listed, with a note saying so, so the agent answers the dialog instead of clicking behind it.
Electron and Chromium apps (macOS). Slack, VS Code, Discord and other Chromium-based apps expose their tree only to a client that asks; get_app_state asks, and says so while a background app is still building it.
Text the tree cannot describe. When an app exposes no labelled controls, its window is read with on-device OCR (Apple Vision on macOS, Windows OCR on Windows) and the text is listed with ids that click, hover, right_click and drag accept. The image never leaves the machine. ocr: "always" or "never" overrides it.
Ids that fail instead of misfiring. Every id is rechecked before an action. If its element is gone, or its role or name changed (a dialog swapped its buttons, a list re-sorted), the action fails with a clear error instead of acting on whatever sits there now.
Waits as long as the app takes.return_state waits for the app's accessibility notifications to stop instead of a fixed pause: about 230 ms for a field edit (was 350 ms), and long enough for a sheet to finish sliding in.
Background control. Events are addressed to the target window (SkyLight on macOS, UI Automation patterns and posted window messages on Windows). On macOS the agent never takes your mouse or keyboard; see Works alongside you for the exact guarantee on each platform.
Pointer overlay, not your pointer. A soft lavender agent pointer shows where the agent is acting. Your cursor is untouched.
Coordinates that land. Every screenshot and zoom carries its screen origin and pixels-per-point. zoom captures any region at full physical resolution.
Your browser, your logins. The Chrome extension gives the agent its own labelled tab group in your signed-in Chrome, and leaves your tabs alone unless you point it at one.
Or the tab you already have open.browser_list_tabs all=true shows every tab in the browser and browser_use_tab takes one over in place — useful when the page is already signed in or mid-flow and re-opening the URL would throw that away. An adopted tab is not moved into the agent's group, not activated and not reloaded; cleanup releases it rather than closing it, and browser_release_tab hands it back early.
Tabs the agent's clicks open are the agent's too. A target=_blank link or popup opened by an agent click or key press within 10 seconds joins that task's tabs and group, and its cleanup closes it. A tab opened from an agent tab without a recent agent action stays yours.
Parallel tasks, one extension. Every MCP process gets its own tab group, and one process can run several tasks by passing a stable session_id on its browser calls. A task cannot drive or adopt another task's tabs, and cleanup (including a process exiting) closes only its own. Any number of MCP processes share the one extension: the first owns it and the rest go through it, and if the owner exits another takes over without closing anyone's tabs. Tasks share Chrome's cookies and logins, and desktop apps and the clipboard are not isolated.
Model-agnostic. No vision model is required for interaction; local models work too.
Look → act → verify.hover for mouse-over menus, wait for loads, query to find a control by label without reading a whole tree.
Act and look in one call. Pass return_state: true to any action (click, type_text, set_value, browser_click, browser_navigate, …) and the result carries the app's fresh accessibility tree, or the page's current snapshot. Browser actions report readiness separately: a loaded tab is not proof that delayed SPA UI has rendered. Use wait_for_selector for an expected visible element (see below). state_query narrows the desktop tree the same way query does.
Several steps, one call.run_steps takes a list like click the field, type, press return and runs it in order, each step with the same checks as a direct call. The whole list is checked before anything runs, the first failing step stops the rest, and the result says what ran and what did not. Put return_state on it to see the app after the last step.
Pages as text.browser_read returns a page's readable text with its headings, including what is scrolled out of view, from a tab in the background. It comes in chunks you can continue with offset, query keeps only the matching lines under their heading, and include_links lists the links.
Frames too. A support chat, payment form or sign-in box embedded in a page is its own document, often from another site. browser_snapshot lists its controls after the page's, marked (in frame "…"), and their indices work with browser_click, browser_fill and browser_upload_file; browser_read adds each frame's text under a [frame "…"] line, and wait_for_selector looks inside frames. Your site rules apply to a frame's own site, and browser_request_credentials never fills a field inside a frame.
See a tab without taking your window.browser_screenshot captures what one of the agent's tabs shows, frames included, while it stays in the background: the tab is not selected, its window is not raised and the screen is not captured, so you keep working in your own tab. browser_select_tab bring_to_front=true is the only browser call that takes your window, and it is a last resort.
JavaScript dialogs, answered on purpose. When a page opens an alert, confirm or prompt, browser_snapshot shows its type, message and id, and other commands on that tab say a dialog is open instead of timing out. browser_handle_dialog accepts or dismisses it, with text for a prompt, in a background tab without the keyboard. A dialog that replaced the one the agent read is refused, and a dialog never counts as an answer on its own.
Why a page is broken.browser_page_errors lists the uncaught exceptions and console errors a tab's page has logged since it loaded (as far as Chrome keeps them), and the requests that failed (network errors and HTTP 400 or above) since the agent attached, the latest 50, with URLs stripped of query strings and fragments. Exceptions and console errors are read from Chrome's own record during the call, so the DevTools Runtime domain, which bot checks on signed-in sites look for, is not left on in between. Headers, cookies and bodies are never read, and the tab is not reloaded or shown.
Passwords stay out of the conversation.browser_request_credentials asks the user to sign in in a small Chrome window that shows the site's real origin. What they type goes straight into the page's fields and is never returned to the model. browser_snapshot never shows a password field's value, and on the desktop, typing into password fields is refused by default.
Pick up what you are looking at.get_app_state and screenshot take app: "frontmost" for the app in front of the user, so a client can hand the agent "this window" in one step.
Works alongside you
The agent has its own pointer; yours stays yours.
macOS — guaranteed. Every action goes through accessibility (press, set value, select text, show menu, scroll bars) or through events addressed to the target app's process and window. The server never moves your pointer, never posts into the system-wide input stream, and never holds or blocks your input, so you can keep clicking and typing in other apps while the agent works — even in the same app, on another window. Events come from a private source, so a modifier you are holding does not leak into the agent's clicks. The target app may be brought forward when that is the point of the step (activate_app, or handing you a password field), but not on every action.
Command shortcuts sent to a background app run through its menu bar (macOS gives menu key equivalents only to the front app), so cmd+W, cmd+S and the like work without bringing it forward. A minimized window, or a hidden app, is moved for the moment it needs pixels or input (typing, a coordinate click, a screenshot, OCR) onto an invisible display, where it works while nobody sees it; it is minimized or hidden again after 20 idle seconds, when another app's window needs the display, or when the server exits. If you bring the app up yourself meanwhile, its windows come back on screen where they were. Accessibility actions on such a window (press, set value) need none of this. COMPUTER_USE_PARK_WINDOWS=0 turns it off, and those actions then return an error asking for the window to be restored.
The exceptions refuse instead of borrowing your pointer: a click, hover or scroll with no target app (coordinates over the desktop before any get_app_state), and drags that leave the source window (between apps, or onto the desktop). The error says what to pass instead.
Windows — best effort, reported. Element presses (Invoke), set_value, select_text and scrolling through UI Automation's ScrollPattern never touch your pointer, and classic Win32 controls also take clicks, hovers, wheel and drags as posted window messages. Other UI (Chromium, Electron, WPF, UWP) only reacts to real mouse input, so those clicks, hovers and drags move your pointer, and type_text/press_key go to the focused window. Whenever the real pointer was used, the result says via cursor. A coordinate action on an element that another app's window covers is refused rather than clicking that window, and pointer and keyboard actions on a minimized window return an error asking for it to be restored (accessibility actions still work on it).
Linux — pointer actions use it. XTEST input moves the real pointer and goes to the focused window, and results say via cursor. Element actions through AT-SPI (press, set value, insert text, select) do not move it.
Apps and sites the agent may use
You decide which apps and websites the agent may touch. Put a policy.json in the server's support directory (~/Library/Application Support/computer-use on macOS, %LOCALAPPDATA%\munim-computer-use on Windows, ~/.local/share/munim-computer-use on Linux; munim-computer-use identity prints it as supportDir), or point COMPUTER_USE_POLICY at a file anywhere. An app that embeds the server under its own profile reads the file from its own support directory:
allow, ask or block per app or site. ask shows the user a prompt the first time the agent reaches for it (a system dialog for apps, a small Chrome window for sites), and an approval lasts until that agent task ends. An unanswered prompt counts as no after two minutes.
Apps match their name or bundle id, ignoring case. The rule applies when the agent reads, captures or activates the app. Actions then target elements from a snapshot that was allowed.
Sites match a host and all its subdomains, and the most specific pattern wins. They are checked on every browser action against the page the tab is showing at that moment, so following a link into a blocked site does not get around it.
* sets the default, which is otherwise allow, so {"apps": {"*": "block", "Notes": "allow"}} is an allow-list.
"foreground" is one rule for the whole screen, not per app. Set it to "block" to keep the agent in the background, so it never takes the screen from you. It covers:
activate_app;
browser_select_tab with bring_to_front;
on Windows, typing into a window that is not in front, and any action that would fall back to your real pointer;
on Linux, every pointer action (they all move the real pointer).
Those calls fail with foreground_required before anything moves, and work that runs in the background goes ahead. "ask" prompts once per agent task. A tool argument cannot override the rule. Remote control from a viewer is exempt, because there you are the one driving.
Changes apply at once, with no restart. A file that is not valid blocks everything rather than being ignored, so a typo cannot quietly turn a block into an allow.
This is a guard rail for an agent that follows its instructions, not a sandbox. A coordinate click lands wherever it points, and a whole-display screenshot shows every window.
Every action tool also takes return_state, which returns current state after the action in the same call. run_steps runs a list of them in one call and stops at the first that fails.
Editing a page in a background tab
The agent's tabs stay in the background: the browser tools never select them, raise Chrome's window, or send OS keystrokes. While the agent holds a tab, its DevTools session emulates focus, so Chrome keeps rendering the tab and real input reaches it:
browser_click presses with trusted mouse events, which frameworks that ignore synthetic clicks (Angular Material buttons, for example) respond to. Page events remain only as a fallback, and the reply says which route ran.
browser_press_key takes any key plus modifiers (cmd, ctrl, alt, shift): Delete, arrows, Home/End, PageUp/PageDown, F-keys, characters and shortcuts. Editing shortcuts such as select all (key: "a", modifiers: ["cmd"]) mean the same with cmd or ctrl on every platform. Browser shortcuts like close-tab never reach Chrome itself.
browser_fill replaces a field's whole value by its snapshot index: input, textarea or contenteditable, open shadow roots included. It selects everything, inserts the new text through Chrome's text input (trusted beforeinput/input events), and says whether the field now holds exactly that text.
browser_upload_file attaches local files to a file input by its snapshot index, or by the index of a label for one. Chrome sets the input's files (DOM.setFileInputFiles) and the page gets its usual input/change events. No file picker opens and no OS input is sent. Snapshots list file inputs as input[file], including ones a page hides behind a styled button, and show how many files each holds. Paths are on the computer running Chrome and the server, and the server checks each is a readable regular file before sending it. Several files need an input with multiple. The tool only stages the files: it never submits the form, and the reply says the site has not received them yet.
browser_snapshot also lists table and grid rows (row "Ana | Admin | Active"), cells and other elements that are only clickable by handler or pointer cursor, so list rows can be found and clicked. Checkboxes, radios and switches are named by their label, never their value ("on"), and show their state: input[checkbox] "Admin (all permissions)" (unchecked).
A click never lands on the wrong thing. An element out of view is scrolled into view, and the press goes only to a point where the element itself (or its own label) is on top. When a banner, dialog or another control covers it, nothing is pressed and the call fails naming what is in front. A checkbox click reports what the box is now.
The desktop press_key and type_text go to whichever Chrome window is in front, which may be the user's, so never use them on an agent tab.
Typing alongside the user
The person at the computer can keep typing in their own windows while the agent types elsewhere. Browser tools never touch the OS keyboard. On the desktop, delivering keys or a click to a background app lends that app keyboard focus for the length of the action. On macOS, keys the person presses during that moment are held in a short-lived session event tap and replayed, in order, to their own window once it has focus again (set MUNIM_COMPUTER_USE_HOLD_USER_KEYS=0 to turn this off). Windows holds and replays them the same way. A desktop click by element_id that has to use a position checks it with the app's own hit test first, and fails instead of clicking whatever else is there.
Browser readiness and safe re-observation
browser_click, browser_type, browser_fill, browser_upload_file, browser_press_key, and browser_navigate accept
wait_for_selector together with return_state: true. browser_snapshot accepts
the same wait without performing an action:
The extension polls for any matching visible element in the page, open shadow
roots included (each root is matched on its own, so one selector cannot span a
host and its shadow tree), independently of resource loading, including
same-document rendering. It shares
the snapshot's visibility rules: at least 2×2 CSS pixels; no hidden, aria-hidden,
inert or fully transparent ancestor. Self-transparent native form controls remain
available for styled upload/checkbox hit targets; transparent containers do not.
This is not a guarantee of clickability or
lack of occlusion. Frames are searched too (a match there names the frame);
closed shadow roots are not, and a timeout says how many frames could not be searched. Choose a selector specific to the new expected UI;
an element already present can satisfy it immediately. No arbitrary page script
is accepted, and a selector may not test the value attribute: React mirrors a
controlled input's value there, password fields included, so a wait could
otherwise confirm a password one guessed character at a time.
wait_timeout_ms: defaults to 8000, accepts integers 0–10000; 0 checks
once. Requires wait_for_selector. The budget covers polling, not browser/API
latency or user site-approval prompts. Site rules are rechecked before every
observation and before the final snapshot.
readiness: met (visible_selector, … ms): the specified condition was observed,
not a guarantee that all network requests, animations, or application work ended.
readiness: timeout: the condition was not observed within the budget. The
action's success remains separate, and the current snapshot is still returned.
readiness: error: the condition could not be evaluated (for example invalid
CSS); the action can still have succeeded. Correct the wait and re-observe.
Without a selector, return_state retains the short pause and bounded
navigation-load wait, but reports not_requested (document_load, … ms) rather
than claiming SPA readiness. A navigation-load budget expiry reports timeout.
A plain browser_snapshot without a wait remains an immediate observation.
If a returned snapshot looks unchanged, do not repeat a non-idempotent click
just because expected UI is absent. Wait/re-observe first:
Send this to browser_snapshot, inspect its readiness and fresh indices, then
choose the next action. A timeout or tool error does not roll back a click,
submission, typing, or navigation. Use current state to decide whether a retry is
safe. Invalid wait types/bounds or an action wait without return_state: true are
rejected before acting. null for an optional argument means the same as leaving
it out. These options require an updated extension as well as the native server;
when an older extension ignores them, the result says so and asks for a reload.
Regression checks: node --test chrome-extension/background.test.mjs covers the
wait contract and site-policy guards. The real-browser CI harness
scripts/e2e-browser.mjs tests a dialog delayed by 1100 ms without navigation,
one click only, plus timeout and read-only recovery. The dependency-free CDP
regressions cover both snapshot scoping/pagination and SPA readiness. Run against
an isolated Chrome/Chromium profile; no Playwright module is required:
bash
node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome
# Extract the pre-fix source; each suite must fail independently:
git show deb4c7a:chrome-extension/background.js > original-background.js
node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome --source original-background.js --suite snapshot
node scripts/snapshot-dom.test.mjs --chrome /path/to/chrome --source original-background.js --suite readiness
browser_snapshot automatically scopes to the topmost visible dialog when one is
open, so covered background controls cannot consume the dialog's budget. A
dialog's portalled popups stay in scope: elements its controls point at through
aria-controls/aria-owns, open popovers, and controls painted on top of the
dialog, such as a select's options rendered at the end of <body>. Hidden
and inert subtrees are excluded; reachable offscreen controls remain available.
Native modeless dialogs and explicit aria-modal="false" do not scope the page.
Native modal dialogs and explicit aria-modal="true" are definitive candidates.
Implicit ARIA dialogs are heuristically scoped only when they cover the viewport
centre and are topmost there, rather than merely at their own centre. With no
hit-test stack, centre coverage is the fallback. Corner chat widgets and bottom
cookie banners therefore retain the page's indexed controls.
It returns up to limit controls (default and maximum 250), reports scope,
offset, limit, and total, and explicitly says when it is truncated and which
offset to request next. Continue on the same tab_id and session_id. Indices
are global within the selected scope and stay actionable across pages of an
unchanged UI; after a UI change or navigation, take a fresh snapshot from
offset=0. An action's return_state also starts at offset=0 and uses the same
scope and truncation reporting.
Names, argument shapes and descriptions are identical on every platform, and CI enforces it (node scripts/check-tool-parity.mjs); a model that learned them on a Mac needs nothing new on Windows. To change a tool, edit both literals at once with scripts/tool-defs.mjs rather than by hand.
Repository layout
Directory
What
Build
macos/
Swift server on the Accessibility API and ScreenCaptureKit (macOS 14+)
swift build -c release → .build/release/munim-computer-use
windows-linux/
Rust server: UI Automation on Windows, AT-SPI + X11 on Linux
These are needed by the screen-capture dependencies even on X11. In particular,
a missing libpipewire-0.3.pc during the build means pipewire-devel is missing.
sh
# macOScd macos && swift build -c release
# Windows / Linuxcd windows-linux && cargo build --release
Linux notes: element actions work everywhere; coordinate clicks need an X11 or XWayland client, since native Wayland apps do not expose absolute geometry.
Tests
sh
cd windows-linux && cargo test# Rust server
node chrome-extension/background.test.mjs # extension, against a fake Chrome
node scripts/check-tool-parity.mjs # Swift and Rust tool lists match# Snapshot and SPA readiness regressions (dependency-free CDP, isolated profile):
node scripts/snapshot-dom.test.mjs --chrome <Chrome for Testing binary>
# The browser tools end to end, in a throwaway Chrome for Testing profile that# never touches your own Chrome (npx playwright install chromium to get one):
node scripts/e2e-browser.mjs --server <munim-computer-use binary> --chrome <Chrome for Testing binary>
Register the native messaging host: munim-computer-use install-native-host (for example npx -y munim-computer-use install-native-host), or from a checkout sh chrome-extension/install.sh (macOS/Linux) / powershell -File chrome-extension/install.ps1 (Windows), which find the build and run the same command. Point COMPUTER_USE_PATH at the binary if it is not in the default build location.
If the extension still gets Specified native messaging host not found., or the browser tools say the extension is not connected, run munim-computer-use check-native-host in an ordinary terminal. It looks the host up the way Chrome does (on Windows: registry views and policies; everywhere: the manifest's encoding, name, type and allowed origins), starts it with Chrome's command line, and names the step that would fail. Two Windows causes look fine from the inside. An installer run by an MSIX-packaged app (some agent desktop apps are packaged) writes to a private, virtualised copy of HKCU and %LOCALAPPDATA% that Chrome never reads. One run by a sandbox with its own Windows account writes to that account's HKCU. The check detects both; run install-native-host again from your own terminal. install-native-host reports the same findings under problems.
The standalone server's host is com.munimtech.computer_use.desktop; MT Code, which bundles this server, registers com.munim.mtcode.desktop. The extension connects to every host it knows at once and answers each on its own connection, so MT Code and a standalone server (npx, Claude Code, Cursor, a checkout) can both drive Chrome at the same time, each in its own tab groups.
Sandboxed agents (Windows)
Some agents run their commands in a sandbox: a restricted or low-integrity token, a separate Windows account, or another desktop. A server started there can still list windows and their titles, but other apps' UI Automation refuses it and captures come back black, without any error. When that happens the server says so: get_app_state explains an empty tree, screenshot and zoom flag a capture that is one flat colour, and activate_app checks that the window really came forward. Each names the cause it found (for example "runs at low integrity" or "runs as another account"). Retrying from the same context will not help; configure the agent to start the MCP server in your own, unsandboxed context.
Environment flags
Variable
Effect
COMPUTER_USE_BROWSER=0
Hide the browser_* tools
COMPUTER_USE_AGENT_CURSOR=0
Do not draw the agent pointer
COMPUTER_USE_AGENT_CURSOR_TASK_FADE_SECS
How long the pointer stays after the last tool call (default 8)
COMPUTER_USE_ALLOW_SECURE_FIELD_INPUT=1
Allow typing into password fields (refused by default)
COMPUTER_USE_REMOTE_CONTROL=1
Remote-desktop mode: input takes over the real pointer
COMPUTER_USE_PARK_WINDOWS=0
macOS: do not move minimized or hidden windows to an invisible display to act on them
COMPUTER_USE_POLICY=<file>
Where to read the app and site policy (default policy.json in the support directory)
Remote control
Normally this server never touches the pointer: coordinate clicks are routed to
a specific window, keystrokes are posted to a specific process, and an action
that cannot be targeted is refused rather than taking over the machine. That is
what lets an agent work while the user keeps using their computer.
COMPUTER_USE_REMOTE_CONTROL=1 inverts that contract for one process, for the
case where a person is watching this machine's screen from another one and is
steering it themselves. Then click, right_click, drag, hover and
scroll move the real cursor and type_text and press_key go to whatever is
focused, the way Chrome Remote Desktop or Screen Sharing behave. scroll also
accepts x/y so the wheel acts over the point the viewer scrolled at, and the
agent-cursor overlay stays hidden — there is only one pointer now.
A host that runs both an agent and a viewer runs them as two processes, so
turning this on for the viewer never takes the pointer away from the user on
the agent's behalf.
Embedding in an app
An app can ship this binary inside its own bundle and run it under its own identity, so it never shares a browser bridge, agent-cursor app or native-messaging host with a standalone install on the same machine. MT Code does exactly this. Pass a profile, either as a JSON object or a path to a JSON file, with --profile <json|file> (any position) or COMPUTER_USE_PROFILE:
json
{"name":"example-desktop","envPrefix":"EXAMPLE_DESKTOP_","agentCursorName":"ExampleAgentCursor","agentCursorBundleId":"com.example.agent-cursor","nativeHostNames":["com.example.desktop"],"extensionIds":["abcdefghijklmnopabcdefghijklmnop"],"nativeHostDescription":"Example desktop control bridge"}
Key
Default
Meaning
name
none (standalone paths)
Moves the support dir, bridge socket and Windows pipe under this name
What install-native-host registers, and for which extension. The first name is this identity's own; later ones are aliases, written only if no other installed app owns them
Each of supportDir, bridgeSocket, envPrefix, agentCursorName, agentCursorBundleId and historyDir can also be overridden by COMPUTER_USE_<SNAKE_CASE> (for example COMPUTER_USE_SUPPORT_DIR), which wins over the profile. munim-computer-use identity prints the resolved values.
Browser bridge. Run munim-computer-use install-native-host with the same profile. It writes a wrapper that relays Chrome into this identity's bridge (replaying the profile), and a host manifest for each name in every Chrome/Chromium profile directory (the registry on Windows). It rewrites nothing that is already current, so an app can call it on every launch.
Extension. Use the stock extension (it connects to com.munim.mtcode.desktop and com.munimtech.computer_use.desktop), or build a variant with its own host names, tab-group title and key: node scripts/build-extension.mjs --out <dir> --host com.example.desktop --group-title "Example" --key <base64>. It prints the variant's extension id for extensionIds.
macOS permissions. The MCP server is a bare executable, so Accessibility and Screen Recording are granted to the app that spawns it. Only the agent-cursor overlay has a bundle of its own; ship <agentCursorName>.app (a copy of the binary plus an LSUIElement Info.plist) beside the binary, or it is materialised under supportDir on first use.
Prompting your agent
Look → act → verify. get_app_state for ids, act by id with return_state: true to see the result in the same call, and screenshot when the tree cannot show it. Read pages with browser_read, and let the user type passwords through browser_request_credentials. Use zoom for small text, hover for menus that appear on mouse-over, wait after loads, keyboard shortcuts for stubborn widgets. The system-prompt text MT Code gives its agents lives in CodexDeveloperInstructions.ts and is a good starting point.
Contributing
The server is developed here. MT Code ships it by pinning a release of this repository (native/munim-computer-use.json), so changes reach MT Code with the next release. Issues and pull requests are welcome.
Credits and license
Designed and built by Munim Technologies (Munim, Inc.) for MT Code. On-screen pruning, dialog scoping, notification-based settling, OCR fallback, menu shortcuts and window parking follow arc-cua (MIT). Copyright 2026 Munim, Inc. Licensed under the Apache License 2.0; see LICENSE.