open-compute-mcp

npm launcher for the open-compute MCP server — model-agnostic computer-use tools exposed over the Model Context Protocol (MCP).

EN | DE

npm downloads Tests Contributing: Guidelines Verified

📦 View on npm → • 📋 Security Policy • 🤝 Contributing Guidelines • ⚖️ Licenses • 🤖 LLM Context (llms.txt)

Quick Navigation

• 1. 🏗️ Architecture
• 2. ✨ Key Capabilities & Invariants
• 3. 🚀 Quick Start & Prerequisites (Use with an MCP Client)
• 4. 🛠️ MCP Tool Reference (16)
• 5. ⚙️ Configuration & Environment
• 6. 🔄 Safe Interaction & Signal Lifecycle
• 7. 🎯 Target Personas & Discoverability
• 8. 🔍 High-Intent Search Term Matrix
• 9. 📊 10-Dimension Comparative Matrix vs. Alternatives
• 10. 🏛️ Governance & Runtime Invariants
• 11. 🌐 ellmos-ai Ecosystem & Sibling Matrix
• 12. 🔒 Security Policy & Zero-Egress Safety
• 13. 📋 Level 1 SBOM Transparency & Invariant Matrix
• 14. 🧪 Testing & Verification Suite
• 15. 📦 Registry Manifests & Schema Parity
• 16. 🚦 Quality Gates & Pre-Release Checklist
• 17. ⚖️ Licensing & Canonical Attribution
• 18. 📜 Statutory Notice (§ 521 BGB) & 48h Security Response SLA

The MCP client is the reasoner (no API key, model-agnostic): it calls capture to see the screen, then acts with do / click_name / invoke. This is the keyless Mode-A loop of open-compute, but as native tool-calls.

1. Architecture

graph TD
    A["AI Reasoner<br/>(Claude / Antigravity / Cursor)"] -- "MCP stdio (JSON-RPC)" --> B["npx open-compute-mcp<br/>(Node.js Launcher)"]
    B -- "Spawns via uvx" --> C["open-compute Python Engine<br/>(GitHub @ main)"]
    C -- "Screenshots / WGC" --> D["Windows Display"]
    C -- "UIA / Mouse / Keys" --> E["Windows Desktop Apps"]
    C -- "Glowing Border & Cursor" --> F["Signal Overlay UI"]

    subgraph Safety Gate
        C -. "OC_SAFETY_MODE<br/>(confirm / read_only / allow_all)" .-> C
        C -. "OC_DENY<br/>(hard action blacklist)" .-> C
    end

Four-View System Architecture Topology

+====================================================================================================+
|                     OPEN-COMPUTE-MCP — FOUR-VIEW SYSTEM ARCHITECTURE TOPOLOGY                     |
+====================================================================================================+

[VIEW 1: AI CALLER RUNTIMES & MULTI-AGENT CLIENTS]
  +------------------------------------------------------------------------------------------------+
  |  +--------------------+  +--------------------+  +--------------------+  +-------------------+ |
  |  | Claude Code / Desktop |  | Antigravity (AGY)  |  | Cursor / VS Code   |  | Custom MCP Agents | |
  |  +---------+----------+  +---------+----------+  +---------+----------+  +---------+---------+ |
  |            |                       |                       |                       |           |
  |            +-----------------------+-----------+-----------+-----------------------+           |
  |                                                | Model Context Protocol (MCP)                  |
  |                                                v Stdio JSON-RPC Transport                      |
  +------------------------------------------------------------------------------------------------+
                                                   |
[VIEW 2: ZERO-EGRESS NODE.JS LAUNCHER & PROTOCOL DISPATCH]
  +------------------------------------------------------------------------------------------------+
  |  npx open-compute-mcp (Node.js Thin Bridge, CommonJS / RunAsInvoker)                           |
  |  - stdio pipe isolation (0 network sockets, zero telemetry egress)                            |
  |  - Environment sanitizer & launch parameter resolver (OPEN_COMPUTE_MCP_CMD / uvx)            |
  |  - Version parity guard (0.1.0-alpha.20 manifest sync)                                         |
  +------------------------------------------------+-----------------------------------------------+
                                                   | Stdio Process Stream
                                                   v (uvx --from git+https://github.com/ellmos-ai/...)
[VIEW 3: PYTHON OPEN-COMPUTE SOVEREIGN ENGINE]
  +------------------------------------------------------------------------------------------------+
  |  open-compute Python Engine (@main)                                                            |
  |  +---------------------------+  +---------------------------+  +-----------------------------+ |
  |  | Vision & Perception       |  | Input & Semantic Tree     |  | Signal Overlay & Voice      | |
  |  | - capture (mss / WGC GPU) |  | - click_name (UIA tree)   |  | - signal_show (glowing UI)  | |
  |  | - coords / scale factors  |  | - invoke (pattern action) |  | - signal_abort (hotkey)     | |
  |  | - one-shot observation_id |  | - do (mouse / keyboard)   |  | - talk (push-to-talk WAV)   | |
  |  +---------------------------+  +---------------------------+  +-----------------------------+ |
  |                                                |                                               |
  |  +---------------------------------------------v---------------------------------------------+ |
  |  | Operator Safety Ceiling & Execution Guard (Fail-Closed)                                   | |
  |  | - OC_SAFETY_MODE (confirm / read_only / allow_all)                                         | |
  |  | - OC_DENY (hard action deny list) | One-Shot Observation Token Enforcement                   | |
  |  +-------------------------------------------------------------------------------------------+ |
  +------------------------------------------------+-----------------------------------------------+
                                                   | Direct OS Windows Handles
                                                   v
[VIEW 4: HOST DESKTOP RUNTIME & ISOLATION BOUNDARY]
  +------------------------------------------------------------------------------------------------+
  |  Interactive Windows Desktop Session (Win32 / UIA / DirectX Graphics)                          |
  |  - Desktop Screen Buffer (multi-monitor, DPI scaling, GPU acceleration)                        |
  |  - Native Windows UI Automation Tree (exact-first element resolution)                          |
  |  - User-Mode Isolation Boundary (RunAsInvoker INV-PROC-07, zero kernel drivers)                |
  +------------------------------------------------------------------------------------------------+

This package is a thin launcher. It contains no server logic — it spawns the Python open-compute server (pulled from GitHub) and pipes MCP stdio through. Real screen capture and input require the interactive Windows desktop session.

2. Key Capabilities & Invariants

1. State-bound Perception & Window Targeting: Captures/trees return one-shot observation IDs; window enumeration returns stable window/process IDs and issued tokens. WGC remains the GPU-window fallback.
2. Fail-closed Action Execution: Coordinates consume one observation and exact window binding; UIA names resolve exact-first; text is segmented with focus checks and character-count postconditions.
3. Leased Signal Overlay & Abort Control: The glowing border/cursor signal has owner/session metadata, a bounded TTL, turn-end cleanup, and immediate human abort.
4. Multimodal Collaboration & Voice Notes: Push-to-talk voice recording (talk), screen chat messaging (chat), directory monitoring (watch_dir), and macro replay (rec_replay).

3. Quick Start & Prerequisites

Prerequisites

• Python 3.10+ and uv on the host. The default launch uses uvx to fetch open-compute (with the mcp extra) from GitHub on first run — the mcp extra tracks the GitHub repo, so this works regardless of PyPI release timing.
• Windows for real capture/input (mss + UIA). Other platforms import the tools but cannot drive a desktop.

Use with an MCP client

Via this npm launcher (npx):

{
  "mcpServers": {
    "open-compute": {
      "command": "npx",
      "args": ["-y", "open-compute-mcp"]
    }
  }
}

Directly via Python (uvx), no npm:

{
  "mcpServers": {
    "open-compute": {
      "command": "uvx",
      "args": ["--from", "open-compute[mcp,local,uia] @ git+https://github.com/ellmos-ai/open-compute.git", "open-compute-mcp"]
    }
  }
}

4. MCP Tool Reference (16)

All coordinates are normalized 0..1 relative to the virtual desktop. Tool descriptions are localized in six languages (de/en/es/ja/ru/zh) via OC_LANGUAGE.

do also accepts the hold primitives mouse_down / mouse_up / key_down / key_up for press-and-hold sequences (rubber-band selection, modifier-held clicking, game input); anything still held is released when the server stops. capture(window=...) falls back to Windows.Graphics.Capture when a plain grab of a hardware-composited window (Roblox Studio, Blender, a GPU-accelerated browser) comes back all-black — install the wgc extra for that.

5. Configuration (Environment Variables)

Capture size — why this launcher halves it by default

A vision model is billed per pixel, and every frame stays in the conversation, so a full-HD grab is charged again on each following request. The cost of a session therefore grows with the square of the number of screenshots, not linearly.

Because open-compute's coordinates are normalized 0..1, shrinking the image costs nothing in click accuracy — do works in fractions of the image either way. Only legibility drops, and at 0.5 buttons and field borders stay clearly identifiable; small body text is what gets hard to read.

The Python library itself defaults to full resolution — its callers are not necessarily paying per pixel. Only this launcher, which exists to serve agents, opts into the smaller frame and prints a one-line notice when it does.

What saves more than any scale factor: prefer tree where the accessibility model carries the content — note that in browsers it usually exposes only the browser chrome, not the page; and use capture(window=…) rather than the full desktop. Coordinate actions deliberately follow observe → one action → automatic refresh; do not batch multiple coordinate steps against one stale frame.

6. Safe Interaction & Signal Lifecycle

The v0.8 Python engine enforces observe → one action → automatic refresh. Keep the full descriptor or window_token from list_windows, then pass it as expected_window together with the latest observation_id from capture or tree. click_name/invoke require that issued window too. Reuse, changed state, focus mismatch, covered windows, and ambiguous UIA targets are rejected before input. type returns requested/sent character counts and complete/partial status without echoing the text. Signals have a hard TTL and are removed at action turn end unless keep_signal=true.

An explicit signal_show starts the engine's configured pre-action grace period. The static grace color is distinct from the mode color and the visible text counts down Start in N Sekunden once per second. At zero, both phase and color switch once to active. signal_status exposes the same phase, remaining seconds, current color, and screenreader label. Duration, grace color, and text template come from OC_SIGNAL_GRACE_SECONDS / OC_SIGNAL_CONFIG; 0 skips the countdown. The design uses no flashing, pulsing, or motion animation.

sequenceDiagram
    autonumber
    actor Reasoner as AI Reasoner (Claude / AGY)
    participant Launcher as Node.js Launcher (open-compute-mcp)
    participant Engine as Python Engine (open-compute)
    participant UI as Windows Desktop / UIA
    actor Operator as Human Operator

    Note over Reasoner,Operator: Phase 1: Visual Perception & State Inspection
    Reasoner->>Launcher: capture(window?) / tree()
    Launcher->>Engine: Forward stdio JSON-RPC
    Engine->>UI: Grab Screen (mss/WGC) or Read UIA Tree
    UI-->>Engine: Frame Image / Semantic Element Tree
    Engine-->>Launcher: Observation ID + normalized response/image
    Launcher-->>Reasoner: State-bound visual observation

    Note over Reasoner,Operator: Phase 2: Signal Overlay Activation
    Reasoner->>Launcher: signal_show(mode="control")
    Launcher->>Engine: Invoke Signal Overlay
    Engine->>UI: Render static grace color + Start in N seconds
    UI-->>Operator: Text countdown + accessible window name
    Engine->>UI: At zero, switch once to the mode color

    Note over Reasoner,Operator: Phase 3: Action Request & Safety Gate
    Reasoner->>Launcher: do(one action, window token, observation_id) / click_name(target)
    Launcher->>Engine: Process Action Payload
    alt OC_SAFETY_MODE == "confirm" (Default)
        Engine-->>Launcher: Status "needs_confirmation" (Report Only)
        Launcher-->>Reasoner: Human confirmation needed
    else OC_SAFETY_MODE == "allow_all" (Isolated VM)
        Engine->>UI: Execute Mouse/Keyboard / Hold Primitives
        UI-->>Engine: Action Completed
        Engine-->>Launcher: Post-observation + window/modal/text postconditions
        Launcher-->>Reasoner: Action completed - old observation invalid
    end

    Note over Reasoner,Operator: Phase 4: Emergency Abort or Completion
    opt Operator Triggers Emergency Abort
        Operator->>Engine: Hotkey Pressed (Abort Signal)
        Engine->>UI: Auto-release all held keys/mouse buttons
        Engine-->>Reasoner: signal_abort message returned
    end
    Engine->>UI: Remove overlay on turn end/error/abort (unless keep_signal=true)

7. Target Personas & Discoverability

open-compute-mcp is architected for four technical personas across autonomous AI operations, system engineering, accessibility assurance, and multimodal human-in-the-loop workflows:

8. High-Intent Search Term Matrix (SEO & Discoverability)

High-Intent Search Term Matrix (SEO & Discoverability)

9. 10-Dimension Comparative Matrix vs. Alternatives

open-compute-mcp delivers a model-agnostic, safety-bounded bridge between LLM reasoners and the Windows desktop environment. The following matrix illustrates how open-compute-mcp compares with alternative desktop interaction patterns across 10 operational dimensions:

10. Governance & Runtime Invariants

11. ellmos-ai Ecosystem & Sibling Matrix

This MCP server is part of the ellmos-ai ecosystem — AI infrastructure, MCP servers, and intelligent tools.

MCP Server Family

AI Infrastructure & Sibling Tooling

Open Bricks Umbrella

Our partner organization open-bricks bundles AI-native desktop applications — a modern, open-source software suite built for the age of AI. Sibling suites include DevCenter, CodeBox, MethodenAnalyser, CleanMarkdown, and PDFtoPDFocr.

12. Security Policy & Zero-Egress Safety

Computer-use is powerful. OC_SAFETY_MODE is an operator ceiling (confirm default · read_only · allow_all); a per-call mode can only tighten it, never loosen it. Because MCP stdio has no server→client confirm callback, confirm / read_only report an action without performing it. For interactive use, run in an isolated VM/session, set OC_SAFETY_MODE=allow_all, and let your client's tool-approval dialog be the human-in-the-loop. OC_DENY (comma-separated action types) is a hard deny list. Treat on-screen content as untrusted (prompt-injection risk).

Troubleshooting: do/click_name only ever return needs_confirmation and never act. That is the confirm ceiling working as designed under stdio MCP. Fix for interactive use: set "env": {"OC_SAFETY_MODE": "allow_all"} in the server registration and let the client's tool-approval dialog gate each action (do not auto-allow do/click_name/invoke there). The env change only takes effect when the server process (re)starts — an already-connected client keeps the old ceiling until it reconnects.

13. Level 1 SBOM Transparency & Invariant Matrix

open-compute-mcp maintains a transparent Level 1 Software Bill of Materials (SBOM) ensuring comprehensive supply-chain hygiene and zero-copyleft isolation. Full details are documented in THIRD_PARTY_LICENSES.md and THIRD_PARTY_LICENSES.txt.

Runtime Dependency Overview

Invariant Cross-Reference Matrix

Non-Elevation Certification (RunAsInvoker)

The launcher and spawned processes are certified to run entirely under standard user permissions (RunAsInvoker). They never request or require administrator or UAC elevation.

Zero-Copyleft Isolation Guarantee

No GPL, AGPL, LGPL, or other copyleft-licensed dependencies are included or linked. All code is distributed under permissive MIT and BSD licenses.

14. Testing & Verification Suite

The repository features a comprehensive automated test suite validating contract invariants, manifest parity, and repository hygiene:

# Run all automated tests
npm test

# Run repository hygiene and secret leakage checks
npm run test:hygiene

# Verify packaging integrity
npm pack --dry-run

All pull requests and commits are verified through GitHub Actions CI (.github/workflows/ci.yml) with automated matrix builds on Ubuntu, Windows, and macOS across Node.js 18.x, 20.x, 22.x, and 24.x.

15. Registry Manifests & Schema Parity

open-compute-mcp adheres to multi-registry standard specifications:

16. Quality Gates & Pre-Release Checklist

Every commit and pull request must satisfy rigorous defensive quality gates:

1. Clean Working Tree: No untracked scratch files, temporary logs, or conflict copies in repository root.
2. UTF-8 Cleanliness: All documentation files encoded in UTF-8 without BOM or replacement characters (U+FFFD).
3. Plan D Lock Discipline: Fail-closed verification of multi-agent lock status (lock_status.py) prior to mutations.
4. Manifest Parity: Version, description, and repository URLs strictly harmonized across package.json, server.json, and glama.json.
5. Release Decoupling (T-20260920-167562623): Pfad A/B discoverability and hygiene runs strictly preserve the version field without unauthorized bumps.
6. 100% Automated Test Pass Rate: Full test suite passing across all platforms.

17. Licensing & Canonical Attribution

Distributed under the MIT License. See LICENSE for full legal text.

Part of the ellmos-ai family under the open-bricks open-source umbrella. Canonical attribution and dependency inventory are documented in NOTICE, THIRD_PARTY_LICENSES.md, and THIRD_PARTY_LICENSES.txt.

18. Statutory Notice (§ 521 BGB) & 48h Security Response SLA

Statutory Disclaimer under German Law (§ 521 BGB)

The software is provided free of charge as an open-source courtesy gift (Gefälligkeit / Schenkung gem. § 516 BGB). In accordance with § 521 BGB, liability is strictly limited to intentional misconduct (Vorsatz) and gross negligence (grobe Fahrlässigkeit).

48h Security Response & 5-Day Triage SLA

The ellmos-ai / open-bricks team provides a binding commitment to acknowledge all security disclosures within 48 hours and deliver preliminary triage within 5 business days. Reports should be submitted to security@ellmos.ai, support@lukasgeiger.com, or security@open-bricks.org in accordance with SECURITY.md.