OSWright

Desktop automation for AI agents, without paying for a screenshot every step.

mcp-name: io.github.Ask-812/oswright

An MCP server that lets an LLM drive real desktop applications — the desktop equivalent of Playwright MCP. It keeps a model of the screen between actions and re-reads only the parts that changed, so the same work costs an order of magnitude fewer tokens.

Eight fields read off an invoice and typed into an expense form, verified by the application itself. Same task, same result, 7.4× less context than returning a screenshot after every action. Every number on screen is measured during the run — regenerate the whole thing with python benchmarks/record_demo.py.

Why this exists

Most GUI agents re-perceive the entire screen on every step: screenshot, OCR, hand the model an image, repeat. Measured on a live desktop, the median observation changes 0.012% of the screen's pixels. Re-reading everything does far more work than the change warrants, and charges ~2,800 image tokens whether anything happened or not.

OSWright asks the compositor what changed, rescans only that, and answers element lookups from the cheapest source that can. The claims below are measured on this machine and reproducible from benchmarks/ — including the ones that did not come out in its favour.

Key Features

• Cross-platform. Windows (Win32 API), Linux (pynput/X11), macOS (pynput/Quartz).
• Accessibility tree. Find elements deterministically by role and name via Windows UI Automation — 100% accurate, instant, no model needed.
• Fast OCR. Windows OCR (built-in, instant) with EasyOCR fallback for Linux/macOS. Results are cached automatically.
• Lightweight on Windows. No PyTorch download — Windows uses the built-in OCR engine, so a full install is a few MB rather than a few GB.
• Image matching. Locates elements by template image via OpenCV.
• Window management. List, focus, minimize, close, and screenshot specific windows.
• Screenshot diffing. Detect when the screen changes with wait_for_change.
• Clipboard access. Read and write system clipboard for data transfer.
• App launcher. Launch applications and wait for them to load.
• Auto-snapshot. Every action returns a screenshot so the agent always sees current state.
• 43 MCP tools. Screen, OCR, UIA, mouse, keyboard, windows, clipboard, and compound actions.
• Incremental perception. Rescans only the parts of the screen that changed, and can return what changed instead of a full screenshot — ~21× fewer tokens per step.
• Screen memory. Recognises screens it has read before and reuses them, verified by pixels — 89× cheaper than reading again.
• Speculative perception. Learns what actions do and confirms the expected result instead of re-reading — 19–23× cheaper, with a surprise report when the interface does something unexpected.
• Adaptive waiting. Waits for the screen to actually settle rather than sleeping a fixed 300 ms — 11.9 s saved over a 50-step task.
• Resolution cascade. Element lookups stop at the cheapest method that works; repeat lookups cost ~0.05 ms.
• DPI-correct. Coordinates are physical pixels everywhere, so clicks land correctly on scaled displays.
• Test suite. 254 automated tests; the desktop-driving ones skip themselves when no display is available.

Requirements

• Python 3.10 or newer
• VS Code, Cursor, Windsurf, Claude Desktop, or any other MCP client

Getting started

First, install the OSWright MCP server with your client.

Standard config works in most tools:

{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright"]
    }
  }
}

Note: If you don't have uvx, you can use pip install oswright and then set "command": "oswright" directly.

claude mcp add oswright uvx oswright

{
  "mcp": {
    "servers": {
      "oswright": {
        "command": "uvx",
        "args": ["oswright"]
      }
    }
  }
}

code --add-mcp '{"name":"oswright","command":"uvx","args":["oswright"]}'

{
  "mcpServers": {
    "oswright": {
      "type": "stdio",
      "command": "uvx",
      "args": ["oswright"],
      "disabled": false
    }
  }
}

pip install oswright

{
  "mcpServers": {
    "oswright": {
      "command": "oswright"
    }
  }
}

python -m oswright

Incremental perception

Most GUI agents re-perceive the entire screen on every step: full screenshot, full OCR, then hand the model a fresh image. Measured on a live desktop, the median observation changes 0.012% of pixels — so a full rescan does roughly 240× more work than the change warrants, and the screenshot it returns costs ~2,800 image tokens whether anything happened or not.

OSWright keeps a model of the screen between observations and rescans only the regions that actually moved.

observe()  ->  {"changed": true,
                "added":   [{"text": "Saved", "x": 812, "y": 447}],
                "removed": ["Unsaved changes"],
                "screen_fraction_scanned": 0.015}

Measured on this machine over a 14-step agent loop:

The busier the screen, the larger the gap: full OCR scales with how much text is on screen, whereas the incremental path scales with how much changed. The same comparison measures 6.5× on a quiet desktop and 14.3× with a dense web page open. Re-measure with benchmarks/ rather than trusting these.

Cost is a proxy, though, and a cheaper perception path that quietly degraded accuracy would be worse than none. So it is checked against task completion: scripted tasks driving the real tool surface across four applications, graded against each application's own state — UI Automation for Calculator and Explorer, the window title for Chrome and VS Code — never against OCR.

Accuracy is identical across every configuration while token cost falls 23×. Run it with python benchmarks/bench_tasks.py.

Why both pixels and accessibility

The design bets that neither perception path wins everywhere. Turning each half off measures that rather than asserting it:

Accessibility-only — the posture most Windows GUI agents take — is perfect on XAML and blind on a Win32 list view and on web content. Probed against VS Code it sees 18 elements, the entire IDE being a single node named Chrome Legacy Window, while OCR reads 94 including every filename.

Pixels-only fails Calculator's buttons, because the button a human reads as 7 is named Seven, and Windows OCR returns no digits from Calculator at all.

The cascade is the only configuration that passes everywhere.

The resolution cascade

find_element and click_element stop at the first method that can answer, so cost tracks how novel the request is rather than how large the screen is:

Looking up text the model already knows is ~5,000× cheaper than the v0.4.0 path (0.05 ms versus 244 ms). The response reports which rung answered, so you can see what a task is actually costing.

Rung 3 is worth understanding: UIA's TextRange.FindText searches the application's own text buffer and returns exact bounding rectangles. It is immune to font, DPI, antialiasing and OCR error. It sits below the pixel rungs only because scanning a window's controls for it costs a few hundred milliseconds of cross-process COM — it is the accurate rung, not the fast one.

Note on ordering. These rungs are ordered by measurement, not by theory. The common advice is to make the accessibility tree primary, but on real applications it is not always cheaper: walking Chrome's tree took 537 ms here, slower than a full-screen OCR pass, and VS Code exposed only 18 elements to it. Neither pixels nor accessibility wins everywhere, which is why this is a cascade rather than a choice.

Asking the compositor instead of looking

On Windows, the desktop compositor already knows which pixels changed and exposes them through DXGI Desktop Duplication. Asking it costs 0.14 ms and transfers no pixels, against tens of milliseconds to capture a frame and discover it was identical — so an idle observation skips the capture entirely.

When something has changed, the compositor is left holding that frame, so its pixels are read directly from the GPU rather than grabbed a second time through a different API — 1.5–2.3× faster than mss in measurements here.

The compositor is used only as a fast negative for change detection. When it reports a change, the dirty regions still come from hashing the captured frame: the two are measured over slightly different intervals, so compositor rectangles can under-report relative to the pixels actually captured, and an under-reported region is text that never gets re-read. It degrades silently to tile hashing and normal capture wherever Desktop Duplication is unavailable.

Enable delta observations for action tools with --observation-mode delta. The default remains screenshot for compatibility with existing clients.

Reproduce all of this yourself: see benchmarks/. The reasoning behind each decision, including the dead ends, is in docs/ENGINEERING_LOG.md.

Configuration

OSWright MCP server supports the following arguments. They can be provided in the JSON configuration as part of the "args" list:

An explicit command-line flag always wins over the corresponding environment variable.

Example: Multi-language OCR

{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright", "--ocr-languages", "en", "es", "fr"]
    }
  }
}

Standalone MCP server (SSE)

When running from a worker process or another machine, use SSE transport:

uvx oswright --port 8931

Then in your MCP client config:

{
  "mcpServers": {
    "oswright": {
      "url": "http://127.0.0.1:8931/sse"
    }
  }
}

Security

OSWright has no authentication. Anyone who can reach the port gets full keyboard, mouse, screen and clipboard control of the machine — it is remote desktop takeover, not a sandboxed API.

The server therefore binds to 127.0.0.1 by default and refuses to start on a non-loopback address unless you pass --allow-remote. To reach it from another machine, prefer an SSH tunnel over exposing the port:

ssh -L 8931:127.0.0.1:8931 user@desktop-host

Stdio transport (the default, used by every MCP client config above) is not network-exposed at all and is the recommended way to run OSWright.

Tools that can destroy work are annotated accordingly: close_window is marked destructive, and launch_app starts arbitrary programs. Screenshot tools refuse to overwrite an existing save_path.

Platform Notes

On Windows, EasyOCR is not installed, because the built-in Windows OCR engine is faster and needs no model download. Install it only if you need a language Windows OCR does not support:

pip install "oswright[easyocr]"

Coordinates

All coordinates returned by OCR, image matching and UI Automation are absolute physical screen pixels, ready to pass straight to mouse_click. This holds for sub-regions and for multi-monitor setups where the virtual desktop starts at a negative origin. screenshot also reports origin_x/origin_y, the absolute position of the image's top-left pixel, for when you read a coordinate off the image yourself.

Tools

Python Library

OSWright also works as a standalone Python library with a Playwright-style API:

from oswright import OSWright

with OSWright() as ow:
    screen = ow.screen()
    screen.click(text="Start")
    screen.type_text("Hello World")
    screen.press("Ctrl+S")
    screen.screenshot("desktop.png")

See the examples/ directory for more.

Architecture

oswright/
  __init__.py          # Package entry point (single source of __version__)
  core.py              # OSWright class (= Browser)
  screen.py            # Screen class (= Page)
  locator.py           # Locator + Assertions (= Locator + expect)
  capture.py           # Screen capture (mss - cross-platform, thread-safe)
  dirty.py             # Change detection - which parts of the screen moved
  screenmodel.py       # Persistent screen model, updated incrementally
  cascade.py           # Resolution cascade - cheapest method that can answer
  atlas.py             # Remembers screens across visits and sessions
  settle.py            # Knowing when the screen has finished responding
  speculate.py         # Predicting what an action does, instead of looking
  textprovider.py      # Exact text from the app itself via UIA TextPattern
  detect.py            # OCR dispatcher with caching (auto-selects best backend)
  _ocr_windows.py      # Windows OCR backend (instant, built-in)
  accessibility.py     # Windows UI Automation (deterministic element finding)
  cache.py             # Screenshot diffing, image hashing, OCR result cache
  _dpi.py              # Process DPI awareness (keeps every API in physical pixels)
  _dxgi_windows.py     # Compositor dirty rectangles via DXGI Desktop Duplication
  input.py             # Platform dispatcher for input backends
  _input_windows.py    # Windows input backend (Win32 API)
  _input_pynput.py     # Linux/macOS input backend (pynput)
  window.py            # Window management (list, focus, close)
  clipboard.py         # Clipboard read/write (cross-platform)
  mcp_server.py        # MCP server (43 tools for AI agents)
tests/
  conftest.py          # Fixtures that skip when no display/OCR is available
  test_core.py         # Unit tests (no desktop required)
  test_perception.py   # Incremental perception (stubbed, runs headless)
  test_atlas.py        # Screen memory and its failure modes (headless)
  test_speculate.py    # Prediction, settling, and their limits (headless)
  test_e2e.py          # End-to-end tests against the real desktop (marked `e2e`)

Remembering screens

Applications are deterministic — the same dialog has the same layout every time. OSWright remembers screens it has read and reuses them on the next visit, across sessions: 125 ms cold read → 1.4 ms warm recall (89×).

A remembered screen is never trusted on recognition alone. A few regions are spot-checked by pixels before the layout is reused, so a screen that has changed is rejected rather than acted on. Verification fails closed: a screen with nothing checkable is not remembered at all.

Disable with --no-atlas. Remembered screens live in ~/.oswright/atlas.json.

Predicting actions instead of observing them

Applications are deterministic — clicking Save produces the same dialog every time — so after the first observation the outcome of an action is already known. OSWright learns what actions do and confirms the expected screen rather than reading it again: 19–23× cheaper than observing (2.3 ms versus 43–50 ms).

A prediction has to be seen twice before it is trusted, is retired if it proves wrong, and is checked the same two ways a remembered screen is. A failed prediction is reported to the agent as a surprise — the interface did something it does not normally do, which is worth knowing rather than silently absorbing.

What a confirmed prediction guarantees: the layout — the same controls in the same places. Not that every character is identical. A single changed digit alters fewer pixels than a blinking caret, so no whole-screen check can separate them at any resolution. Use observe(force_full=True) when exact text matters.

Disable with --no-speculate.

Waiting only as long as needed

Action tools used to sleep a fixed 300 ms, chosen for the slowest case, so every action paid the worst case. The compositor knows when the screen stops changing, so the wait now ends when the interface actually settles:

"Settled" means no large change recently, not no change: a real desktop is never still — a caret and a clock produce a change event every ~18 ms covering about 32 pixels, while genuine UI changes cover tens of thousands.

Not done yet

• Wayland input injection, and macOS AXTextMarker as a TextPattern equivalent.
• A vision-model rung for surfaces that are neither accessible nor text-legible: games, canvases, image editors.
• Transitions keyed on more than the previous screen, for actions whose outcome depends on state that is not visible.

What is measured, and what is not

Perception cost and task success are both measured on this machine and reproducible via benchmarks/ — across four applications, cheaper perception does not cost accuracy, and the pixel/accessibility split is measured rather than argued.

Against Windows-MCP

Same tasks, four scenarios, each graded by the application itself. Neither tool grades itself, and Windows-MCP runs at its own defaults:

Read that honestly: Windows-MCP was more reliable, and oswright was 16.9x cheaper. oswright dropped one click in twenty, on a window that had just opened.

The cost difference is structural rather than a tuning win. Windows-MCP returns the screen to the agent -- Snapshot renders the accessibility tree as (x,y) button "Seven" [action: click] -- and takes coordinates back, so a description of the screen is charged to the model's context on every action. oswright takes the text and returns the outcome.

The reliability gap may be caused by the speed: oswright resolves and clicks in ~100 ms, sometimes before a freshly-focused window is ready for input, where a slower loop gives the application time it never had to ask for. That is a hypothesis, not a finding -- adding a pre-action settle made no measurable difference over ten trials, so it is recorded rather than fixed.

The Chrome, 2 steps scenario exists because every other task here is short enough that a tool can read the screen once and reuse those coordinates. There the first click moves the controls 325 px down the page, and the snapshot-once configuration had to re-read the screen -- so on tasks whose interface moves, its cheap number does not exist and its real cost is the per-action one.

Reproduce with python benchmarks/bench_head_to_head.py (setup in the file's docstring).

What this does not establish: four short tasks on one laptop. Nothing about long multi-step work, recovery, or product maturity -- Windows-MCP has OAuth, analytics, a watchdog and an installer; oswright has none of those. Its accessibility traversal also reads Chrome's page content, which oswright's own accessibility rung does not. "Substantially cheaper per action, at a small reliability cost" is the claim. "Better product" is not.

Development

pip install -e ".[dev]"

pytest tests/                # everything available on this machine
pytest tests/ -m "not e2e"   # unit tests only, no desktop needed
ruff check oswright tests    # lint
python benchmarks/bench_pipeline.py   # reproduce the performance numbers
python benchmarks/bench_tasks.py      # task success (opens Calculator repeatedly)

Design decisions, measurements and dead ends are recorded in docs/ENGINEERING_LOG.md.

License

MIT