🧠 vision-memory-mcp

An MCP (Model Context Protocol) server and CLI tool designed to cache visual UI states using perceptual hashing, local CLIP embeddings, and state transitions. It helps AI agents remember seen screens, reducing frontier model token usage and execution latency by up to 90%. Official Documentation & Demos: visionmemorymcp.com

🛠️ Quick Start

1. Global Installation

npm install -g @putervision/vision-memory-mcp

Or install from source for development:

git clone https://github.com/putervision/vision-memory-mcp.git
cd vision-memory-mcp
npm install
npm run build
npm install -g .

2. Workspace Initialization

Run the following command in your target project root:

vision-memory-mcp init

This scaffolds .vision-memory-mcp, generates a default .env template, and adds configuration properties to your .gitignore.

3. Configure MCP Clients

Cursor IDE (.cursor/mcp.json)

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"],
      "env": {
        "LANCEDB_PATH": ".vision-memory-mcp"
      }
    }
  }
}

Claude Desktop (Linux: ~/.config/Claude/claude_desktop_config.json)

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"],
      "env": {
        "LANCEDB_PATH": "/absolute/path/to/your/project/.vision-memory-mcp"
      }
    }
  }
}

4. Configure Agent Permissions (Bypass Prompt Dialogs)

To allow AI agents to query the visual cache and manage brain images automatically without requesting permission prompts, configure the security grants:

Google Antigravity Client (~/.gemini/config/config.json)

Add the following permission entries to "userSettings" -> "globalPermissionGrants" -> "allow":

"command(vision-memory-mcp)",
"read_file(.*\\.gemini/antigravity/brain/.*)",
"write_file(.*\\.gemini/antigravity/brain/.*)"

VS Code / Cursor IDE (settings.json)

Ensure you allow:

• command(vision-memory-mcp)
• Read/write permissions for the local .vision-memory-mcp/ workspace database.

🚀 Key Features

• Zero-Token Fast Path (L1/L2): Uses Difference Hash (dHash) and Average Hash (aHash) to recognize identical or near-duplicate layouts in <5ms without sending images to LLMs.
• Element Grounding & Actionability Engine: Returns structured accessibility tree elements (grounded_elements) with bounding boxes, ARIA roles, CSS selectors, and target handles (grounded_target) for deterministic UI clicks and typing.
• Local OCR & Text-Layer Enrichment: Extracts text tokens and computes n-gram Jaccard similarity to invalidate cache hits when status text differs (e.g. "Payment Succeeded" vs "Payment Failed").
• Sensitive-Data Redaction & Privacy Scrubbing: Detects and masks PII/passwords (emails, SSNs, credit cards, OpenAI/GitHub API keys) with solid composite SVG rectangles before saving or embedding screenshots. forget_state purges sensitive states on demand.
• Semantic Retrieval (L3): Runs local CLIP ViT-B/32 model inference to find conceptually similar screens (e.g. "billing configuration form").
• Monorepo & Sub-Directory Discovery: Automatically discovers nested Git repositories, submodules, and sub-directory .vision-memory-mcp databases, aggregating visual memory queries across packages.
• State Transition Graph & Reliability Pathfinding: Tracks agent actions and transition outcomes (success/failure rates, execution duration) to calculate reliable BFS/Dijkstra navigation paths.
• First-Class CI/CD Visual Spec Engine: Baseline design contracts and visual spec regression testing via vision-memory-mcp spec verify CLI and composite GitHub Action.
• Visual Checkpoints: Save, list, and diff snapshots of memory to identify visual regressions or layout modifications.
• Interactive Visualizer: Open a local force-directed graph view of the memory in your browser.
• Dual MCP Synergy: Deeply integrates with @putervision/state-memory-mcp to cross-link UI workflow tasks with perceptual caching, providing first-class bidirectional renders_state graph edges, synergistic token metrics, and unified multi-modal trajectory exports for agent training.

📦 Architecture Blueprint

                     Incoming Screen
                            │
                            ▼
              ┌──────────────────────────────┐
              │ L1: In-Memory Cache Lookup   │ ──(Hit)──▶ Return Cached Description & Grounded Elements
              └──────────────┬───────────────┘
                             │ (Miss)
                             ▼
              ┌──────────────────────────────┐
              │ L2: Perceptual Hash Scan     │ ──(Hit)──▶ Return Cached Description & Grounded Elements
              └──────────────┬───────────────┘
                             │ (Miss)
                             ▼
              ┌──────────────────────────────┐
              │ L3: Local CLIP Vector Search │ ──(Hit)──▶ Return Semantically Close
              └──────────────┬───────────────┘
                             │ (Miss)
                             ▼
              ┌──────────────────────────────┐
              │ L4: Vision LLM Fallback      │ ──(Ingest)──▶ Save Redacted State to DB
              └──────────────┬───────────────┘

🔌 MCP Tools (20 Available)

📜 MCP Prompts (3 Available)

💻 CLI Commands

• vision-memory-mcp run: Launches the MCP stdio server.
• vision-memory-mcp init: Bootstraps environment and IDE rules.
• vision-memory-mcp doctor [--json]: Health checks LanceDB writability, sharp bindings, Node runtime, Git repos, and .gitignore protection.
• vision-memory-mcp update: Checks the npm registry and updates @putervision/vision-memory-mcp globally to the latest version.
• vision-memory-mcp audit [--json]: Performs a deep workspace audit of Git repos, submodules, database locations, and state counts.
• vision-memory-mcp inspect: Prints an ASCII table of stored states.
• vision-memory-mcp metrics: Displays ROI metrics, token savings, and cached sizes.
• vision-memory-mcp view: Opens a local force-directed graph view of the memory in your browser.
• vision-memory-mcp spec <set|verify>: Baseline visual design contract registration and live visual regression verification.
• vision-memory-mcp snapshot <save|diff|list>: Manage visual checkpoints.
• vision-memory-mcp undo: Revert the last visual mutation.
• vision-memory-mcp optimize: Compacts LanceDB storage.
• vision-memory-mcp prune: Cleans up expired states.
• vision-memory-mcp backup --out <file>: Back up LanceDB to tarball.
• vision-memory-mcp restore <file>: Restore LanceDB from tarball.

⚙️ Configuration

Set these environment variables in your .env file:

LANCEDB_PATH=.vision-memory-mcp        # Storage path for LanceDB
LANCEDB_CACHE_SIZE=100                  # Maximum hot items in LRU Cache
MAX_LANCEDB_SIZE_MB=1000                # Eviction threshold (MB)
STRICT_MODE=false                       # Refuse external L4 calls & enforce projectRoot paths
STRIP_EXIF=true                         # Strip EXIF metadata from stored screenshots
OFFLINE_MODE=false                      # Restrict CLIP loading to local files only
CLIP_MODEL_PATH=                        # Optional local path to pre-downloaded CLIP model
LIMIT_INPUT_PIXELS=16777216             # Sharp decompression bomb pixel limit
HASH_EXACT_THRESHOLD=5                  # Hamming distance <= this = exact hit
HASH_SIMILAR_THRESHOLD=10               # Hamming distance <= this = similar hit
CLIP_MODEL=Xenova/clip-vit-base-patch32 # Embedding model name
EMBEDDING_DIMENSIONS=512                # CLIP embedding output dimension
VISION_MODEL_ENABLED=false              # Enable L4 vision fallback
VISION_MODEL_ENDPOINT=http://localhost:1234/v1 # Vision model server URL
VISION_MODEL_NAME=gpt-4o               # Vision model identifier
OPENAI_API_KEY=your-api-key-here        # Required if using OpenAI endpoints for L4 fallback
LOG_LEVEL=info                          # log levels (debug, info, warn, error)
TTL_DEFAULT_MS=604800000                # Eviction TTL (default: 7 days)

🔒 Local-First Privacy & Zero Telemetry Guarantee

100% of data—including screenshots, perceptual hashes, vector embeddings, and transition graphs—remains stored locally on your machine in .vision-memory-mcp/. Zero telemetry, analytics, or external API calls are made unless you explicitly enable L4 LLM Vision fallback endpoints.

🔌 Multi-IDE & Client Integration Guide

Quick 2-Step Setup

1. Bootstrap Workspace

Run the initialization command in your repository root to create .vision-memory-mcp/, .gitignore, .env, and IDE rules:

vision-memory-mcp init --yes

2. Configure Your IDE / Client

1. Google Antigravity / Gemini CLI

Config Location: ~/.gemini/config/mcp_config.json

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"],
      "env": {
        "OPENAI_API_KEY": "sk-your-openai-key-optional"
      }
    }
  }
}

2. Cursor IDE

Config Location: .cursor/mcp.json or Cursor Settings ➔ Features ➔ MCP

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"]
    }
  }
}

3. Roo Code / Cline / Continue (VS Code)

Config Location: .vscode/mcp.json or Global MCP Settings

{
  "servers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"]
    }
  }
}

4. Windsurf IDE

Config Location: ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"]
    }
  }
}

5. Zed Editor

Config Location: ~/.config/zed/settings.json

{
  "context_servers": {
    "vision-memory-mcp": {
      "command": {
        "path": "vision-memory-mcp",
        "args": ["run"]
      }
    }
  }
}

6. Claude Desktop

Config Location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "vision-memory-mcp": {
      "command": "vision-memory-mcp",
      "args": ["run"]
    }
  }
}

🤖 Agent System Rules & Integration Prompt

To ensure your AI agent automatically leverages vision-memory-mcp before sending raw screenshots to vision LLMs, add this rule to your project's .cursorrules, .windsurfrules, AGENTS.md, or system prompt:

<!-- vision-memory-mcp:start -->

# Visual Memory Rules

This project uses vision-memory-mcp to cache visual UI states and prevent redundant LLM vision calls.

## Mandatory Workflow

1. **Before visual checks**: Call `analyze_screenshot` with base64 screenshots.
2. **On Cache Hit (`is_known: true`)**: Do NOT query external vision models; reuse the cached `description`.
3. **On Cache Miss (`is_known: false`)**: Query your vision model, then call `analyze_screenshot` with the image and new description to seed the cache.
4. **Transition Tracking**: Call `record_outcome` after click/type/navigation steps.

<!-- vision-memory-mcp:end -->

🤝 Contributing

Contributions are welcome! Please read our CONTRIBUTING.md guide for details on development setup, architecture, and submission guidelines.

📄 License & Disclaimer

© 2026 PuterVision LLC. Released under the MIT License.

Software & Data Loss Disclaimer: This software is provided "as is", without warranty of any kind, express or implied. Under no circumstances shall the authors, copyright holders, or PuterVision LLC be liable for any database corruption, data loss, filesystem modifications, or issues resulting from execution. Always back up your database files before performing destructive operations.

Sensitive Data & Privacy Notice: Screenshots are stored locally in .vision-memory-mcp/ unencrypted at the application level. Users are solely responsible for ensuring captured screens do not expose confidential credentials, API keys, passwords, PII, or regulated data (e.g., HIPAA, PCI-DSS).

Third-Party API Fees Notice: PuterVision LLC and the software authors assume no liability for third-party API costs, token fees, billing overages, or rate limits incurred via optional L4 vision fallback endpoints configured by the user.

Performance & Cost Estimates Notice: Any token savings estimates (e.g. "up to 90%"), execution latency figures, or ROI calculations stated in this documentation are benchmark estimates based on typical UI automation workloads with repeated visual states. Actual token savings, execution speedups, and financial cost drops depend on workflow layout repetition, image resolution, model provider rates, and prompt structure. PuterVision LLC does not guarantee specific monetary savings or API throughput.

Trademark Notice: All product names, trademarks, and registered trademarks (such as Cursor, Claude, OpenAI, Playwright, Zed, Windsurf, Roo Code, LanceDB) mentioned herein belong to their respective owners and are used solely for identification purposes. Use of these names does not imply affiliation or endorsement.