modelwatch-mcp
MCP server for ModelWatch — continuous behavioral
drift monitoring for LLM-powered applications.
Lets Claude Desktop / Claude Code (or any MCP-compatible client) create drift
specs, run them, and surface drift events without leaving the chat.
Install
npm install -g modelwatch-mcp
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": ["-y", "modelwatch-mcp"],
"env": {
"MODELWATCH_API_KEY": "mw_PASTE_FROM_EMAIL"
}
}
}
}
Get a free key at https://modelwatch.app/#signup (5 specs, 500 runs/mo, no
card). Then restart Claude Desktop.
The example configuration stores a plaintext key in a local config file.
Restrict that file to your user account, do not commit it, and rotate the key
if it is exposed. The key never belongs in a command-line argument.
| Tool | What it does |
|---|
list_endpoints | Show monitored LLM endpoints |
create_endpoint | Add an endpoint (provider + model) |
list_specs | Show behavioral specs |
create_spec | Add a spec (prompt + schedule + optional semantic similarity threshold) |
run_spec | Run a spec on demand; baseline if first run |
reset_baseline | Re-baseline after an intentional model swap |
get_drift_events | Recent drift events across the workspace |
get_spec_history | Run history for one spec |
get_health | Per-spec health status, last drift score and last run time |
Example session
You: Create a refusal canary against the OpenAI endpoint, asking about
chest pain. Run it now and tell me the baseline.
Claude calls list_endpoints → create_spec → run_spec and reports the
score. The same spec then runs daily forever and emails you when drift is
detected.
Environment variables
| Var | Required | Default |
|---|
MODELWATCH_API_KEY | Yes | — |
MODELWATCH_API_BASE | No | https://api.modelwatch.app |
Self-hosted? Point MODELWATCH_API_BASE at an HTTPS origin. HTTP is accepted
only for loopback development; paths, query strings, URL credentials and
redirects are rejected before a key is sent.
Source
https://github.com/bch1212/modelwatch — MIT licensed.