io.github.bch1212/modelwatch

Official

by bch1212 · Python

Continuous behavioral drift monitoring for LLM apps — catches silent provider model updates.

Monitor LLM app behavior for silent model updates and performance drift.

Topics

ai-agentsanthropicclaudedrift-detectionllmllm-monitoringmcpmcp-servermodel-context-protocolobservabilitytypescript

Install

Configuration

Environment variables

MODELWATCH_API_KEYrequiredsecret

API key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).

MODELWATCH_API_BASEdefault https://api.modelwatch.app

Override the API base URL for self-hosted deployments.

claude_desktop_config.json

{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp@0.1.1"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<YOUR_MODELWATCH_API_KEY>",
        "MODELWATCH_API_BASE": "https://api.modelwatch.app"
      }
    }
  }
}