The agent eval standard for MCP. Score every agent output for quality, safety, and cost.
The Iris MCP server implements the agent evaluation standard for Model Context Protocol (MCP). It scores agent outputs for quality, safety, and cost, enabling observability and benchmarking within MCP workflows.
🛠️ Key Features
Agent output scoring for quality, safety, and cost
MCP-aligned evaluation framework
Observability and tracing baked into the server
Topics include agent-evaluation, eval, security, and model-context-protocol
Readme excerpt highlights installation and npm distribution
🚀 Use Cases
Automatic evaluation of agent responses in MCP pipelines
Quality and safety scoring for ML agents
Cost-aware assessment in evaluation cycles
Observability-enabled MCP server deployments
⚡ Developer Benefits
Standardized MCP evaluation endpoint
Clear metrics for agent performance
Easy integration via npm package @iris-eval/mcp-server
Clear topics and documentation for maintenance
⚠️ Limitations
Source data provides a readme excerpt and basic description; exact API surface not fully enumerated here.
Know whether your AI agents are actually good enough to ship. Iris is an open-source MCP server that scores output quality, catches safety failures, and enforces cost budgets across all your agents. Any MCP-compatible agent discovers and uses it automatically — no SDK, no code changes.
The Problem
Your agents are running in production. Infrastructure monitoring sees 200 OK and moves on. It has no idea the agent just:
Leaked a social security number in its response
Hallucinated an answer with zero factual grounding
Burned $0.47 on a single query — 4.7x your budget threshold
Made 6 tool calls when 2 would have sufficed
Iris evaluates all of it.
What You Get
Trace Logging
Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly.
Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend.
Web Dashboard
Real-time dark-mode UI with trace visualization, eval results, and cost breakdowns.
Requires Node.js 20 or later. Check with node --version.
Quickstart
Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent.
That's it. Your agent discovers Iris and starts logging traces automatically.
Turn on the dashboard
Iris ships with a real-time web dashboard showing traces, eval results, cost breakdowns, and rule pass-rates. It's off by default so the MCP server stays lightweight — flip it on with a flag.
Add the mcpServers JSON config above to ~/.gemini/settings.json.
Anything else that speaks MCP
Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.
Other Install Methods
bash
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard
# Docker
docker run -p 3000:3000 -v iris-data:/data ghcr.io/iris-eval/mcp-server
Tip: Global install (npm install -g) stores traces persistently at ~/.iris/iris.db. With npx, traces persist in the same location, but startup is slower due to package resolution.
MCP Tools
Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:
log_trace — Log an agent execution with spans, tool calls, token usage, and cost
evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
get_traces — Query stored traces with filtering, pagination, and time-range support
deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
delete_rule — Remove a deployed custom rule (destructive, idempotent)
delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.
When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.
Self-hosted Iris runs on your machine with SQLite. As your team's eval needs grow, the cloud tier adds PostgreSQL, team dashboards, alerting on quality regressions, and managed infrastructure.
