MCP Observatory is an MCP security scanner. It supports CI-native testing, attack simulation, and health scoring, and outputs results in SARIF. The repository focuses on security and supply-chain concerns relevant to MCP, including schema-drift and code-scanning workflows.
๐ ๏ธ Key Features
MCP security scanning
CI-native testing
Attack simulation
Health scoring
SARIF output
๐ Use Cases
Regression-testing for MCP integrations
Agent-security and ai-security validation in CI
Code-scanning and GitHub Code Scanning reporting
Detecting MCP schema-drift during development
โก Developer Benefits
Developer-tools oriented workflow for MCP security checks
GitHub Actions compatibility via code-scanning and CI patterns
Standardized security results via SARIF
โ ๏ธ Limitations
Server details limited to a brief security/testing description and partial README excerpt; specific configuration, commands, or deployment options are not provided here.
Free, open-source MCP security testing from ClearFrame. MCP Observatory is the CI-native security tool for teams shipping custom MCP servers. Test during development, catch schema drift, simulate attacks, and generate compliance evidence โ before agents depend on your servers.
Open Terminal or PowerShell in your project folder and paste:
bash
npx -y @kryptosai/mcp-observatory@latest
This downloads Observatory and checks your first configured MCP server. With no configured server, it uses an included example. To try only that example, run npx -y @kryptosai/mcp-observatory@latest demo --example.
2. Read the result
The terminal shows the target, grade, findings, and saved receipt path. Failed checks are findings to review, not a request to pay. If connection failed, fix the startup command or credentials and rerun. A demo grade describes the example, not your own server.
The dashboard shows your local scan history, HTML reports, and JSON downloads. No account, upload, subscription, or time-based retention limit. Stop it with Ctrl+C.
The clearframe command writes a private instruction file and prints a read-only CLI command. Review the file, then run that command or paste it into a new ClearFrame Mac chat with your project open. It does not launch ClearFrame or upload evidence automatically.
Get ClearFrame CLI or the Mac app ยท Currently Apple Silicon Mac. ClearFrame has its own account and usage terms; all Observatory features work without it.
MCP Observatory is free and MIT licensed. Product telemetry is off by default and makes no telemetry network request before explicit opt-in. Use telemetry preview to inspect fields or telemetry enable to opt in. Privacy details.
bash
npx -y @kryptosai/mcp-observatory@latest test npx -y @modelcontextprotocol/server-everything
yaml
-uses:KryptosAI/mcp-observatory/action@v1
If this repo is an MCP package, that tests npx -y <package.json name>. Otherwise set command:.
Or generate a full workflow: npx -y @kryptosai/mcp-observatory@latest setup-ci --all --command "npx -y my-mcp-server" --sarif
docker run --rm ghcr.io/kryptosai/mcp-observatory:latest demo
docker run --rm ghcr.io/kryptosai/mcp-observatory:latest test npx -y @modelcontextprotocol/server-everything
Scanning a server on the host needs network and, for local files, a volume mount (-v "$PWD:$PWD" -w "$PWD").
Default-deny connect: npx -y @kryptosai/mcp-observatory@latest protect rewrites every discovered client config (Cursor, Claude, VS Code, .mcp.json, โฆ). Undo: protect --undo. HTTP/SSE servers are left alone.
Runtime deny-default: npx -y @kryptosai/mcp-observatory@latest enforce --start-proxy writes a Seatbelt policy from the scan and starts the proxy. Local scan stays free.
Proof: Safety Index. Local history, evidence export, CI, and remediation task preparation are free.
Learn from production usage
Connect real tool calls to user intent with the usage-learning SDK and CLI: sanitized invocation evidence, opted-in โasked / expected / gotโ reports, selected conversation excerpts, one-click dissatisfaction flags, intent grouping across captured requests, and regression-case export. Run mcp-observatory usage review --dir private-usage to inspect linked evidence in a loopback-only browser screen. Server logs cannot reveal unshared conversations; observed execution and inferred breakdown signals stay separate. Collection is local and requires explicit integration.
Why MCP Observatory
MCP servers are becoming production dependencies. If agents rely on them, teams need a way to catch broken tools, unsafe schemas, schema drift, slow responses, and security footguns before those failures reach users.
Observatory gives maintainers and teams:
One-command CI setup with setup-ci --all
Profile-mapped audits with audit --profile nsa-mcp
MCP receipts that package target, evidence, verdict, action, and reproduction commands
MCP risk graphs that group servers by capability boundary, receipt state, CI posture, and recommended action
Action receipts that say allow, gate, rerun, quarantine, or escalate
GitHub PR comments for compatibility, drift, and security findings
GitHub Code Scanning SARIF for normalized MCP findings
Health score badges for public trust signals
Record/replay/verify workflows for regression testing
MCP server mode so agents can inspect other MCP servers directly
Review and remediation with free local history, CI, and optional ClearFrame CLI or Mac tasks
We scan ourselves with mcp-observatory on every release. See results โ
Self-hosted evidence
The MIT-licensed hosting source includes private receipt storage, GitHub sign-in, multi-target history, downloads, hosted HTTP checks and retention cleanup. Deploy it with your own configuration; no Observatory subscription is required. The account-free local dashboard is the fastest start.
Free tools for maintainers and teams
All scanner features, source audits, skill scans, package-name review, cross-server capability review, receipts, record/replay, local dashboards, and CI integrations are open source under MIT. No hosted account or private API is needed.
bash
# Inspect local code without executing it
npx -y @kryptosai/mcp-observatory@latest source-audit ./src
# Review a launch package name without running it
npx -y @kryptosai/mcp-observatory@latest package-check -- npx -y your-mcp-package
# Review capability combinations across saved receipts
npx -y @kryptosai/mcp-observatory@latest toxic-flow first-receipt.json second-receipt.json
Recommend servers โ scans your project for languages, frameworks, databases, and cloud providers, then cross-references the MCP registry to suggest servers you're missing.
bash
npx @kryptosai/mcp-observatory suggest
Or ask your agent "what MCP servers should I add?" when running in MCP server mode.
Security scanning โ analyzes tool schemas for dangerous patterns: shell injection surfaces, broad filesystem access, missing auth, and credential leakage in responses.
bash
npx @kryptosai/mcp-observatory test --security npx -y my-mcp-server
Record / replay / verify โ capture a live session, replay it offline in CI, and verify nothing changed. Like VCR for MCP.
bash
# Record a session
npx @kryptosai/mcp-observatory record npx -y @modelcontextprotocol/server-everything
# Replay offline (no server needed)
npx @kryptosai/mcp-observatory replay .mcp-observatory/cassettes/latest.cassette.json
# Verify the live server still matches
npx @kryptosai/mcp-observatory verify cassette.json npx -y @modelcontextprotocol/server-everything
Watch for regressions โ re-runs checks on an interval and alerts when something changes.
Successful test, run, and single-target scan checks also offer to convert the passing result into a CI adoption kit. That automatic conversion enables SARIF/Code Scanning and weekly scheduled checks by default; pass --no-ci-sarif when you only want a conservative workflow without Code Scanning upload.
Runs the same test scan on merge requests and on main, publishing the SARIF as a GitLab SAST report so findings appear in the merge request security widget. This matches what setup-ci --ci-provider gitlab-ci generates.
Path to MCP config file for multi-server matrix scan
deep
Also invoke safe tools
false
security
Run security analysis
false
fail-on-regression
Fail the action on issues
true
fail-on-baseline-drift
Fail the action when baseline verification detects drift
true
comment-on-pr
Post report as PR comment. Requires pull-requests: write.
true
set-status
Set a commit status check (green/red) on the HEAD SHA. Requires statuses: write.
true
github-token
Token for PR comments and commit statuses
${{ github.token }}
The action can comment on PRs and set commit statuses when the workflow grants write permissions. setup-ci generates read-only third-party-friendly workflows by default and lets maintainers opt into comments/statuses later. init-ci remains available as a backward-compatible alias. See action/README.md for all options.
See the evidence distribution loop for the GitHub Action template, maintainer PR body, and badge rollout playbook. A badge is a public evidence signal, not a certification or endorsement.
Generate a production/security report from local run artifacts:
For clearer internal account attribution in CI, set:
bash
MCP_OBSERVATORY_ORG=your-company.com
To deliberately associate a contact email and optional contact-channel slug with private telemetry analytics, run mcp-observatory telemetry identify --email <address> [--channel <slug>]. Setting an organization or identifying a contact does not authorize outreach.
$ npx @kryptosai/mcp-observatory lock # Snapshot all server schemas
$ npx @kryptosai/mcp-observatory lock verify # Verify no drift since last lock
Lock files are the package-lock for AI tools: commit the MCP contract, then make every tool, schema, prompt, or resource drift visible in CI. See MCP lock files.
Trend Tracking
bash
$ npx @kryptosai/mcp-observatory history# Show health trends over time
Nightly Scans
bash
$ npx @kryptosai/mcp-observatory ci-report # Generate regression report for CI
MCP Server Mode
No other testing tool is itself an MCP server. Add Observatory as a server and your AI agent can autonomously test, diagnose, and monitor your other MCP servers.
bash
claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve
Your agent gets 10 tools:
Tool
When to use it
scan
Check if all your configured MCP servers are healthy
check_server
Test a specific server before installing or after updating
score_server
Get a quick health score and grade for a server
record
Capture a baseline of a working server for future comparison
replay
Test against a recorded session โ no live server needed
verify
Confirm a server update didn't break anything
watch
Check a server and see what changed since the last check
diff_runs
Find regressions between two check results
get_last_run
Retrieve previous check results for a server
suggest_servers
Discover MCP servers that match your project stack
An AI tool that checks other AI tools. It is a tool testing tools that serve tools.
Security
The MCP server runs inside AI hosts where an LLM chooses which tools to call. To prevent prompt-injection attacks:
Command allowlist: Only npx, node, python, python3, uvx, docker, deno, bun are permitted as base executables. The CLI has no restrictions.
Path validation: File-reading tools are constrained to the runs/cassettes directories.
No arbitrary execution: Use the CLI for unrestricted commands.
CLI vs MCP: Intentional Differences
Feature
CLI
MCP Server
Why
watch
Polling loop
Single check + diff
Request/response doesn't support long-polling
Interactive menu
Arrow-key navigation
Not available
MCP has no interactive UI
Color output
--no-color flag
Always plain text
MCP returns structured content
report
Renders saved artifacts
Not available
Agents read artifacts directly
serve
Starts MCP server
N/A
Is the MCP server
run
Reads target config files
Inline params
MCP tools accept params directly
get_last_run
Not available (use ls + diff)
Available
Convenience for agents
Compatibility
Works with any MCP server that uses standard transports:
Servers needing API keys work via env in the target config. Python servers work via uvx. See the full compatibility matrix for tested servers and known issues.
Target config files
For more control (env vars, metadata, custom timeout):
Each tool has strengths. Observatory focuses on regression detection and CI-friendly workflows. mcp-recorder is great as a transparent proxy. MCPBench is the go-to for performance benchmarking. mcp-jest is ideal if you're already in a Jest workflow.
Scan before you trust. Then run npx -y @kryptosai/mcp-observatory@latest enforce --start-proxy โ Observatory writes a deny-default mcp-seatbelt policy and starts the proxy. Observatory validates; seatbelt enforces.
Works with agent-obs
Secure your servers with Observatory. Trace your agents with agent-obs โ an open-source agent execution tracer that records every tool call, computes A-F session grades, and shows you exactly where your agents spend time, burn tokens, and hit errors. Observatory tells you if a server is safe. agent-obs tells you what your agent did with it. Free, local-first, npm install -g agent-obs.
Contributors โจ
Thanks to these amazing people who have contributed:
git clone https://github.com/KryptosAI/mcp-observatory.git && cd mcp-observatory && npm install && npm test
The most common first contribution is adding an MCP server to the Safety Index (10-15 minutes). See CONTRIBUTING.md for full guidelines, code standards, and the contributor recognition ladder.
If Observatory saved you a broken deploy, consider giving it a star. It helps others find the project.