@yawlabs/mcp-compliance

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

Test any MCP server for spec compliance. Two test suites — 88 tests for MCP 2025-11-25 and 103 tests for MCP 2026-07-28 — covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. The tool probes the server and grades the newest spec revision it speaks, or you pin one with --spec-version. Works against HTTP endpoints (https://my-server.com/mcp) and stdio servers (npx @modelcontextprotocol/server-filesystem /tmp) alike. CLI, MCP server, and programmatic API.

Built and maintained by Yaw Labs.

Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

• Two spec revisions, one tool — 88 tests for MCP 2025-11-25 (the initialize + session era) and 103 tests for MCP 2026-07-28 (stateless, per-request _meta, server/discover, caching hints, MRTR). Both cover the same 8 categories: transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. HTTP-specific tests (CORS, TLS, header validation, rate limiting) are gated out on stdio: the 2025-11-25 suite runs 85 tests on HTTP and 55 on stdio, the 2026-07-28 suite 99 and 75.
• Auto-detected revision — one server/discover probe tells the tool which era the server speaks; dual-era servers are graded against 2026-07-28 with a warning. Pin either revision with --spec-version. See Spec version.
• Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
• Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
• CI-ready — --strict mode exits with code 1 on required test failures. Drop it into any pipeline.
• Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
• Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
• Published methodology — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.

Quick start

Remote HTTP server:

npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp

Local stdio server (the vast majority of MCP servers on npm):

# Pass the command directly, Inspector-style
npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp

# Or a local build
npx @yawlabs/mcp-compliance@latest test node ./dist/server.js

# With env vars
npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github

Install globally:

npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcp

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

CLI usage

HTTP targets

# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp

# JSON / SARIF for scripting + GitHub Code Scanning
mcp-compliance test https://my-server.com/mcp --format json
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

# Strict mode for CI — exits 1 on required-test failure
mcp-compliance test https://my-server.com/mcp --strict

# Auth (shorthand or full header)
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"

# Focus the run
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
mcp-compliance test https://my-server.com/mcp --verbose

stdio targets

Pass the command and its args as positional arguments (MCP Inspector-style). Use -- to disambiguate when the target needs flags that collide with ours.

# npm-distributed stdio servers
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-compliance test uvx mcp-server-git

# Local build
mcp-compliance test node ./dist/server.js

# With env vars (repeatable -E, or --env-file)
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
mcp-compliance test --env-file .env -- node ./server.js

# Set working directory
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js

# Target uses a flag that collides with ours — use `--` to separate
mcp-compliance test --verbose -- node ./server.js --verbose

On Windows, npx and other .cmd shims are handled automatically by spawning through the shell.

Spec version

The tool ships one test catalog per MCP specification revision it supports — 2025-11-25 (88 tests: initialize handshake, sessions, ping) and 2026-07-28 (103 tests: stateless, per-request _meta, server/discover, caching hints, MRTR). A run grades exactly one revision, chosen by --spec-version:

# auto (default): probe the server, grade the newest revision it speaks
mcp-compliance test https://my-server.com/mcp

# pin a revision — no probe, the named catalog runs regardless of what the server answers
mcp-compliance test https://my-server.com/mcp --spec-version 2026-07-28
mcp-compliance test https://my-server.com/mcp --spec-version 2025-11-25

# --list never connects: under auto (the default) it prints every catalog in its own
# section; --spec-version <date> previews just that one
mcp-compliance test --list --spec-version 2026-07-28 --transport stdio

How auto decides, following the spec's own rules for dual-era clients: the tool sends one conformant 2026-07-28 server/discover (on HTTP it doubles as the preflight connectivity check, so detection costs no extra round-trip; on stdio it is the first exchange). A DiscoverResult, or a JSON-RPC error with a modern code (-32020, -32021, -32022), selects 2026-07-28. Anything else — -32601, -32000, a 400 "not initialized", a 404, an HTML page, or no reply within the timeout — selects 2025-11-25; a 401/403 with no modern error body also lands on 2025-11-25, the note says the era could not be determined, and the report's first warning (in every output format) says the grade below is not meaningful. With no Authorization header configured, a 401, or a 403 carrying a WWW-Authenticate: Bearer challenge, reads as authentication required (note authentication required -- pass --auth; the warning says to re-run with --auth). With one configured, a 401, or a 403 whose Bearer challenge carries an error parameter, reads as the credential rejected (note credential rejected -- check --auth; the warning names the reason from that error, e.g. invalid_token is an invalid or expired token and insufficient_scope a missing scope or permission). Any other 403 (the SDK's Host validation behind a tunnel hostname, Origin validation, a gateway) is worded neutrally with or without a header: the note says forbidden -- Host/Origin validation, a gateway, or missing credentials (or ... or token permissions with a header), and the warning quotes the body's JSON-RPC error message when there is one, points at Host/Origin validation and any gateway first, and without a header suggests --auth only if the server does require a credential. The fallback is deliberately not keyed to a single error code; a legacy server that ignores unknown methods is still classified correctly, just after the timeout. The report header names the reason (auto-detected from server/discover: supportedVersions [2026-07-28], ... JSON-RPC error -32601, legacy, ... no response, legacy) and the JSON warnings carry the same note (Spec version auto-detected as <v> (server/discover -> <reason>). Pin with --spec-version to override.). An unreachable server is graded as 2025-11-25 so every test that needs the server fails (the 2026-07-28 post-hoc scans fail too when nothing was received, rather than passing over an empty recording).

Two budgets bound the probe. On stdio it is bounded by --startup-timeout, and in terminal mode a dim status line appears on stderr after ~2 s naming the wait and the --spec-version 2025-11-25 skip, so a legacy server that stays silent on unknown methods does not look like a hang. If the child exits on the probe (a legacy server whose dispatcher throws on unknown pre-initialize methods), the report warns with the exit code and stderr tail, spawns a fresh instance and grades it as 2025-11-25. If the fresh instance exits at startup as well, before answering initialize, the warning says the server exits at startup regardless of the probe and points at the command, its arguments and its environment instead of suggesting a pin. benchmark re-spawns a probe-killed child the same way, bounds its probe by its own --startup-timeout, prints the same status line, and carries the warning in its output. On HTTP the probe is the preflight and is bounded by --preflight-timeout; a preflight that times out is re-probed once within --startup-timeout before the run defaults to 2025-11-25, so a slow cold start is graded in its real era instead of as unreachable.

A server that answers both eras (the default for servers built on the official SDK 2.0) is graded as 2026-07-28, and the report carries a warning naming the other era and how to pin it — keyed on supportedVersions listing 2025-11-25 or on the informational lifecycle-dual-era probe being served an InitializeResult (SDK 2.0 servers advertise only modern versions yet still serve initialize; on stdio that probe goes to a fresh process, since a dual-era server pins its era per process). To grade its legacy side too, run again with --spec-version 2025-11-25. A server that serves initialize while rejecting the conformant server/discover is not dual-era but legacy-only: lifecycle-dual-era says legacy-only: ... and the warning tells you to re-run with --spec-version 2025-11-25 (or auto). The report's specVersion is always the resolved revision, never auto.

A pinned run runs the named catalog whatever the server answers, with two honest side notes. On HTTP the preflight is still a modern server/discover, so a pinned run whose server plainly answered in the other era gets a warning (Server answered the 2026-07-28 server/discover probe with a DiscoverResult ...; this run is pinned to 2025-11-25. Re-run with --spec-version 2026-07-28 (or auto) to grade it). The warning needs an era signal -- a DiscoverResult, a JSON-RPC body or an HTTP 4xx; a 5xx or an HTML page says nothing about the era and draws none -- and a dual-era server whose supportedVersions lists 2025-11-25, pinned to 2025-11-25, is told Server is dual-era (...); this run grades its 2025-11-25 side. instead of being sent to the other pin. And a legacy-only server pinned to 2026-07-28 fails lifecycle-discover and the six _meta / standard-header rejection tests as not evaluable: a server that rejects the conformant server/discover proves nothing by also rejecting a malformed variant, so those required tests are failed rather than credited.

Two things to know when you rely on auto in CI:

• The grade can move without a config change. The day your server upgrades to an SDK that speaks 2026-07-28, auto switches suites: different test ids, different required set, a new baseline. Pin --spec-version (or "specVersion" in the config file) if you want the suite to change only when you say so.
• diff refuses to compare reports from different revisions (ids are only comparable within one catalog) and tells you which --spec-version to pin. SARIF uploads are tracked per revision (automationDetails.id is mcp-compliance/<specVersion>/), so a switch opens a fresh set of alerts rather than closing the old ones silently.

--only / --skip values are matched against the catalog that was resolved; a value that matches nothing in it (say lifecycle-init on a 2026-07-28 run) is reported as a warning instead of silently producing an empty run, and an empty run prints No tests ran -- check --only/--skip rather than All tests passed. In the 2026-07-28 suite a filtered run still measures the server: the tools/list / resources/list / prompts/list results that a filtered-out feature test would have cached are fetched once on demand by whichever test needs them, so --only security, --only schema or --only lifecycle exercise the real checks instead of skip-passing. When such a list call fails and its -list test is filtered out, the checks that needed the list fail with the recorded reason (tools/list failed (JSON-RPC error -32603 (boom)); no tools list to validate) rather than grading A over a broken list. An --only value that matches only tests gated off the target transport (transport-post on a stdio server) is warned about too, and --list prints the same warnings under each catalog it lists.

Options

CI integration

GitHub Action (drop into any .github/workflows/*.yml):

- uses: YawLabs/mcp-compliance@v0
  with:
    target: 'node ./dist/server.js'   # or a URL like https://my-server.com/mcp
    format: github                     # ::error per required failure, ::warning per optional failure and per report warning, ::notice summary with the spec version
    strict: 'true'                     # exit non-zero if any required test fails
    min-grade: 'A'                     # also exit if grade slips
    spec-version: auto                 # or pin 2025-11-25 / 2026-07-28; the resolved value is the `spec-version` output

Manual CLI invocation:

# GitHub Actions: ::error per required failure, ::warning per optional failure and per
# report warning, and a ::notice summary naming the grade and the spec version graded
mcp-compliance test https://my-server.com/mcp --format github --strict

# Slack/Linear/PR comment: drop the body straight into a comment
mcp-compliance test https://my-server.com/mcp --format markdown > report.md

# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
mcp-compliance test https://my-server.com/mcp --format html > report.html

# Block release if grade slips below B
mcp-compliance test https://my-server.com/mcp --min-grade B

# Preview which tests will run before connecting (handy for --only/--skip authoring)
mcp-compliance test --list --transport stdio --skip security --spec-version 2026-07-28

# Diff two runs — exit 1 if anything that was passing (or skipped) is now failing;
# checks that start or stop skipping are listed as "Newly skipped" / "No longer skipped" and never fail the diff.
# A baseline from an earlier version works too: its skips are read from their wording, and both
# reports are scored from their results as this version scores them (skips left out).
# Both reports must grade the same spec revision; pin --spec-version on both runs
# so a server upgrade cannot flip the suite under the baseline.
mcp-compliance test https://my-server.com/mcp --format json --spec-version 2026-07-28 > current.json
mcp-compliance diff baseline.json current.json

# Watch mode for stdio dev loop — re-runs on file changes in cwd
mcp-compliance test --watch -- node ./dist/server.js

# Latency benchmark (--spec-version and --startup-timeout work as for test)
mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4

Docker:

docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp

Scaffold a config

mcp-compliance init

Interactive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a mcp-compliance.config.json you can commit.

Config file

Check in a mcp-compliance.config.json so CI and your dev loop can run mcp-compliance test with no arguments. Supported locations (searched in order): mcp-compliance.config.json, .mcp-compliancerc.json, .mcp-compliancerc, and the "mcp-compliance" field of package.json. Pass --config <path> to load an explicit file.

HTTP:

{
  "target": {
    "type": "http",
    "url": "https://my-server.com/mcp",
    "headers": { "Authorization": "Bearer tok123" }
  },
  "timeout": 20000,
  "specVersion": "2026-07-28",
  "strict": true
}

stdio:

{
  "target": {
    "type": "stdio",
    "command": "node",
    "args": ["./dist/server.js"],
    "env": { "LOG_LEVEL": "error" }
  },
  "skip": ["security"],
  "strict": true
}

Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line. specVersion accepts auto, 2025-11-25 or 2026-07-28 and is new in 0.19 — a config that sets it is rejected as an unknown key by older tool versions (0.18.x included), so pin @yawlabs/mcp-compliance@^0.19 wherever that config is consumed.

Local SVG badge

Write a local SVG reflecting the real grade and commit it to your repo:

mcp-compliance test https://my-server.com/mcp --output badge.svg
mcp-compliance test node ./dist/server.js --output badge.svg

Then embed it in your README:

![MCP Compliance](./badge.svg)

What the 88 tests check (2025-11-25)

The 2025-11-25 catalog grades the initialize + session era. It is what --spec-version auto selects for every server that does not answer a modern server/discover.

What the 103 tests check (2026-07-28)

The 2026-07-28 catalog grades the stateless, per-request _meta era: server/discover instead of initialize, no sessions, caching hints on every cacheable result, resultType on every result, MRTR input_required results instead of server-initiated requests, and -32602 for missing resources. An id shared with the 2025-11-25 list covers the same feature, but its criteria and required flag may differ between the catalogs (stdio-framing and error-invalid-jsonrpc are optional here, error-method-code is required), so ids are comparable only within one catalog; a check whose verdict flipped carries a new id, and diff refuses to compare reports across revisions. Post-hoc tests scan a recording of every message the server sent during the run (and fail, rather than pass vacuously, when nothing was received). The _meta and standard-header rejection tests are attributable: a rejection is credited only when the conformant server/discover was served, so a legacy-only server pinned to this catalog fails them as "not evaluable". The setup server/discover is resent once after a 429, and every rule that reads it judges the answer it settles on. The checks that credit any rejection — transport-batch-reject, transport-content-type-reject, lifecycle-jsonrpc, lifecycle-subscriptions-listen, and six error checks (error-unknown-method, error-invalid-jsonrpc, error-invalid-json, error-missing-params, error-capability-gated, error-invalid-cursor), plus the bodiless 4xx that error-parse-code and error-invalid-request-code accept — also ask whose answer it is: a gateway's 401 with a -32001 body, a 403 that reads as an auth refusal (without --auth any Bearer challenge; with --auth, one carrying an error parameter), a 429 still a 429 after one resend, or any other 403 that a conformant twin sent next to the probe (a server/discover; for error-invalid-cursor, its list method without the cursor) could not get past either fails as "not evaluable", while a 5xx without the check's own code fails as the server failing on the request. lifecycle-jsonrpc's probe is the conformant server/discover itself, so no twin is sent for it: any 403 on it, and a 5xx without a server's own refusal code, is "not evaluable". transport-batch-reject and transport-content-type-reject also fail any rejection as "not evaluable" when the setup server/discover was itself rejected. tools-call-unknown is not read this way yet: a gateway's -32001 on its tools/call still passes it. The four injection tests probe a single (tool, argument) target from the safest annotation tier that has a string argument — readOnlyHint: true, then destructiveHint: false, then unannotated tools (the spec defaults destructiveHint to true), then destructiveHint: true, with passed-over and live-probed tools named in warnings — fill the other required arguments with schema-honouring placeholders, and report how many payloads were rejected, returned without evidence of execution, or never reached the tool (all never-reached is inconclusive, with a warning, and flagged as a skip). On stdio a check whose own request kills the server (an injection payload, the 1 MB argument, unknown tool arguments, stdio-unicode's CJK/emoji probe, one of lifecycle-progress-token's calls) has it restarted for the checks after it, every time that happens, --retries included — the security checks and stdio-unicode fail as died; security-tool-rug-pull then compares two lists from the new process. stdio-unicode and lifecycle-progress-token also charge a child that answers and exits right after (a plain server/discover follows the answer), and stdio-unicode fails one that answers and then stops answering. security-rate-limiting bursts 50 tools/calls at a read-only, no-argument tool when there is one and passes a quiet burst with a warning either way.

Grading

Required tests are worth 70% of the score, optional tests 30%, counted over the tests that measured something. A test that measured nothing (a skip) is left out of the score entirely, neither a pass nor a failure; when every required or every optional test skipped, the other pool carries the whole score, and a run in which every test skipped scores 0 (F) like a run in which none ran. The terminal, markdown, HTML and GitHub reports, the SARIF invocation properties (note) and the MCP mcp_compliance_test tool then say "No test measured anything"; the JSON report shows it in its counts (summary.skipped equals summary.total). Skips still count toward the pass totals (summary.passed, the category counts); every report format counts or lists them (the terminal lists them under SKIPPED CHECKS), the terminal category bars and HTML category cards score what was measured, --verbose prints skips as SKIP, and diff lists checks that start or stop skipping. Reports written by 0.19.0 and earlier scored a skip as a pass, so their scores run higher for the same results. By how much depends on the run: on the reference servers no rounded score moved, but the drop grows with a pool's failures times its skips, enough to cost a server whose required checks all pass a whole grade, and a run where most checks measured nothing can drop several. diff scores both reports from their results the way this version does, so a baseline written by 0.19.0 or earlier still compares like with like, with a note naming the score that file recorded. See the full scoring algorithm in the methodology doc.

CI integration

# GitHub Actions example
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict

# With JSON output for parsing
- name: MCP Compliance Check
  run: |
    npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format json > compliance.json
    cat compliance.json | jq '.grade'

# With retries for flaky network conditions
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000

# SARIF output for GitHub Code Scanning
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: compliance.sarif

MCP server (for Claude Code, Cursor, etc.)

This package also exposes an MCP server with 2 tools that can be used from Claude Code, Cursor, or any MCP client.

Setup

Claude Code (one-liner):

claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance@latest mcp

Or create .mcp.json in your project root:

macOS / Linux / WSL:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "npx",
      "args": ["-y", "@yawlabs/mcp-compliance@latest", "mcp"]
    }
  }
}

Windows:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "cmd",
      "args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance@latest", "mcp"]
    }
  }
}

Tip: This file is safe to commit — it contains no secrets.

Restart your MCP client and approve the server when prompted.

Tools

• mcp_compliance_test — Run the full compliance suite against an HTTP MCP endpoint (URL). Supports auth, custom headers, timeout, retries, category/test filtering, and specVersion (auto | 2025-11-25 | 2026-07-28, default auto). Returns grade, score, the resolved specVersion, and detailed results (each test marked PASS, FAIL or SKIP; the Tests line counts the skips). stdio targets are CLI-only.
• mcp_compliance_explain — Explain what a specific test ID checks and why it matters, with a link to the spec section. Ids are per catalog: pass specVersion to read one catalog, or omit it to search both (a shared id is explained once per suite, with a note saying whether the required flag / category also differ, and an id that only exists in the other suite is pointed out).

All tools have MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so MCP clients can skip confirmation dialogs for safe operations.

Programmatic usage

import { runComplianceSuite } from '@yawlabs/mcp-compliance';

const report = await runComplianceSuite('https://my-server.com/mcp');
console.log(`Grade: ${report.grade} (${report.score}%)`);

// With options
const report2 = await runComplianceSuite('https://my-server.com/mcp', {
  headers: { 'Authorization': 'Bearer tok123' },
  timeout: 30000,
  retries: 1,
  only: ['transport', 'lifecycle'],
  specVersion: '2026-07-28', // 'auto' (default) | '2025-11-25' | '2026-07-28'
});
console.log(report2.specVersion); // always the resolved revision, never 'auto'

// Per-revision catalogs and helpers
import { getTestDefinitions, specBaseFor, SUPPORTED_SPEC_VERSIONS } from '@yawlabs/mcp-compliance';
getTestDefinitions('2026-07-28'); // 103 TestDefinitions; TEST_DEFINITIONS stays the 2025-11-25 list
specBaseFor(report2.specVersion); // base URL for that revision's spec links

// Live progress for streaming UIs (e.g. server-sent-events to a browser)
await runComplianceSuite('https://my-server.com/mcp', {
  onTestComplete: (result) => {
    // result has the full TestResult: id, name, category, required,
    // passed, skipped (true when the check measured nothing), details,
    // durationMs, specRef. Push it to your client. (onProgress carries no
    // skip flag: a skip arrives there as passed.)
    sendToClient(result);
  },
  // Pre-test status lines: the stdio era probe still waiting after ~2 s,
  // or an HTTP preflight timeout being re-probed within startupTimeout.
  // The CLI prints these dimmed on stderr in terminal mode only.
  onStatus: (message) => sendToClient({ status: message }),
});

Report schema

The JSON output of the test suite is a stable, versioned contract. Every report includes a schemaVersion field at the top level. The full JSON Schema lives at schemas/report.v1.json and is shipped with the npm package.

{
  "schemaVersion": "1",        // bumped on breaking changes to the report shape
  "specVersion": "2026-07-28", // MCP spec revision the run graded — the RESOLVED value ("2025-11-25" or "2026-07-28"), never "auto"
  "toolVersion": "0.19.0",     // mcp-compliance version that produced the report
  "url": "...",
  "timestamp": "...",
  "grade": "A",
  "score": 92.5,
  "summary": { "total": 99, "passed": 97, "failed": 2, "required": 24, "requiredPassed": 24, "skipped": 4 },
  "tests": [                   // ids are only meaningful together with specVersion
    { "id": "security-auth-malformed", "passed": true, "skipped": true,
      "details": "Skipped: needs a valid credential to compare against (pass --auth)", ... },
    ...
  ],
  // ...
}

A test that ran but measured nothing (no --auth, no tools declared, a check that does not apply on the transport, or a refusal an earlier check could not attribute) is recorded as passed: true with skipped: true; summary.skipped and each categories[*].skipped count them. Skips count in passed but are left out of score (reports written by 0.19.0 and earlier scored them as passes), so read skipped next to passed before treating a pass total as evidence. SARIF output keeps its results to failures and names the skips in the run's invocation properties (testsSkipped, skippedTests), so Code Scanning opens no alert for them.

Consumer guidance:

• Pin against schemaVersion. Reject reports with an unknown version rather than guessing at the shape.
• Check report.specVersion before interpreting tests. Since 0.19 one tool version produces reports for either revision, so a dashboard that keyed on toolVersion alone will see 2025-11-25 and 2026-07-28 reports from the same version; test ids, the required set and serverInfo.protocolVersion all belong to the revision named there. Treat an unknown specVersion the way you treat an unknown schemaVersion: reject, do not guess.
• The schema validates with any Draft 2020-12 validator (e.g. ajv). It is strict (additionalProperties: false), so validate a report against the report.v1.json shipped with the tool version that wrote it: 0.19 added optional skipped fields (on a test, in summary and in categories) to v1, and an older copy of the schema rejects them.
• Renames, removals, or type changes bump schemaVersion; the report shape is otherwise stable within a major version.
• Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps) — provided they resolve to the same specVersion. Under auto that is a property of the server, so pin --spec-version where determinism across deploys matters.

Methodology & docs

The testing methodology is published openly so the grading is auditable:

• Testing methodology — test execution model per spec revision, scoring algorithm, all 88 + 103 test rules with pass/fail criteria (CC BY 4.0)
• Machine-readable rule catalog — every rule tagged with the spec revision it applies to, for programmatic consumption (kept in lock-step with the code by src/tests/catalog-parity.test.ts)
• Why mcp-compliance — the problem, existing alternatives, what this tool does differently
• Fixing common failures — recipes for the most frequent test failures with code snippets, for both revisions
• Spec version policy — one tool version, several spec catalogs: how auto decides, how to pin, what changes when your server upgrades
• Performance deep-dive — why the suite is sequential and what parallel execution would cost
• Spec PR drafts — our proposed MCP spec clarifications for ambiguous cases we've hit

The methodology is not an authoritative conformance standard — it's one tool's choices, published so they can be inspected, adopted, or forked. The official MCP specification (2025-11-25, 2026-07-28) defines what servers must do; this document describes how @yawlabs/mcp-compliance verifies it.

Requirements

• Node.js 20+

Contributing

git clone https://github.com/YawLabs/mcp-compliance.git
cd mcp-compliance
npm install
npm run build
npm test

Development commands:

Links

• MCP Specification 2025-11-25
• MCP Specification 2026-07-28
• Testing methodology
• Yaw Labs

License

MIT — see LICENSE.