Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence
Agent Reliability MCP Server
This Model Context Protocol (MCP) server provides a remote endpoint for testing, benchmarking, and auditing autonomous AI agents, using methods, harnesses, and evidence. It operates over a curated knowledge graph and is designed so the server’s returned claims are tied to registered sources.
Full-text search over the knowledge graph. Matching ignores accents and apostrophes, so query in the user's own words; every hit carries its relevance score and the fields it matched. Use this whenever you have a question rather than an id, then follow up with get_entity.
List the knowledge objects carrying a tag (topics are content-backed tags). Use this to browse a known topic; use search when you have a question rather than a tag, and get_overview to see which tags exist.
Answer a question from the corpus. Returns the matched object's claims with sources and confidence — never an unsourced answer. Use this when the user asked a question in words; use search when you want to see the candidates yourself.
The instance's whole source registry, or just the sources cited by one object. Use this to check what a claim rests on, or to judge the corpus before trusting it.
Graph neighbours of an object: outgoing and incoming relations, each with its relation type. Use this after get_entity to widen an answer with adjacent objects.
Most recently verified knowledge objects (freshness signal). Use this to judge how current the corpus is, or to see what changed since you last read it.
Corpus overview: what this instance knows, counts by type, published tags, freshness. Use this first when you land here and do not yet know whether this corpus can answer your question.
A remote MCP server over a curated knowledge graph. Every claim it
returns is bound to a registered source: the tools hand back claims with
their citations and a confidence value, so an agent can show its work
instead of asserting.
Nothing to install. It is a hosted streamable-HTTP endpoint:
Eight, each with an outputSchema, each returning structuredContent.
tool
arguments
what it does
get_overview
—
Corpus overview: what this instance knows, counts by type, published tags, freshness. Start here when you land and do not yet know whether this corpus can answer your question.
search
query, limit?
Full-text search over the knowledge graph. Accent- and apostrophe-insensitive, so query in the user's own words; every hit carries its relevance score and the fields it matched.
answer
question
Answer a question from the corpus. Returns the matched object's claims with sources and confidence — never an unsourced answer.
get_entity
id
Fetch one knowledge object by id, with its claims and the sources each claim cites.
get_topic
tag
List the knowledge objects carrying a tag (topics are content-backed tags).
get_related
id
Graph neighbours of an object: outgoing and incoming relations, each with its relation type.
get_sources
object_id?
The whole source registry, or just the sources cited by one object. Use it to judge the corpus before trusting it.
get_latest
limit?
Most recently verified knowledge objects — a freshness signal.
The intended path is get_overview → search or answer → get_entity
→ get_related. get_overview exists because an agent that has just
arrived needs to know whether this corpus can help before it spends a
call guessing.
What is in the corpus
knowledge objects
38
registered sources
31
published topics
93
type
objects
entity
25
guide
9
comparison
2
faq
1
glossary
1
Subject matter: evals and benchmarks (GAIA, AgentBench, Inspect), LLM-as-judge and its failure modes, Goodhart and benchmark contamination, fault injection and chaos testing, approval gates and autonomy levels, grounding and faithfulness.
Questions it is built to answer
How do I tell a real eval from a benchmark my agent has memorised?
What does calibration mean for an LLM judge, and how is it measured?
Which failure modes does fault injection actually catch?
What an answer actually looks like
A real call against the live endpoint — answer with
"how do I tell a real eval from benchmark contamination" — returns this structuredContent, trimmed:
json
{"answered":true,"entity":{"id":"agent-reliability-glossary","name":"Agent reliability glossary","evidence_tier":"secondary","confidence":0.85,"last_verified":"2026-08-08","canonical_url":"https://agentreliability.dev/k/agent-reliability-glossary"},"claims":[{"text":"An eval is a structured, repeatable test that measures an LLM or LLM-based system against a defined dimension; frameworks package evals as registries of reusable templates.","sources":[{"title":"openai/evals — framework for evaluating LLMs and LLM systems"}]}]}
Note what travels with the answer: the evidence tier, a confidence,
the date it was last verified, and the source behind the claim — not
as prose an agent has to parse, but as fields it can act on. An agent can
decline to use a weak claim, or cite the primary source directly.
When the corpus cannot answer, answered is false. It does not
improvise, and the miss is recorded so the gap can be filled.
Machine-readable surfaces
The MCP endpoint is one of several. The same corpus is served as plain
files an agent can read directly:
Each knowledge object has a human page and a machine twin at the same id,
with a canonical URL that agrees across all of them.
Behaviour worth knowing before you integrate
POST only. Every other method answers 405 with an Allow: POST, OPTIONS header.
Rate limit: 120 requests per minute per client, counted in a shared
store, published on every response as RateLimit-Limit,
RateLimit-Remaining and RateLimit-Reset (all three exposed via CORS).
It fails open: if the store is unreachable the request is served.
Malformed input gets a spec-correct JSON-RPC error — -32700 for
unparseable bodies, -32602 for an unknown tool — never an HTML error page.
Request bodies are capped and validated before transport.
Privacy
No accounts, no cookies, no ads. Usage is measured in aggregate with
daily-rotating hashed identifiers and a 200-day retention; raw IPs are
never stored. Full policy: PRIVACY.md.
Provenance and licence
Knowledge content is CC-BY-4.0: use it, cite it. The source registry is
public precisely so a claim can be checked rather than trusted —
get_sources returns what any given claim rests on.
Claims carry an evidence tier and a last_verified date. Where the
evidence is weaker, the object says so rather than rounding up.
How it is built
Compiled and served by Citarium, a source-available framework for turning a
knowledge graph into a website, an API, an MCP server and agent-readable
files from a single source, under external evaluation.
The framework's code is licensed under the Business Source License 1.1 and
its repository is not public. What is public — and what actually matters for
trusting an answer — is this server, the corpus it serves, and the registered
source behind every claim: get_sources returns what any given claim rests
on, so it can be checked rather than trusted.
This repository is the server's public face: its manifest and its
documentation. The corpus itself lives at agentreliability.dev.
Running it yourself
The endpoint above is the server. If you want to run a copy locally — or
if a directory needs to start one in order to inspect it — the recipe is
in CONTAINER.md: fetch-corpus.mjs downloads this
instance's published corpus and @citarium/mcp serves it. Both files are
GENERATED by citarium mirror; edit the instance, not them.
The container serves a snapshot taken when its image was built. This
endpoint is the thing itself.