Tekmeria

Citations that can be checked: server-issued handles and a deterministic audit for biomedical evidence

DOI

Overview • Quickstart • Results • How a citation works • Architecture • Evaluation • Usage • Citing

The audit view on the offline demo: each sentence with its status, each citation with its verdict. The answer fails on a forged and a malformed citation.

Overview

The project stays out of the race for coverage. BioMCP already reaches about 70 biomedical sources, and retrieval itself helps little: biomedical RAG typically improves on no retrieval by 1 to 2 points (Nourbakhsh et al.). Citations are where models fail:

• when asked to cite, the best model in Ahmed et al. gave 15.3% relevant PMIDs, most of the rest resolving to real papers in unrelated fields;
• in Chen et al., citation fabrication by a medical agent rose from 16.5% to 31.8% when reinforcement learning rewarded only correct answers.

The protocol points the same way: the 2026-07-28 MCP revision says that servers needing state across calls use "explicit, server-minted handles passed as ordinary tool arguments".

Others also put trust in the tool rather than the model. An agent runtime can sign a receipt for each tool call and check the claims that cite it (Basu); a renderer can mark a number verified only when it matches the record it names (Solatorio); and a genome-writing tool layer reachable over MCP requires every quantity to come from a validated tool (Ali et al.). Tekmeria applies the idea to citations of biomedical passages: the handle the model writes is itself authenticated, so the server that issued it can check it with its key alone, whichever model or MCP host wrote the answer.

Status. Research software, version 0.1.1. The citation contract is versioned separately, at 0.1.0; while it is 0.x, a breaking change raises its minor version. Changes are listed in CHANGELOG.md. A technical note describes the design and the evaluation: 10.5281/zenodo.23374728.

Quickstart

You need uv (it installs a suitable Python if needed). No API keys are required. One command runs the offline demo on recorded responses and opens its audit in your browser:

uvx tekmeria demo --open

To use tekmeria in a chat, add it to an MCP host such as Claude Desktop or VS Code: see Usage. The command installs the released package from PyPI. To run the development version instead, use uvx --from git+https://github.com/PedroMarkovicz/tekmeria tekmeria, or point --from at a local clone.

Results

How a citation works

pmid:30831578/abstract.7~ae2tc
\__/ \______/ \________/ \___/
scheme  id      path     check code: HMAC of everything before the "~"

flowchart TD
    C["A citation: a handle, and perhaps a quote"] --> G{"Does it match the handle grammar?"}
    G -- "no" --> M["malformed"]
    G -- "yes" --> K{"Does the check code match this installation's key?"}
    K -- "no" --> F["forged"]
    K -- "yes" --> R{"Could the source be reached?"}
    R -- "no" --> U["unavailable"]
    R -- "yes" --> P{"Do the record and the passage still exist?"}
    P -- "no" --> W["withdrawn"]
    P -- "yes" --> Q{"A quote, and not verbatim in the text as issued or as it is now?"}
    Q -- "yes" --> X["quote_mismatch"]
    Q -- "no" --> O["ok, with flags: retracted, preprint, version_changed"]

Design choices

Architecture

flowchart TB
    subgraph HOST["MCP host: Claude Desktop, VS Code, MCP Inspector"]
        MODEL["Model"]
        VIEW["Audit view<br/>an MCP App"]
    end

    subgraph SERVER["server/ : MCP layer"]
        TOOLS["7 tools, a prompt,<br/>2 resource templates"]
    end

    subgraph CONTRACT["contract/ : pure and typed"]
        CODEC["Handle codec<br/>HMAC-SHA256, 25-bit check code"]
        AUDIT["Citation parsing and the audit"]
    end

    subgraph SOURCES["sources/ : evidence service"]
        SVC["Routing, merging, time budgets,<br/>handle minting and resolution"]
    end

    subgraph UP["Upstream APIs"]
        APIS["PubMed, Europe PMC, PubTator3, Crossref,<br/>ClinicalTrials.gov, openFDA"]
    end

    INFRA["infra/ : keys, rate-limited HTTP,<br/>SQLite cache, recorded fixtures"]

    MODEL <--> TOOLS
    TOOLS --> SVC
    TOOLS --> AUDIT
    AUDIT --> CODEC
    AUDIT --> SVC
    SVC --> CODEC
    SVC --> APIS
    SVC --> INFRA
    AUDIT -- "report" --> VIEW

Layer responsibilities

The demo

tekmeria demo runs offline on recorded responses. It searches, fetches two papers, and audits an answer with six planted cases: a correct quote, a misquote, a forged check code, a malformed handle, a citation of a retracted paper and an uncited sentence. This is its audit:

verdict: FAIL  (5 citations: 1 malformed, 1 forged, 1 quote_mismatch, 2 ok)
fabricated: 1  retracted cited: 1  cited sentences: 2/6  verbatim quotes: 1/2
[1] cited: A nationwide Danish cohort found no increased risk of autism after MMR vaccination.
    ok             pmid:30831578/abstract.7~ae2tc "During 5 025 754 person-years of follow-up, 6517 children were diagnosed with autism"
[2] invalid_only: The same study concluded that MMR vaccination raises autism risk.
    quote_mismatch pmid:30831578/abstract.9~wdbv3 "MMR vaccination strongly increases the risk of autism"
                   closest: "MMR vaccination does not increase the risk for autism" (0.8491)
[3] invalid_only: A second cohort reported the same null association.
    forged         pmid:30831578/abstract.7~ae2t0
[4] invalid_only: Earlier reviews reached the same conclusion.
    malformed      pmid:30831578-abstract
[5] cited: A 1998 case series that proposed a link was later retracted.
    ok             pmid:9500320/abstract.4~z567x [retracted]
[6] uncited: Parents should discuss the vaccination schedule with their paediatrician.

In a chat

A real run

On 2026-10-07, a model in Claude Desktop answered "Does MMR vaccination cause autism?" with the cite_by_handle instructions, then audited its own answer:

verdict: WARN  (16 citations: 16 ok)
fabricated: 0  retracted cited: 0  cited sentences: 13/21  verbatim quotes: 4/4

Evaluation

tekmeria evaluate measures the audit's decisions. It plants citations whose correct verdict is known in advance, in 34 categories:

• genuine citations, with verbatim quotes and with quotes written in typographic variants the comparison must accept: spacing, decomposed accents, every dash, apostrophe and quote mark it folds, and the invisible characters it removes;
• quotes with one detail changed: a number, a decimal point, a comparison sign, a unit, a minus sign, a letter's case, an added accent, a superscript digit, a removed space, an inserted "not" or a swapped word;
• forged check codes, paths and record IDs, a forged handle written outside brackets, malformed handles, and a note written inside the citation;
• passages and records that do not exist, and a source that is down;
• retracted papers, preprints and older record versions;
• a forged handle hidden inside an unclosed quote;
• a misquote or a withdrawn record cited beside a genuine citation in the same sentence.

How the cases are built.

• They rest on 7 real records (213 passages) from the demo's recorded responses.
• The withdrawn and unavailable cases cite made-up paths, made-up records, and a record treated as unreachable.
• What each case plants is decided by a reference oracle, written from the spec and sharing no code with the audit. It is not independent: the same author wrote both.

How they are scored.

• Each case is audited as structured claims and as answer text in one of the citation styles a model might write. The bare-handle and hidden-handle cases run on answer text only, and the two categories that put quote marks inside a quote on structured claims only. The two audits of a case use the same planted citations, so they are not independent observations.
• A case passes only when every citation gets its planted verdict and exactly its planted flags.
• The cases are also written into whole answers of several sentences. A whole answer passes only when every sentence's status, every citation, every metric and the overall verdict come out as planted. Faults in claim status, metrics or the overall verdict leave every citation right, so only whole answers catch them. There are three kinds:
  • mixed: every case, four at a time, with a heading, bold lead-ins and uncited sentences;
  • clean: genuine citations only, which pass;
  • paired: the first case of each category that can be written as answer text (32 of 34) between two sentences that each carry a genuine citation, so its own effect on the overall verdict is scored. A mismatched or withdrawn citation gives warn, alone in its sentence or beside a genuine one.
• CI runs the evaluation on pushes to main and on pull requests.

Its output, for seed 0 and 25 cases per category:

Options. --seed (default 0) and --per-category (default 25) choose the cases, and --json-out writes the full report, with the confusion matrix.

How to read it.

• Small pools are drawn from more than once: see "Distinct cases".
• The drug label and the trial supply most planted citations: see the last table.
• The evaluation shows that the audit decides correctly on these planted errors. It does not show how often models make such errors: that takes runs with real models, like the one above.

Injected faults. python -m tekmeria.evaluation.faults breaks one decision of the audit at a time, as a real bug would, and reruns the evaluation. Its 38 faults are chosen by hand:

• lenient comparators: case, punctuation, accents, whitespace, NFKC, or only the first 40 characters;
• each of 19 folds of the contract removed in turn, and a no-break space not read as whitespace (deleting that fold entry itself changes nothing, since whitespace normalization already reads it as a space);
• each of the 4 invisible characters no longer removed;
• a check code that ignores the path, dropped flags, and wrong rules for claim status, coverage, validity and the overall verdict, among them the two earlier rules under which a sentence cited only by invalid citations, or an invalid citation beside a valid one, did not warn.

It catches 37 of them. The one it misses is listed with its reason: no passage of the snapshot contains an ellipsis, so the ellipsis fold cannot be planted. CI runs it with 10 cases per category, enough for every dash and quote mark to appear.

Mutation score. python -m tekmeria.evaluation.mutants is less selective. It makes one small change at a time in the audit's code (the citation parser, the handle codec, the quote comparison and the audit itself) and runs the evaluation against each one. A change negates a comparison or moves it across its boundary, swaps and and or, removes a not, flips a boolean, or adds one to an integer. String and regular-expression constants are not changed, so the score says little about the citation grammar or the fold table: the injected faults cover those.

• The evaluation kills 71 of 78 changes: 52 by a wrong result and 19 by a crash.
• One more change stops the snapshot from loading, so the evaluation never judges it, and it is left out of the score.
• Three of the seven survivors change nothing for any record the current source parsers can build: two in the preprint rule, whose two signals the parsers always set together, and one in the handling of bold lead-ins, whose effect a later step removes.
• The remaining four are faults it misses: checked_against, which it does not score; an empty quote (two changes); and the text of a plain [1] bracket.
• Outside its scope, so never changed: the diagnostic closest_match, the check_handles tool, the key-length check, the cursor signature, and the record-ID parsing and passage addressing that only the evidence service uses.

Some categories rest on few passages: one passage has an accented letter, so the decomposed-accent category tests a single character.

Usage

The quickstart runs the offline demo. To use tekmeria in a chat, add it to an MCP host:

{
  "mcpServers": {
    "tekmeria": {
      "command": "uvx",
      "args": ["tekmeria"]
    }
  }
}

{
  "servers": {
    "tekmeria": {
      "type": "stdio",
      "command": "uvx",
      "args": ["tekmeria"]
    }
  }
}

npx @modelcontextprotocol/inspector uvx tekmeria

Commands

Configuration

Everything is optional and read from environment variables; a .env file is not read.

Tools

There is also a prompt, cite_by_handle, and two resource templates, evidence://record/{+record_id} and evidence://passage/{+handle} (the + lets the slashes in a handle through).

Development

How it is checked

uv sync
uv run pytest            # Python tests, offline (recorded fixtures)
uv run ruff check && uv run ruff format --check && uv run pyright
uv run pytest -m live    # smoke tests against the real APIs

The audit view:

cd web
npm ci
npm test && npm run typecheck
npm run build            # rewrites src/tekmeria/ui/audit-view.html; commit it

The repository

What it cannot tell you

• Whether a passage supports its sentence. The audit checks that a citation was issued by this server, that its passage still exists, and that any quote is verbatim (in the text as issued or as it is now). Support is not checked.
• Medical advice. Tekmeria retrieves and audits evidence; it does not judge it, and nothing it returns is clinical guidance. It queries public biomedical APIs and is not built for patient data.
• Old versions of registries. ClinicalTrials.gov and openFDA serve only the current version of a record; an older cited version is flagged version_changed, but its old text can come only from tekmeria's cache.
• A handle from another installation. A handle verifies only on the installation (key) that issued it.
• PubMed book chapters. Records such as StatPearls or GeneReviews chapters are left out of searches, and fetching one reports it as not found.
• A deduplicated stream across pages. search_literature never shows the same record twice on one page, but a record can reappear on a later page.

Contributing and support

• A problem or a question: open an issue. For a wrong verdict, include the answer text you audited and the report; a handle verifies only on the installation that issued it, so say whether you used the demo key.
• A vulnerability: report it privately, through the repository's Security tab, as SECURITY.md describes.
• A change: pull requests are welcome. CONTRIBUTING.md lists the checks to run and what a change needs; a change to what the audit decides needs a case in the evaluation.
• A rights holder who wants a recorded item removed can open an issue, as NOTICE says.

Citing

@software{markovicz_tekmeria,
  author  = {Markovicz, Pedro},
  title   = {tekmeria},
  note    = {Server-issued citation handles and a deterministic audit for biomedical evidence},
  version = {0.1.1},
  year    = {2026},
  doi     = {10.5281/zenodo.23376855},
  url     = {https://github.com/PedroMarkovicz/tekmeria},
  license = {Apache-2.0}
}

@misc{markovicz_tekmeria_note,
  author    = {Markovicz, Pedro},
  title     = {Tekmeria: Server-Issued Citation Handles and a Deterministic Audit for Biomedical {LLM} Answers},
  year      = {2026},
  publisher = {Zenodo},
  note      = {Preprint, version 1},
  doi       = {10.5281/zenodo.23374729}
}

Data and licence