Explain why two scientific papers disagree, every claim grounded in a verbatim source quote.
io.github.nickjlamb/studydiff MCP Server
StudyDiff is an LLM-centered research tool that explains why two scientific papers reach different conclusions, with each claim grounded in a verbatim source quote. It is positioned for evidence-based literature review and scientific reasoning workflows across life-sciences and clinical research.
🛠️ Key Features
Compares two scientific studies and explains disagreement
Verifies claims against source text using verbatim quotes
Focus areas include literature review and evidence-based medicine
🚀 Use Cases
Debug or reconcile conflicting conclusions between papers
Support clinical-research and research-tools workflows
Assist life-sciences and bioinformatics literature analysis
Ground responses to reduce hallucination risk in scientific reasoning
⚡ Developer Benefits
Topics indicate alignment with grounding, hallucination handling, and PubMed-style research tooling
Repository metadata includes Node 20+ and an MIT license
⚠️ Limitations
No additional tool capabilities beyond the paper-disagreement explanation are described in the provided data
Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.
It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.
Why it's different
Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:
It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath.
It refuses to guess. Any field the source doesn't state is shown as not reported, never inferred.
It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded before it can be used as a reason.
It knows what it can't do, because it was measured — see below.
Does it work? A measured answer
StudyDiff used to rank the divergent design dimensions and present the top one as the
likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.
15 documented contradictions where the literature has since established why the papers
disagreed — each label carrying its own citation, the set built blind before any accuracy
number existed. Scored against StudyDiff's top-ranked driver:
code
Top-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]
Baseline "always say assay" 13.3% (95% CI 3.7-37.9%) [2/15]
→ discordant on 0 of 15 cases
Oracle ceiling (reachable) 66.7% (95% CI 41.7-84.8%) [10/15]
Non-assay-labelled cases 0.0% (95% CI 0.0-22.8%) [0/13]
The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay
outranks everything. Two papers almost always use somewhat different methods, so assay
almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly
the two assay-labelled cases. It is not merely as good as guessing a constant; it is
behaviourally identical to it on every case in the set.
Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive
rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an
available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at
all. The ranker was handed the right answer five more times and took none of them.
What changed as a result: the app no longer nominates a primary driver. It presents the
divergent dimensions as an unranked list, because that list is informative (it contains the
established cause 10 times in 15) while the ordering is not. Choosing among them needs
domain knowledge the tool doesn't have.
What still holds: which dimensions differ, which are identical (ruled out), and the
verbatim sentence behind every value. None of that depends on the ranking.
Confirmed blind, on a second set
Those 15 cases had by then been read across two phases — failures analysed, spans re-audited —
so every post-fix figure from them is development-set accuracy, not a blind measurement. So we
built a second set and measured it once.
eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written
and committed before any case was selected (eval/HELDOUT-PROTOCOL.md),
by a curator kept blind to the dev set's per-case failures, in deliberately different fields:
microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease.
No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.
code
Top-1 accuracy (strict) 13.3% (95% CI 3.7-37.9%) [2/15]
Baseline "always say assay" 20.0% (95% CI 7.0-45.2%) [3/15]
→ discordant on 1 of 15 cases
Oracle ceiling (reachable) 73.3% (95% CI 48.0-89.1%) [11/15]
Non-assay-labelled cases 0.0% (95% CI 0.0-24.3%) [0/12]
On unseen data the prior scores below the constant guess — by exactly one case. The intervals
overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is
that it remains indistinguishable from guessing assay every time, not that it is worse.
The line that does not move is the last one. Across both sets, 25 cases where the established
cause was something other than assay, the prior identified none of them. And the ceiling here
is higher than on the dev set — 73.3% against 66.7% — so extraction put the right answer in
front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion
reproduced on data the development loop never saw, which is the only way it could have been
strengthened.
The number is reported as-is, and is never pooled with the dev-set number: summing them into an
"n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis
of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean
pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored
after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and
corrected before any scoring, and the reasoning behind each label are recorded in the file's own
provenance block rather than summarised away.
bash
npm run eval:heldout # the blind number, offline and free
npm run eval:selftest:heldout # set integrity, incl. zero overlap with the dev set
npm run eval# offline, free, no API key — regenerates the numbers above
npm run eval:selftest # validates the harness maths and set integrity
eval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output,
they're evidence: the published numbers are reproducible from artefacts in the repo rather than
taken on faith.
Quick start
Under 60 seconds, no API key, no network:
bash
git clone https://github.com/nickjlamb/studydiff && cd studydiff
npm install
npm run demo # explains a real, famous contradiction
npm run demo -- resveratrol-sirt1 # a second worked example
npm run demo -- treg-stability # a third: Treg lineage stability
Run the web app:
bash
cp .env.example .env# add ANTHROPIC_API_KEY for live comparisons
npm run serve # http://localhost:4173
The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.
How it works
flowchart LR
IN["Two papers<br/>PMID · DOI · PDF · text"] --> R["Retrieve<br/>PubMed / PMC / PDF"]
R --> E["Extract<br/>Claude → structured study cards"]
E --> V["Verify<br/>deterministic grounding"]
V -- "ungrounded → not reported" --> E
V --> C["Compare<br/>divergent vs. shared design"]
C --> X["Present<br/>divergent dimensions + ruled out"]
Retrieve — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (fulltext / abstract / pasted); uploaded PDFs are text-extracted server-side.
Extract — Claude turns each paper into a fixed study card (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to not reported.
Verify — grounding runs first: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to not reported. StudyDiff can't cite a fact it hasn't verified.
Compare — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's polarity matters where a method list's content does.
Present — the divergent dimensions unranked, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly ruled out. StudyDiff does not nominate one as the cause: it was measured and it doesn't work.
Two things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.
Why Claude?
StudyDiff's reliability comes from how it uses Claude, not just that it does:
Structured extraction via tool-use. Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a verbatim supporting quote — no free-text parsing, no "mostly-JSON" failures.
Reasoning that maps claims to evidence. Claude reads the methods/abstract prose and identifies both the design value and the exact sentence that supports it — the hard part of turning unstructured papers into comparable, auditable cards.
Claude proposes, grounding disposes. StudyDiff never uses an LLM as the judge. A deterministic check verifies Claude's output against the source and downgrades anything unsupported before it's shown. Pairing Claude's tool-use extraction with non-LLM verification is what lets the tool trust its own output and sidesteps the usual LLM-as-evaluator pitfalls.
Built with Claude Code. The whole app — pipeline, UI, hardening, deploy — was built iteratively with Claude Code in a single agentic loop.
A landmark T-cell controversy — stable vs. unstable, driven by the fate-mapping method.
All three ship as offline demos (npm run demo / npm run demo -- resveratrol-sirt1 / npm run demo -- treg-stability).
Using it
Web app (npm run serve) — examples, PMID/DOI lookup, PDF upload, or paste; streams each pipeline step live, and exports a reproducible Markdown report with every value's verbatim supporting sentence.
MCP server (npm run mcp) — lets Claude, or any agent, call the contradiction engine directly. See MCP server.
StudyDiff's engine is also a Model Context Protocol server, so an agent can ask why two papers disagree as a tool call — the same grounded pipeline, no browser.
The full pipeline on two papers. Each paper is {id} (PMID or DOI) or {citation, text}. Needs ANTHROPIC_API_KEY.
compare_example(example)
Runs a cached worked example — no API key, no network. The quickest way to see the grounded output.
list_examples()
Lists the built-in worked examples.
It returns the same auditable report the web app exports: the verdict, the divergent design dimensions (unranked) and the identical ones (ruled out), every value with the verbatim sentence that supports it, the verification counts, and what evidence would resolve the disagreement. Ungrounded fields come back as not reported — never guessed — and the tool never picks a winner, never nominates a primary driver, and never reports a confidence it didn't compute.
Add it to Claude Code:
bash
claude mcp add studydiff -- npx -y studydiff-mcp
…or to Claude Desktop (claude_desktop_config.json):
ANTHROPIC_API_KEY is only needed for compare_studies. Leave it out and list_examples / compare_example still work — the cached examples run with no key and no network. (To run from a local clone instead, swap the command for node /absolute/path/to/studydiff/src/mcp.mjs.)
Then just ask:
Use studydiff to compare PMID 19633673 and 20929851 — why do they disagree?
No key handy? list_examples, then compare_example("treg-stability") runs entirely offline.
Project layout
code
src/ncbi.mjs PubMed / PMC retrieval (+ DOI resolution, source-depth tagging)
src/pdf.mjs PDF text extraction (pure JS)
src/extract.mjs Claude tool-use → structured study cards
src/grounding.mjs deterministic verification (OpenGATE)
src/compare.mjs divergence detection (divergent vs. shared design dimensions)
eval/ driver-ranking benchmark: 15 cited contradictions + scorer
src/gaps.mjs bounded "observed across these papers"
src/pipeline.mjs orchestration: retrieve → extract → verify → compare
src/report.mjs shared Markdown report (answer, drivers, quotes, verification)
src/server.mjs web server + streaming API (+ rate limiting, caching)
src/mcp.mjs MCP server: compare_studies / compare_example / list_examples
public/index.html single-file dashboard UI
fixtures/ cached real papers for the offline demos
Roadmap
Keyword search with a results picker, a source viewer that highlights each grounded quote in the original text, batch comparison, and an exportable report. Full list in ROADMAP.md.
Contributing
Contributions are welcome — see CONTRIBUTING.md for the setup and the invariants that keep the trust guarantee intact.
Provenance
StudyDiff was built for Anthropic's Built with Claude: Life Sciences hackathon (Builder track). All application code in this repository was written from scratch during the event. Grounding uses OpenGATE and PDF extraction uses unpdf, both as published dependencies. Retrieval uses public NCBI E-utilities; extraction uses the Claude API.
