crawlcheck-core

The dependency-free modules of CrawlCheck, the verification layer for the agentic web, extracted from the production Worker. These are the readers behind its signed answer about what a site permits and serves to AI crawlers and agents. No build step, no dependencies, ESM, Node 20+.

The hosted service fetches a site as 15 crawler identities and grades 22 sections across three stages — reach (can the crawler get in), read (what it actually receives), quote (can an answer engine lift it). This repository is the open core: the readers and checks that need no network, no store and no key, published so the measurements can be reproduced.

What is here today: the robots.txt policy reader, the crawler identifier / log-line forgery verifier, the quotable-content reader, the image-signal reader, and the markdown-negotiation and agent-surface probes. What stays in the service: the scoring — the thresholds each reading is judged against, the weight every section carries in the grade, and the corpus calibration those numbers were set from — plus fetching as fifteen crawler identities, the record over time, monitoring, the dataset, and billing. The rule is simple: a reading of your page is a fact you are entitled to check, and the calibration is the product.

Install

npm install github:emmanuelorta/crawlcheck-core

Not on the npm registry yet, so install it from the repository. Every subpath below resolves from that install.

robots — which crawlers your robots.txt actually admits

import { crawlerPolicy } from "crawlcheck-core/robots";

const policy = crawlerPolicy(robotsTxt, /* unreadable */ false, "/wp-admin/");
policy.agents.find(a => a.ua === "GPTBot");
// { ua:"GPTBot", label:"OpenAI training corpus", role:"train",
//   allowed:true, allowed_root:true, named:true, rules_shadowed:true }
policy.shadowed;
// [{ uas:["gptbot"], missing:["/wp-admin/","/feed/"], missing_count:2 }]

For each of 92 named agents (answer engines, search indexes, training crawlers, SEO tools, social unfurlers, regional engines) it reports: which user-agent group applies (named), whether the given path and the root are allowed under that group, and whether the group shadows the * rules.

The shadowing defect

A crawler obeys only its most specific group. The moment you write

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Allow: /

GPTBot no longer sees Disallow: /wp-admin/ — the * group stops applying to it entirely. Sites that add an allowlist for AI agents routinely void every rule they thought still held. shadowedGroups() lists each named group and the * disallows it lost. This was found on our own sites first: how to block AI crawlers and the glossary entry for user-agent group.

API

What it does not do

* inside a rule path is stripped and the remainder matched as a prefix; $ end anchors are not honoured; Crawl-delay, Sitemap and unknown fields are ignored. These are the production service's readings too — the tests include byte-for-byte parity against the live /api/tool/robots endpoint on real files, so this module answers exactly what the service answers.

agents — who fetched, and whether they are who they claim

import { identifyBot, loadRanges, verifyLines } from "crawlcheck-core/agents";

identifyBot("Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"); // "GPTBot"

const ranges = await loadRanges(fetch);          // the operators' published IP feeds (cache this for a day)
const r = verifyLines(accessLogText, ranges);
r.tally;            // { verified, spoofed, unverifiable, no_bot }
r.forged_rate_pct;  // spoofed / (verified + spoofed), or null when nothing is checkable
r.rows[0];          // { line, bot, ip, verdict, why }

Three verdicts, never two. Verified: the address is inside a range the claimed operator publishes. Spoofed: it is outside every range that operator publishes. Unverifiable: the operator publishes no feed (Common Crawl, ByteDance, Meta, Amazon) or the feed could not be read — and that is reported as unverifiable, not as forgery. The forged rate divides by verified + spoofed only; counting unverifiable hits as forged would inflate it and counting them as genuine would understate it.

Why this exists: CrawlCheck's own telemetry once counted 78 of 85 "GPTBot" hits as forged — they were our own test requests. Reverse DNS is the older check the feeds replace; a user-agent is a self-assertion and proves nothing on its own.

quotable — what an answer engine can lift from the page

import { quotable } from "crawlcheck-core/quotable";

const { signals } = quotable(html, "Acme Fencing LLC");

signals.lead_defines;    // does the first sentence say what this IS
signals.answer_units;    // paragraphs of 15-70 words that carry a fact
signals.quotable_share;  // % of sentences an engine could attribute unedited
signals.faq_questions;   // declared in FAQPage markup
signals.faq_visible;     // ...and actually readable on the page

The question is not whether the copy is good. It is whether, when an answer engine wants to state a fact about this page, there is a sentence on it that can be lifted as it stands — or whether the engine has to assemble one, in which case the answer carries the engine's wording rather than yours.

This module publishes the reading, not the scoring. Every count and share the production service records for a page is produced here and is reproducible byte-for-byte against a live scan — that is what the parity test does, and it refuses to compare unless the page bytes still match the ones the fixture was captured from. What the service keeps is what those readings are judged against: the thresholds, the weight the section carries in the overall grade, and the multi-hundred-site corpus pass the numbers were calibrated from.

The distinction is not a marketing line, it is the one that keeps the product honest. Anyone can check that we read their page correctly; nobody has to take our word for a number. What they cannot lift is the calibration, which is the part that took a corpus to earn.

Two details that are easy to get wrong and are settled here. FAQ parity is checked against the whole visible page, not the stripped body — an FAQ accordion inside a <footer> is still text a reader can see. And counts come from body paragraphs and list items with navigation, header, footer and forms removed, because a nav menu is not prose and counting it inflates every ratio on the page.

Scoring the reading — quoteRows, quoteBlock, the section weight and the score version — is in the service, not this package. Point the free scan at a URL to see the rows: https://crawlcheck.io/.

rows — the three-state row every reader returns

import { tgt, pctScore } from "crawlcheck-core/rows";

A row is { k, cur, opt, ok, why }: the check, what this page currently does, what it should do, the verdict, and why it matters. ok: null is the third state and it is load-bearing — it means measured but not scored. A signal too rare to judge on, one that does not apply to this kind of site, or one that is purely informational is shown to the reader and deliberately left out of the arithmetic. pctScore() divides by the scored rows only and returns null, never 0, for a section with none: an unmeasured section is not a failing one.

That rule is why a service-area business is not marked down for having no street address, and why a brand-new site with no field performance data is not reported as slow. Most audit tools have two states, and every gap in their own measurement becomes your zero.

surfaces — markdown negotiation, and the Accept header that made a site an F

import { agentSurfaces, grab } from "crawlcheck-core/surfaces";

const root = await grab("https://example.com/");
const out = await agentSurfaces("https://example.com", root);

out.found.markdown;    // Accept: text/markdown got a text/markdown answer
out.found.agents_md;   // an authored /agents.md, not a 200 of HTML
out.found.link_header; // read off the root response — no extra request
out.evidence.markdown; // { content_type: "text/markdown; charset=utf-8" }

A scanner's Accept header is part of its identity, and getting it wrong makes the whole grade a measurement of the wrong document. This fetcher sent Accept: */* until 2026-09-01. vercel.com answered that with 3,025 bytes of text/markdown, while every named-crawler row — which sends text/html — received the 521 KB page. The scan then scored mobile 0 (there is no viewport meta in a markdown file), entity-map parity 0, and graded the site F: a grade about a document no crawler is ever handed. Googlebot, GPTBot and ClaudeBot all send a browser-shaped Accept, so CRAWLER_ACCEPT does too.

Markdown negotiation is therefore measured on purpose, once, in its own request — never as a side effect of how the page was fetched. Conflating "this site offers markdown to agents" with "our fetcher asked for the wrong thing" is the failure mode, and one test in this package exists solely to pin it: exactly one request carries Accept: text/markdown, and every other request carries CRAWLER_ACCEPT.

The other rule here is that a 200 answering with HTML is a soft 404. /SKILL.md, /agents.md and the three .well-known paths are absent on most sites, and a catch-all route returns the SPA shell at 200 for all of them; a naive check reports every one of those as present. Nine of the ten rows this module renders are informational and none is scored — adoption is early, absence is not a defect, and a row that says NOT APPLICABLE: no Product, Offer or checkout found is measuring the site rather than the checklist.

jsonld — the entities a page actually declares

import { ldGraphNodes } from "crawlcheck-core/jsonld";
ldGraphNodes(html); // root nodes and @graph children, merged by @id

Two rules, both easy to get wrong. A node nested inside another node is a value — a PostalAddress, an Offer, a ListItem, an Answer — not a subject the page is asserting, and it is not held to the standards an entity is. And the same @id twice is one subject: RDF merges statements, so a reader that keeps the first node and drops the second reads a site declaring a typeless fingerprint stub before its full Organization as having an untyped business. That was measured on a live site, which is why the merge is here.

images — what an engine can learn from the pictures

import { imageSignals, imageRows } from "crawlcheck-core/images";

const sig = imageSignals(html);
// { total:9, no_alt:0, empty_alt:0, no_dimensions:0, lazy:7, lazy_above_fold:0,
//   modern_format:8, svg:1, og_image:true, ld_image:true, ld_imageobject:true, ... }
imageRows({ images: sig });   // eight rows, five scored

Five rows are scored: a missing alt attribute, width and height declared, nothing lazy-loaded above the fold, an entity image in the JSON-LD, an og:image. Three are reported and deliberately never scored, and that is the point of the module.

alt="" is not a missing alt. It is the correct markup for a decorative image, and most real sites are nearly all decorative alt, so scoring it fails almost every site on a signal that is usually right. Only a missing attribute counts. File format and hero priority are preferences rather than defects. And a page with no images returns one unscored row, so the section scores null instead of zero.

The reader carries a fix that came out of writing these tests: the attribute pattern excluded whitespace, so any quoted value containing a space read back as an empty string and alt="a cedar fence" was indistinguishable from alt="". Measured on six live homepages, that reported 15 of 15, 9 of 9 and 10 of 11 images as decorative when almost all carried real alt text.

Tests

npm test            # 48 tests, no network (4 skipped)
CC_LIVE=1 npm test  # 48: adds the four live parity checks

robots — RFC group parsing, precedence, the shadowing case both ways, unmeasured inputs, table integrity, and parity with the production service on two real robots.txt files.

agents — token matching, IPv4/IPv6 range logic, the three-state verdict, and parity with the production verifier on 11 real log lines against a stored snapshot of the five operator feeds those lines touch (test/fixtures/ranges.json, dated inside the file).

quotable — the reader (what counts as a paragraph, the lead test, question headings, FAQ parity, what makes a sentence quotable), the rows (which are scored, which are deliberately not, what happens under five paragraphs), the JSON-LD graph rules, and parity with a live production scan.

surfaces — the Accept headers actually sent, one negotiation request and only one, soft-404 handling, and the row wording.

How the parity fixtures work

Each .quotable.json holds the output of one real scan at crawlcheck.io — the service's own numbers, copied unaltered, with the scan id, score version and build sha they came from. The page itself is not committed. Instead the fixture stores page_sha256, the digest of the exact response body that scan read, and the live test re-fetches the page with the same user-agent and Accept, refuses to compare unless the bytes are identical, and only then checks that this module reproduces all 21 fields and the section score.

That makes drift loud instead of silent. A stored HTML blob would keep passing forever against a page that no longer exists; a digest mismatch says plainly that the site changed, not the reader, and the fixture needs recapturing.

One of the two fixtures is deliberately not a re-fetch target. emmanuelorta.com renders live figures on an hourly refresh — measured on 2026-09-03, an identical-length response returned a different digest three hours after capture — so its fixture carries reproducible: false and the reason. It stays because the two recorded readings disagree (100 against 67; nine FAQ questions against none; markdown negotiated against not), which is what proves the reader does not flatten one site into another.

images — the three distinctions above pinned as tests (missing vs empty alt, a page with no images, format and priority unscored), the row contract and the three-state verdict, malformed input, and parity with a live production scan reproducing every field the service recorded from the same bytes.

Roadmap

Next: the machine-file readers (llms.txt, entity map, the soft-404 and challenge verdicts behind them). Each ships only with a parity test against the live service.

Licence

MIT. Built by Emmanuel Orta. The service itself is at crawlcheck.io: a free A–F audit of what AI crawlers and agents are served, no account.