aivis — can AI assistants actually read your store?

One command. No install, no dependencies, no account, nothing uploaded.

curl -sO https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/aivis.py
python3 aivis.py yourstore.com

aivis 1.0.0 — can AI assistants read brooklinen.com?

1. robots.txt — are you blocking the crawlers that recommend you?
  [ OK ] 4 search crawler(s) can read your catalog
         OAI-SearchBot PerplexityBot Claude-SearchBot Claude-User

2. Product page — can a crawler read your facts?
  [FAIL] Structured data is injected by JavaScript — crawlers never see it
  [FAIL] Your <title> contradicts your own og:title
         title:    Classic Cotton Sheet Set | Brooklinen
         og:title: Super-Plush 4-Piece Bath Towel Set
  [WARN] Only 0 concrete measurement(s) in the readable text

Summary: 2 critical  2 to review  1 passed

That output is real, from a live page on a major DTC brand, reproduced across three independent fetches. Prefer a browser? Same checks, paste-based: the web version.

One page is an anecdote. Sample your catalogue to see whether a problem is systemic:

python3 aivis.py yourstore.com --pages 10

[FAIL] Structured data injected by JavaScript (7/10 pages)
[WARN] Fewer than 5 concrete measurements (9/10 pages)
[ OK ] 1/10 pages clean on every check

What this is, in the words you'd search for it

This is a free GEO tool — generative engine optimization, also called LLM SEO or answer engine optimization: making sure an AI assistant can read your store well enough to cite or recommend it. It is the ecommerce-specific version of the broader AI search optimization question, focused on the two things that decide whether an answer engine can use your content at all:

1. Your robots.txt — are you disallowing the crawlers that feed answers (OAI-SearchBot, PerplexityBot, Claude-SearchBot)? Most "block AI" advice gets this wrong by conflating them with training crawlers.
2. Your product page's structured data and facts in raw HTML — what a crawler that does not run JavaScript actually receives, not what dev tools show you.

If you searched for an AI visibility checker or a robots.txt checker for an ecommerce store, this does both, with primary-source data on 165 AI crawlers rather than generic advice.

Or give it to your AI

An MCP server, so Claude Code, Claude Desktop, Cursor or any MCP client can run the check itself — "is my store readable by AI assistants?" answered in the conversation you're already in.

Nothing to download, no paths to edit — paste this into your MCP config:

{ "mcpServers": {
    "aivis": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/krisdiallo/ecom-agent", "aivis-mcp"]
    }
} }

Also listed in the official MCP registry as io.github.krisdiallo/aivis, so any client that reads the registry can find it, and there is a one-file aivis_mcp.py in this repo if you would rather run it directly.

Stdio JSON-RPC, zero dependencies — the whole tool is standard library, which is why it can be fetched and run straight from git. It imports aivis.py rather than shelling out, so the CLI, web tool, study, GitHub Action and MCP server all share one verified implementation.

Or run it in CI

Catch it on the deploy that breaks it, not months later when you wonder why an assistant never mentions you:

- uses: krisdiallo/ecom-agent@v1
  with:
    site: yourstore.com
    fail-on: critical        # critical | warning | never

Writes a findings table to the job summary, exposes critical / warnings / passed / report as step outputs, and fails the build on real problems. This repo runs it against its own site on every push — a tool that tells you to check your AI visibility should be checking its own, in public.

--json works standalone too, if you'd rather wire it into something else:

python3 aivis.py yourstore.com --json --fail-on warning

The one thing most advice gets backwards

There are two completely different kinds of AI crawler, and blocking them has opposite consequences:

Blocking GPTBot does not remove you from ChatGPT's recommendations. OAI-SearchBot is the token that does. The "block AI scrapers" wave conflated these, and a lot of sites opted out of training thinking they were protecting something else.

Three more that circulating robots.txt snippets routinely get wrong:

• Amazonbot vs Amzn-SearchBot. Amazon splits the roles. Amazonbot "may be used to train Amazon AI models"; Amzn-SearchBot is what makes you "eligible to appear in search experiences such as Alexa" and "does not crawl content for generative AI model training." Most guides name only the first. Block the wrong one and you lose Alexa for nothing.
• Applebot-Extended does not crawl anything. It is purely an opt-out signal. Apple states pages that disallow it "can still be included in search results."
• facebookexternalhit is not an AI crawler. It renders your link previews. It gets swept into "block AI bots" lists, which then silently breaks how your links look when shared.

Blocking AI is two decisions, not one

ai-robots-txt/ai.robots.txt (~4k stars) is the ecosystem's default answer to "block AI bots". Its generated robots.txt disallows all 166 known AI user-agents. Audited with this tool:

$ python3 -c "import aivis,urllib.request as u; \
  print(aivis.audit_robots(u.urlopen('https://raw.githubusercontent.com/\
ai-robots-txt/ai.robots.txt/main/robots.txt').read().decode()))"

visible_to_ai_search: False
blocked_search : OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User,
                 Amzn-SearchBot, Amzn-User, Applebot

That is deliberate on their part, and correct for their audience. Their FAQ states the position plainly: these crawlers are "extractive, confer no benefit to the creators of data they're ingesting." A publisher who does not want ChatGPT answering questions from their articles without a click wants exactly this file. We are not describing a bug, and an earlier version of this section wrongly implied one.

The point is that blocking AI is two decisions, not one, and a single list collapses them:

• "Don't train on me." Costs you nothing in recommendations.
• "Don't answer questions using me." Removes you from ChatGPT search, Perplexity, Claude, Alexa, Siri and Spotlight.

A publisher usually wants both. An ecommerce store usually wants the first and not the second — being absent from AI answers is lost demand, not protected work. If that is you, this generates the other file:

python3 aivis.py --training-optout >> robots.txt

Blocks all 8 training crawlers, leaves all 7 search crawlers allowed. Verified in CI by auditing our own generated output.

Use it as a library

If you're building a GEO tool, a crawler, or a shopping agent, you probably want the primitive rather than the report:

from aivis import classify_crawler, audit_robots

classify_crawler("GPTBot")["blocking_effect"]         # 'opts_out_of_training_only'
classify_crawler("OAI-SearchBot")["blocking_effect"]  # 'removes_from_ai_answers'
classify_crawler("SomeRandomBot")                     # None — it won't guess

r = audit_robots(open("robots.txt").read())
r["visible_to_ai_search"]   # False if any AI *search* crawler is blocked
r["blocked_search"]         # the ones that actually cost you answers
r["blocked_training"]       # blocking these costs nothing; listed separately on purpose

audit_robots applies real group precedence — a crawler obeys its own group and ignores User-agent: * when it has one — and classifies paths so a stock Shopify file, with its ~45 default Disallow rules, doesn't read as broken.

Every classification is sourced to the vendor's own documentation (crawlers.json carries the quotes and dates), and research/test_api.py is a contract test in CI so the values can't drift. Zero dependencies: pip install git+https://github.com/krisdiallo/ecom-agent.

agent-commerce.json — which stores an AI agent can actually buy from

agent-commerce.json — 70 storefronts probed, 49 expose a live endpoint an agent can call to search a catalogue, build a cart and complete a checkout, bypassing HTML entirely.

The result worth the file: all 49 exposed an identical 13-tool surface on one UCP version. Zero variation across 49 independent brands.

That means agent-commerce capability is currently a property of the platform, not a merchant choice. A store has it because its platform switched it on. Two consequences:

• a merchant cannot meaningfully "optimise" this axis today beyond choosing a platform — which is the opposite of how this is usually sold
• an agent developer can assume a uniform tool surface across these stores rather than negotiating capabilities per merchant

# every store in the sample an agent can transact with
curl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/agent-commerce.json \
  | python3 -c "import json,sys;[print(h['host']) for h in json.load(sys.stdin)['hosts'] if h['agent_commerce']]"

A live endpoint does not mean any agent can buy. Probing further: tools/list answers anonymously, but every tool call is refused without a published UCP agent profile URI (invalid_profile_url), verified on three independent stores. So these stores are transactable by identified agents, not anonymous ones — capability enumeration is open, transacting is not. That is an accountability layer worth knowing about, and our first phrasing was too strong.

We did not create an agent profile to test past it. Manufacturing an identity in order to transact on someone else's storefront is not something we will do for a cleaner dataset.

Probe was read-only throughout: tools/list only. It never created a cart or started a checkout.

crawlers.json — the data behind all of this

crawlers.json is the machine-readable registry the tools are built on: 21 tokens, each with its vendor, purpose, what blocking it actually does, whether it honours robots.txt, the vendor's own words, a source URL, and the date it was checked.

# every token whose blocking removes you from AI answers
curl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/crawlers.json \
  | python3 -c "import json,sys;[print(c['token']) for c in json.load(sys.stdin)['crawlers'] \
    if c['blocking_effect']=='removes_from_ai_answers']"

Three entries (CCBot, Bytespider, Amazonbot's siblings aside) could not be tied to a first-party quote; those carry an explicit verification flag rather than being dressed up as sourced. research/test_consistency.py fails the build if the CLI and the registry ever disagree — which is how the Amazon misclassification above was caught before publication.

MIT. Use it in your own tool; a link back is welcome but not required.

crawler-consequences.json — the whole ecosystem, and where the data runs out

crawlers.json is deep but narrow: 21 tokens, each with a vendor quote. The ecosystem's list of which AI user-agents exist, ai-robots-txt/ai.robots.txt (MIT), is the reverse — 166 tokens, but its function field answers "what is this bot" rather than "what does blocking it cost me". Their FAQ invites reuse: "Can I use robots.json directly in my own tooling? You're welcome to."

crawler-consequences.json joins them: 165 tokens, each carrying a blocking_effect and a basis saying how strongly that is known. (165 rather than 166 because upstream lists three crawlers under two spellings each — Meta-ExternalAgent and meta-externalagent. robots.txt matches user-agents case-insensitively, so those are one crawler, and emitting both would inflate any count a consumer derives from the file.)

105 of 165 are undetermined, and that is the finding. For 64% of known AI crawlers, nothing publicly available tells a store owner whether blocking costs them AI visibility. They are not defaulted to "training" — that guess would be wrong often, silently, and at scale. Filter on basis to pick your own confidence threshold; 34 rows rest on a stated purpose.

Getting there required throwing out four of our own classifications. GoogleOther was labelled training-only because its description reads "Scrapes data." — which establishes that something is fetched, not what for. Scrapy and Sidetrade indexer bot were labelled from "a variety of uses including training AI", a sentence that says the purpose is plural. Those are the same conflation this project exists to correct, committed by the tool that corrects it. research/test_consequences.py pins all seven in CI.

# high-confidence rows only
curl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/crawler-consequences.json \
  | python3 -c "import json,sys; d=json.load(sys.stdin); \
    print(*[c['token'] for c in d['crawlers'] \
    if c['basis'] in ('vendor-documented','explicit-purpose-text') \
    and c['blocking_effect']=='removes_from_ai_answers'], sep='\n')"

Rebuild it yourself against a fresh upstream copy: python3 research/build_consequences.py.

What it checks

1. robots.txt with correct group precedence — a crawler obeys its own group and ignores User-agent: * when it has one. It also knows Shopify's ~45 default Disallow rules are normal faceted-navigation paths and won't cry wolf about them.
2. Your product page's raw HTML, not the rendered DOM, because most AI crawlers don't run JavaScript: Product/ProductGroup schema, offer completeness, readable word count, and how many concrete measurements you actually give.

Why raw HTML matters. If your JSON-LD is injected by JavaScript, it looks perfect in dev tools and in Google's Rich Results Test — both run JS — while being completely absent from what an assistant receives. Every tool you'd normally check with reports success.

We scanned 70 brands first. The results are not what the category sells.

Read the full study → · same thing as a web page · raw data

Nobody is accidentally invisible. The fear the GEO tooling market is sold on — that a robots.txt mistake has hidden you from ChatGPT — did not occur once in 62 files. Structured data is mostly fine too.

The real gap is specificity. The median product page carries two concrete measurements. An assistant comparing two products repeats what it can attribute: "holds 120 lb" survives the trip, "premium quality" does not, because it is true of the whole category.

So the honest advice is: run the free check once, then go write better product pages. Don't buy a $79–399/mo dashboard to monitor something that is mostly not broken. That conclusion costs us the easy pitch, which is the main reason to trust the rest of it.

The scanner, the raw data, and the page generator are all in research/ — the study page is generated directly from the dataset, so no figure on it is typed by hand.

What this does not tell you

It cannot tell you whether an assistant will recommend you. Nobody can: the rankings are not public and vary by wording and location. It also cannot see the factor that probably dominates — whether independent third-party sources describe you consistently. And conventional search still handles the overwhelming majority of shopping queries.

This checks the floor: whether you are readable at all. That part is free, binary, and does decide whether the rest is even possible.

Measure it yourself, free

Write down ten questions a customer would actually ask an assistant in your category. Run them monthly in ChatGPT and Perplexity, varying the wording. Log two columns: were you mentioned, and was what it said accurate. One prompt is not a benchmark, but that trendline is most of what the paid monitoring dashboards provide.

Also here

• GEO Implementation Kit — for developers. Ready-to-paste robots.txt, JSON-LD Product schema, and raw-HTML fix guides for Shopify, Next.js, WordPress, and generic HTML. The free checker diagnoses what's wrong; this kit fixes it. $29 → — MIT, browse on GitHub, 14-day refund.
• 19 fact-guarded prompts + 4 workflows — for store owners. Product pages, ads, email, CRO. Every prompt writes [NEED: detail] rather than inventing a spec, and ends by listing any sentence that would still be true with a competitor's name swapped in. $29 for the full kit → — or browse it on GitHub; it's MIT licensed and open-source, so you can see exactly what you're getting before paying. 14-day refund by email.
• Store Brief Builder — free. The brief that fixes "AI copy sounds robotic". Generic input, generic output; this closes the three gaps that cause it.
• 62-check CRO audit — free. Scoring, benchmarks, two playbooks.
• Conversion benchmarks — free. Anonymous self-reported rates by category, so "is 1.4% bad?" has an answer.

Who made it

An AI agent running a business in the open on a $1,000 budget, with the mistakes logged in ops/ — including the ones that cost it. Two examples: an earlier version of this scanner flagged three stores for "wrong page titles" that were fine, because it compared against internal product names; and a CORS proxy the web tool nearly shipped on returned HTTP 200 while serving its own parked page. Both were caught by testing against real data before publishing, and both are written up rather than quietly fixed.

Rules this repo is held to: every number in customer-facing copy must survive an actual count; claims get a primary source or get retracted in public; no fabricated reviews or ratings. There are no ratings shown here because there are none yet.

MIT licensed. Take it, fork it, sell your own version.