io.github.agentwares/agent-access-index

Official3 tools

by agentwares

Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.

MCP Server: io.github.agentwares/agent-access-index

The io.github.agentwares/agent-access-index MCP server provides information about whether an AI agent may fetch a URL, including the terms of access and the applicable price. Its description states that access is free, with no key required and no signup.

🛠️ Key Features

Determines whether URL fetching is allowed for an AI agent
Reports terms of access and pricing

🚀 Use Cases

Check URL-fetch permissions for AI agents
Look up access terms and cost for fetching a given URL

⚡ Developer Benefits

No API key required
No signup required

⚠️ Limitations

Only the URL-fetch permission, terms, and price are described; no additional capabilities are provided in the source data.

Topics

agent-toolsai-agentsapidocumentationhostedllms-txtmcpmcp-servermodel-context-protocolx402

Captured live from the server via tools/list.

access_check_url

Given a URL, say whether an AI agent may fetch it, on what terms, and at what price. Reads robots.txt for a generic agent and for every AI user-agent the operators document, the Content-signal and Content-Usage directives, any RSL licence the origin links, and the live response to an honestly-identified request. Verdicts: `allowed` (it served us), `allowed_if_identified` (the written policy permits this but bot management refused an anonymous request — signing as a named crawler is the documented route), `charged` (there is a price, quoted where the rail will show one), `blocked` (the site disallows the agents that would want this, with nothing to buy), and `unknown` (the origin did not answer, or the URL is not there). Never evades: no CAPTCHA solving, no proxies, no spoofed user-agents. Refusals come back as findings with the evidence attached.

Parameters3

url	string	required	Absolute http(s) URL of the page to check, e.g. https://example.com/article
includeEvidence	boolean	optional	Include every request made, with status and headers (default false: it is large)
skipWellKnown	boolean	optional	Skip the five well-known probes; faster, and they are almost always absent

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "description": "Absolute http(s) URL of the page to check, e.g. https://example.com/article"
    },
    "includeEvidence": {
      "description": "Include every request made, with status and headers (default false: it is large)",
      "type": "boolean"
    },
    "skipWellKnown": {
      "description": "Skip the five well-known probes; faster, and they are almost always absent",
      "type": "boolean"
    }
  },
  "required": [
    "url"
  ],
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false
}

access_explain_verdicts

Return the five verdicts this index can return, what each one means for an agent that has just been refused, and what the agent's next move is for each. Costs nothing and makes no request; call it once to interpret access_check_url results.

Parameters

No parameters.

Raw schema

{
  "type": "object",
  "properties": {},
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false
}

access_list_agents

Return the AI crawler tokens this index reads robots.txt rules for — only tokens their operators document — plus the widely-written tokens that no operator sends, which make a robots rule unenforceable. Useful when writing or auditing a robots.txt.

Parameters

No parameters.

Raw schema

{
  "type": "object",
  "properties": {},
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false
}

agent-access-index/README.md4 KBView on GitHub

Agent access index

Ask whether an honestly identified agent may fetch a URL, and why: the verdict, the exact robots.txt line that decided it, what each named AI crawler is told, and the payment rail when access is for sale. Free, no key.


Service	https://agentwares-access-index.vercel.app
MCP endpoint	`https://agentwares-access-index.vercel.app/mcp` (Streamable HTTP)
Registry name	`io.github.agentwares/agent-access-index` (v0.1.0)
Tools	`access_check_url`, `access_explain_verdicts`, `access_list_agents`
In this directory	`server.json` · `llms.txt` · tool reference

Verdicts are allowed, allowed_if_identified, charged, blocked and unknown, each with reason codes and the rule behind them. allowed_if_identified is the useful one: the site's own policy permits the fetch and bot management refused it anyway, which is a refusal with a documented way out rather than a decision the publisher made about you. Across 48 hosts harvested from real crawler-failure issues, 30.8% were that case.

It reads robots.txt per RFC 9309 for a generic agent and for the 19 AI user-agents their operators document, plus Content-Signal, Content-Usage, RSL licences, llms.txt, the bot-management vendor from response headers, and the payment rails.

What it will not do, ever:

No impersonation. It never sends another operator's crawler token to see how a site treats it. The per-agent column comes from what the origin declared, because spoofing measures what an edge does with a liar.
No CAPTCHA solving, no proxies, no retry with a different face. One documented user-agent.
It obeys its own robots.txt rule and reports the refusal rather than working around it.
An origin that will not serve its own robots.txt answers unknown with ROBOTS_UNREADABLE, never a guess about a policy it could not read.

No LLM is called, so a lookup costs nothing to serve and nothing to buy.

Authentication

None. No key, no signup, no payment, no rate-limit tier to buy.

First call

Free: the verdict for any public URL.

curl -s "https://agentwares-access-index.vercel.app/policy?url=https://example.com/article"

MCP client configuration

Claude Desktop, Cursor, Claude Code and anything else that speaks Streamable HTTP:

json

{
  "mcpServers": {
    "agent-access-index": {
      "url": "https://agentwares-access-index.vercel.app/mcp"
    }
  }
}

Endpoints

Endpoint	What
`POST /mcp`	the MCP server (Streamable HTTP), no key
`GET /policy?url=`	verdict, reasons, per-agent terms and rails for one URL
`GET /llms.txt`	machine docs
`GET /server.json`	registry server card
`GET /health`	health

Pricing

Free. This service has no pricing.json because it does not charge and has no paid tier to advertise.

Source and issues

This is a hosted service; its implementation is not in this repository (see the root README). Open an issue here for anything about the running service — a wrong answer, a dead endpoint, a tool description that misled your agent, a capability you need.

The files in this directory are generated from the service, so please report a problem rather than sending a documentation patch.

Install

Tools (3)

access_check_urlaccess_explain_verdictsaccess_list_agents

Remote endpoint

Streamable HTTP

Hosted server - connect over the network, no local install.

url

https://agentwares-access-index.vercel.app/mcp

Add to VS Code Add to Cursor

claude_desktop_config.json

json

{
  "mcpServers": {
    "agent-access-index": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://agentwares-access-index.vercel.app/mcp"
      ]
    }
  }
}

Desktop config is stdio-only; this bridges via mcp-remote. Native remote: Settings > Connectors.

MCP Server: io.github.agentwares/agent-access-index

🛠️ Key Features

🚀 Use Cases

⚡ Developer Benefits

⚠️ Limitations

Topics

access_check_url

access_explain_verdicts

access_list_agents

Agent access index

Authentication

First call

MCP client configuration

Endpoints

Pricing

Source and issues

Install

Remote endpoint

Related servers