Agent♥︎Age
Catalog

ActableSite AI Crawler Monitor

Official5 toolsLive

by unitedideas · JavaScript

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

ActableSite AI Crawler Robots Monitor (MCP) Server

A remote, read-only Model Context Protocol (MCP) server that provides public website evidence and evaluates AI crawler robots.txt policy. It also monitors public-site policy signals, including sitemap and llms.txt changes, and exposes metadata related to a Crawler Watch offer.

🛠️ Key Features

  • Remote, read-only MCP server
  • Checks AI crawler robots.txt policy and public-site policy
  • Monitors sitemap and llms.txt changes
  • Exposes 5 tools over Streamable HTTP

🚀 Use Cases

  • Website audit workflows focused on public identity, evidence, and discovery files
  • AI crawler policy verification using multiple crawler tokens
  • Change monitoring across policy and discovery resources (public-site policy, sitemap, llms.txt)

⚡ Developer Benefits

  • Tool-based access (toolCount: 5) to structured checks
  • check_ai_crawler_policy evaluates eight tokens (OpenAI, Anthropic, Perplexity, Google)
  • audit_public_website returns identity, offer evidence, action paths, metadata, structured data

⚠️ Limitations

  • Remote and read-only; no updates to site content
  • Tools are provided over Streamable HTTP
  • Crawler Watch offer metadata includes a one-site scope and 15-minute cadence

Topics

aeoai-crawlerscrawler-policydeveloper-toolsmcpmcp-servermodel-context-protocolread-onlyremote-mcpseostreamable-httpwebsite-auditai-agentai-toolschatgptmcp-toolsopenairobots-txtmcp-clientstdio

Related servers

More in Search & Web