Agent♥︎Age
Catalog

ActableSite AI Crawler Monitor

Officialdeprecated3 tools

by unitedideas · JavaScript

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

ActableSite Readiness MCP (com.actablesite/readiness)

A remote, read-only MCP server that provides public website evidence, evaluates AI crawler robots.txt policy, and monitors public-site policy, sitemap, and llms.txt changes. It exposes tools over Streamable HTTP to support website audit and crawler-policy workflows. The server is identified by the slug com-actablesite/readiness.

🛠️ Key Features

  • Remote, read-only MCP server
  • Checks public-site identity, offer evidence, action paths, discovery files, metadata, and structured data
  • Evaluates eight OpenAI, Anthropic, Perplexity, and Google crawler tokens
  • Monitors public-site policy, sitemap, and llms.txt changes
  • Tooling over Streamable HTTP

🚀 Use Cases

  • Website audit for public identity and structured data
  • AI crawler policy verification via robots.txt policy evaluation
  • Ongoing monitoring of sitemap and llms.txt changes
  • Supporting crawler watch metadata workflows (“Crawler Watch” offer/practice radar)

⚡ Developer Benefits

  • Streamlined access via MCP tools for policy and evidence gathering
  • Read-only design for safe public-site analysis
  • Provides structured outputs for audit and policy checks

⚠️ Limitations

  • Remote access and read-only operation
  • Focused on public website evidence, crawler policy, and specific change monitoring (policy, sitemap, llms.txt)

Topics

aeoai-crawlerscrawler-policydeveloper-toolsmcpmcp-servermodel-context-protocolread-onlyremote-mcpseostreamable-httpwebsite-auditai-agentai-toolschatgptmcp-toolsopenairobots-txtmcp-clientstdio

Related servers

More in Search & Web