io.github.jmshinhwa/robots-txt-ai-crawler-audit

Official

by jmshinhwa · JavaScript

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

io.github.jmshinhwa/robots-txt-ai-crawler-audit (MCP Server)

This MCP server checks a site’s robots.txt line by line and reports what each AI crawler token actually controls, specifically in terms of training permissions and related AI access. The server is intended to help verify that the robots.txt directives match the expected behavior for AI crawlers.

🛠️ Key Features

Performs line-by-line analysis of robots.txt
Explains the effect of each AI crawler token
Focuses on what tokens control for training and AI

🚀 Use Cases

Auditing robots.txt for AI crawler behavior
Validating that directives correctly govern training-related access
Identifying mismatches between intended and configured AI controls

⚡ Developer Benefits

Clear mapping from robots.txt entries to AI crawler token control
Deterministic, per-line checking to support review workflows

⚠️ Limitations

Limited to information expressible through robots.txt line-by-line token controls

Topics

compliancecursordevsecopslintermcp-serveropen-vsxvscode-extension

Install

Add to VS Code Add to Cursor

claude_desktop_config.json

json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit@0.1.3",
        "--mcp"
      ]
    }
  }
}