io.github.sharozdawa/robotstxt-ai

Official

by sharozdawa · TypeScript

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Analyze and generate robots.txt files with AI detection for 20+ crawler bots.

Topics

ai-crawlerclaudebotgptbotnextjsopen-sourceperplexitybotrobots-txtrobots-txt-generatorseoai-bot-blocker

npm GitHub stars License

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

</div>

Visual Toggle UI — Block or allow AI bots with simple on/off switches
20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
Generate robots.txt — Create a complete robots.txt with your chosen rules
MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
Check Bot Status — Verify if a specific bot is blocked on any website

Tool	Description
`fetch_robots`	Fetch and analyze a robots.txt from any URL
`analyze_robots`	Analyze pasted robots.txt content for AI bot blocking status
`generate_robots`	Generate a robots.txt with specified blocked bots and custom rules
`list_ai_bots`	List all known AI bots with user-agents, companies, and descriptions
`check_bot_status`	Check if a specific bot is blocked on a given website

bash

git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

Add to your claude_desktop_config.json:

json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Add to .cursor/mcp.json:

json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

The server knows about 25+ bots including:

AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Tool	Description
awesome-seo-mcp-servers	Curated list of SEO MCP servers and agent skills
indexnow-mcp	Instant URL indexing via IndexNow
schema-gen	Schema.org JSON-LD markup generator
ai-visibility	AI brand visibility tracker

MIT

Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert

claude_desktop_config.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp@1.0.1"
      ]
    }
  }
}