io.github.houtini-ai/seo-crawler-mcp

Official

by houtini-ai · TypeScript

Crawl and analyse websites for SEO errors using Crawlee with SQLite storage

io.github.houtini-ai/seo-crawler-mcp (SEO Crawler MCP) — retired

This MCP server crawls and analyzes websites for SEO errors using Crawlee, persisting results in SQLite. It was retired and superseded by the “SEO Audit Console” project. The crawl phase was rewritten to be stdio-safe and integrated as the crawl component of a larger tool.

🛠️ Key Features

Website crawling and SEO error analysis
Crawlee-based crawling
SQLite storage of crawled results
Provides analysis queries over stored crawl output

🚀 Use Cases

Technical SEO audits over a whole site
Running analysis queries against previously crawled site data

⚡ Developer Benefits

stdio-safe crawler for MCP usage
Crawl results stored in SQLite for further querying

⚠️ Limitations

Project is retired; use SEO Audit Console instead
Readme excerpt describes “25+ analysis queries,” without details of supported query types

Topics

crawleecrawlermcpseoseo-analysissqlitetechnical-seo-auditlibrecrawl

README.md2 KBView on GitHub

SEO Audit Console

SEO Crawler MCP - retired

This project has been superseded by SEO Audit Console.

SEO Crawler crawled your whole site into SQLite and gave Claude 25+ analysis queries over the result. The crawler survived the move - it was rewritten to be stdio-safe and it is now the crawl half of a bigger tool.

SEO Audit Console keeps the crawl and adds the thing the crawl could never tell you on its own: what people actually searched for to get there. It merges the crawl with your Google Search Console history and on-demand DataForSEO data, then ranks every finding by the clicks it could recover rather than by severity.

A flat crawler sells you severity. This ranks by yield.

Where to go

bash

git clone https://github.com/houtini-ai/seo-audit.git

Docs and setup: github.com/houtini-ai/seo-audit

What this means for you

The npm package still installs. @houtini/seo-crawler-mcp is deprecated, not unpublished. Nothing you have running will break.
This repository is archived. The code stays readable and every existing link keeps working, but there are no further releases, and issues and pull requests are closed.
The crawler got better on the way over. It no longer depends on Crawlee (which logged to stdout and corrupted MCP's JSON-RPC framing), it seeds discovery from sitemaps and known GSC URLs rather than links alone, and it guards against redirects that wander off-host.

Migrating

Start a crawl on the new tool and it builds its own database:

code

start_crawl for https://example.com/

Run refresh_property instead if you want the crawl, the Search Console sync and URL inspection in one pass.

Part of the Houtini open-source MCP set. Questions: hello@houtini.com.

Install

Configuration

Environment variables

OUTPUT_DIRrequired

Directory where crawl results are saved

DEBUG

Enable verbose debug logging (set to 'true' to enable)

Add to VS Code Add to Cursor

claude_desktop_config.json

json

{
  "mcpServers": {
    "seo-crawler-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "@houtini/seo-crawler-mcp@2.0.1"
      ],
      "env": {
        "OUTPUT_DIR": "<YOUR_OUTPUT_DIR>",
        "DEBUG": "<YOUR_DEBUG>"
      }
    }
  }
}