Spicrawl MCP Server: web scraping for Claude, Cursor and AI agents

To let Claude, Cursor or another AI agent scrape websites, connect the Spicrawl MCP server. Add the hosted endpoint https://mcp.spicrawl.com/mcp (or run npx -y @spicrawl/mcp locally) with a Spicrawl API key, and the agent can call spicrawl_scrape to read any URL as Markdown, text, HTML or JSON. It renders JavaScript pages, extracts structured data, scrapes up to 10,000 URLs in one batch job, takes screenshots and keeps logged-in browser sessions, all as native spicrawl_* tools in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex. The free tier includes 1,000 credits per month.

smithery badge

Docs · Get an API key · TypeScript SDK · CLI · Agent plugins · Changelog

Install from: npm · Smithery · Glama · MCP Registry · Dify plugin · Cursor Directory

Quickstart

1. Create an API key at app.spicrawl.com. Keys start with spicrawl_live_ or spicrawl_test_.

2. Connect the hosted endpoint, https://mcp.spicrawl.com/mcp (MCP Streamable HTTP, nothing to install):

   bash

   export SPICRAWL_API_KEY=spicrawl_live_...
   claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
     --header "Authorization: Bearer $SPICRAWL_API_KEY"
   

   Codex: codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

3. Or run it locally over stdio (Node.js 20+), for clients that only launch local servers:

   bash

   claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
   

4. Ask your agent: "Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price." It calls spicrawl_scrape and gets the page back as Markdown.

Both transports serve the same spicrawl_* tools (25 are defined; the hosted endpoint lists 24 until remote browsers launch, see Tools), and every call runs under your key with the same scopes, limits and credits as a direct API call.

What it does

• Scrape any URL to Markdown, text, HTML, a JSON envelope or a printed PDF with one tool, spicrawl_scrape. Markdown is the default and strips the page to its main content.
• Render JavaScript with render: true (3 credits). A plain fetch costs 1 credit, screenshots, PDFs and actions use a full browser (8 credits), and mode: "auto" escalates from fetch to browser and bills only the rung that worked (price table). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.
• Extract structured data with a CSS/XPath selector map (extract) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (autoparse).
• Drive the page before capture with 8 typed browser actions: click, fill, wait_for, wait_for_navigation, scroll (including infinite scroll), select, evaluate and screenshot.
• Return screenshots and PDFs the model can see: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as application/pdf resource blocks (up to 10 MB).
• Batch scrape thousands of URLs as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open so you can append URLs as you discover them.
• Keep a logged-in browser with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.
• Answer "what did that cost?" from usage and request-log tools, and read the Spicrawl docs from inside the agent.

Use cases

• Scrape docs into a RAG knowledge base. Read documentation pages as clean Markdown with spicrawl_scrape (main_content_only is on by default for Markdown), or queue up to 10,000 URLs in one spicrawl_batch_submit job. Spicrawl returns the Markdown; chunking and embedding are yours. See the knowledge-base recipe.
• Monitor prices. Pull a product's title, price and availability with an extract selector map or autoparse, and set cache: false for a fresh reading. A plain-fetch check costs 1 credit, a rendered one 3. Run it on your own schedule. See price monitoring.
• Extract job postings. Collect listings from job boards, including infinite-scroll pages (render: true with a scroll action), as structured fields. See job postings.
• Read JavaScript-heavy pages. When a plain fetch returns an empty shell, call spicrawl_scrape with render: true (3 credits) so the page builds itself first.

Check each site's terms and robots.txt before you scrape it.

Which tool to use

• If you need one page's content or data now, use spicrawl_scrape.
• If the page is blank or a skeleton without JavaScript, call spicrawl_scrape again with render: true.
• If you know the page's structure, use extract with CSS selectors; if you don't, try autoparse.
• If you have more than about 20 URLs, use spicrawl_batch_submit, then spicrawl_batch_status and spicrawl_batch_results.
• If you discover URLs as you go, submit with open: true, append with spicrawl_batch_add_items and finish with spicrawl_batch_close.
• If you must log in first, use spicrawl_session_create and pass its id as session_id to spicrawl_scrape.
• If a call failed and you need to know why, use spicrawl_request_get with the request id, or spicrawl_requests_list with only_errors.
• If you are unsure of a parameter or an error code, use spicrawl_docs_search before guessing.

Tools

25 tools, grouped by task. The hosted endpoint lists 24 until remote browsers launch: its keyless tools/list and the server card leave out spicrawl_browser_connect_url (Coming soon), while a session opened with your key and the stdio package (npx -y @spicrawl/mcp) list all 25. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a title and explicit readOnlyHint, destructiveHint and openWorldHint annotations, so a client knows which calls to confirm: spicrawl_scrape is not read-only (its method and actions can submit forms), and spicrawl_batch_cancel, spicrawl_session_release and spicrawl_session_delete are destructive. Every tool also declares an outputSchema and returns matching structuredContent beside the same text it always returned, so a client can read fields (credits, status, next_cursor, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (spicrawl_docs_read, spicrawl_batch_task_content) puts it under content in structuredContent. Full argument reference: docs.spicrawl.com/agents/mcp.

Scrape a website to Markdown

Batch scrape many URLs

Browser sessions

Usage and request history

Docs

Coming soon

spicrawl_browser_connect_url will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from any CDP client; remote browsers are not available yet. It is the 25th tool and the only one the hosted keyless listing and the server card leave out (24 tools there). The schemas also accept ai_extract, stealth, extract_preset, premium_proxy, proxy_country and sticky_key ahead of launch. Agents should not send them yet. To list only what works today, set SPICRAWL_MCP_HIDE_UNAVAILABLE=1 (see Environment variables).

Example prompts

• "Scrape https://example.com/pricing as markdown and summarise the plans."
• "Get the product name, price and stock status from this URL." (spicrawl_scrape with extract or autoparse)
• "Take a full-page screenshot of https://example.com after the cookie banner is gone." (render, actions, screenshot_fullpage)
• "Scrape these 500 URLs and give me each page title." (spicrawl_batch_submit, then spicrawl_batch_results)
• "Why did my last scrape fail?" (spicrawl_requests_list with only_errors, then spicrawl_request_get)
• "How many credits have I used this month?" (spicrawl_usage_summary)

Set up your MCP client

Put your key in SPICRAWL_API_KEY and reference it from the config where the client allows, so the key never lands in a committed file.

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

claude plugin marketplace add OfficialSpicrawl/agent-plugins
claude plugin install spicrawl@spicrawl-plugins

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

{
  "inputs": [
    { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
  ],
  "servers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
    }
  }
}

devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"

{
  "mcpServers": {
    "spicrawl": {
      "serverUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"

{
  "mcpServers": {
    "spicrawl": {
      "httpUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
    }
  }
}

codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

[mcp_servers.spicrawl]
url = "https://mcp.spicrawl.com/mcp"
bearer_token_env_var = "SPICRAWL_API_KEY"

[mcp_servers.spicrawl]
command = "npx"
args = ["-y", "@spicrawl/mcp"]
env = { SPICRAWL_API_KEY = "spicrawl_live_..." }

Environment variables

With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at <SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs.

Self-host the MCP server over HTTP

The hosted endpoint runs this package's Streamable HTTP server. To run your own:

SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
SPICRAWL_BASE_URL=https://api.spicrawl.com \
npx -y @spicrawl/mcp --http

• SPICRAWL_BASE_URL defaults to http://127.0.0.1:8080 in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it to https://api.spicrawl.com to front the public API.
• MCP is served at POST/GET/DELETE /mcp; liveness at GET /healthz (no auth).
• GET /.well-known/mcp/server-card.json (no auth, Cache-Control: public, max-age=3600) is a static server card for directories such as Smithery that cannot scan /mcp without a key: serverInfo, the transport, bearer auth and the tool list. The tools are read from the same definitions as tools/list, as if SPICRAWL_MCP_HIDE_UNAVAILABLE were on: 24 tools, without spicrawl_browser_connect_url. A session opened with a key lists that tool too, unless the variable is set.
• Without any key, /mcp answers only discovery: initialize, ping, the initialized notification and tools/list (the same 24-tool list as the server card; no Mcp-Session-Id is issued and nothing is sent upstream). Any other request, a tools/call included, is 401, as is a request that sends a malformed key or names a session. A session-less GET is 405.
• Each caller sends their own key as Authorization: Bearer spicrawl_live_..., or, for a gateway that cannot send the Bearer prefix, as a raw Authorization: spicrawl_live_..., x-api-key or api-key header (see "Smithery and other gateways" above; never a query parameter). The key is checked against the API before a session opens (401 if refused, 503 with Retry-After if the API cannot be reached), and a session is pinned to the key that opened it.
• The server speaks plain HTTP. Put it behind TLS before sending keys across a network.

ChatGPT endpoint (OAuth)

The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless SPICRAWL_CHATGPT_ENABLED is on, and /mcp is unchanged either way.

Authentication takes OAuth access tokens (spicrawl_oat_…) only:

• No token: 401 with WWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch".
• A bad, inactive, expired or wrong-audience token, or an API key (spicrawl_live_…/spicrawl_test_…, never accepted here): the same 401 plus error="invalid_token", error_description="…".
• A tool call the token's scopes do not cover: 403 with error="insufficient_scope". spicrawl_scrape needs scrape, the batch tools need batch, the docs tools need only a valid token.
• The authorization server unreachable or refusing this server's secret: 503 with Retry-After.

Each token is checked with POST <issuer>/api/oauth/introspect (Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>, form body token=…). Only active: true with aud equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its exp. Tools run upstream under the api_key the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose _meta["mcp/www_authenticate"] carries a fresh invalid_token challenge, and the token is dropped from the cache.

The endpoint lists six tools, each with explicit readOnlyHint/destructiveHint/openWorldHint, a title, and securitySchemes: [{type: "oauth2", scopes: [...]}] (also in _meta.securitySchemes):

spicrawl_scrape is read-only here, unlike on /mcp: this endpoint only sends a GET, with no method, body, headers or actions. spicrawl_batch_submit is not read-only, because it creates a job.

spicrawl_batch_results fills in the content of a succeeded item that the results page lists without it, which happens while the job is still finishing, from GET /v1/batch/{id}/tasks/{seq}/content. It fetches at most 25 items and 2 MiB per page. An item that can't be read yet comes back unavailable, and one past the size cap omitted.

Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.

Installed globally (npm install -g @spicrawl/mcp), the command is spicrawl-mcp (alias spicrawl-mcp-server), with --http, --version and --help.

FAQ

Which MCP server scrapes JavaScript sites?

The Spicrawl MCP server does. Call spicrawl_scrape with render: true (3 credits) and the page loads in a browser, runs its scripts and comes back as finished Markdown, text, HTML or JSON. Add wait_for with a CSS selector for content that appears late, or set mode: "auto" to start with a plain fetch and move up to a stronger rendering only when needed. Spicrawl picks the rendering for each request and bills the tier that served it (fetch, js or browser, reported as engine in the JSON result); there is no argument to choose it.

How do I give Claude web access?

Connect the Spicrawl MCP server to Claude with an API key. In Claude Code:

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Claude can then read any URL you give it by calling spicrawl_scrape. Spicrawl reads pages by URL; it is not a search engine. For Claude Desktop, see the install answer below.

Is there a free tier?

Yes. During the beta every organization is on one plan of 1,000 credits per month. A plain fetch costs 1 credit, JavaScript rendering 3, and full-browser rendering (screenshots, PDFs, actions) 8. Failed requests cost 0, and a cache hit is billed at the same price as the fetch that stored it. When the month's credits run out, calls fail with ERR::LIMIT::QUOTA_EXCEEDED until it resets. See Credits.

Does it handle bot-protected pages?

Only with your own proxy. Pass it in proxy (an http:// URL; residential proxies work best). When a site answers with a Cloudflare managed challenge, Spicrawl detects it, moves the request to full-browser rendering on the same proxy exit and returns the real page, for 8 credits. If every step is challenged you get ERR::UPSTREAM::CHALLENGE and pay 0. Success is likely, not guaranteed, and Spicrawl does not solve CAPTCHAs. On the default shared pool a challenged page returns that error. See Scrape Cloudflare-protected pages.

How do I install it in Cursor or Claude Desktop?

Cursor: click Add to Cursor above, or add this to .cursor/mcp.json and start Cursor with SPICRAWL_API_KEY set:

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

Claude Desktop starts local (stdio) servers only, so run the package. Add this to claude_desktop_config.json and restart the app:

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

Get a key at app.spicrawl.com. Steps for Claude Code, VS Code, Windsurf, Gemini CLI and Codex are under Set up your MCP client.

Can it crawl a whole website?

There is no single crawl tool. Scrape a page with links: true to get its links, then feed them to an open batch job (open: true and spicrawl_batch_add_items).

Other questions

Is there a hosted Spicrawl MCP server?

Yes: https://mcp.spicrawl.com/mcp, MCP Streamable HTTP, authenticated with Authorization: Bearer <API key>, so there is nothing to install.

Does it support OAuth?

Not on /mcp: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a 401. The separate ChatGPT endpoint at /chatgpt/mcp takes OAuth access tokens only, when the operator turns it on.

Should I use the MCP server, the SDK or the CLI?

Use the MCP server when an AI agent should call Spicrawl as tools. Use @spicrawl/sdk in your own TypeScript or JavaScript code, @spicrawl/cli in a terminal or shell script, and the REST API from any other language.

Troubleshooting

curl -i https://mcp.spicrawl.com/mcp without a key answers 401: the endpoint is reachable.

Related packages

• @spicrawl/sdk: the official TypeScript SDK for the Spicrawl API.
• @spicrawl/cli: the Spicrawl command-line interface.
• OfficialSpicrawl/agent-plugins: plugins that bundle this server and the Spicrawl skill for Claude Code, Codex, Cursor, Gemini CLI and more.

Links

• Documentation: docs.spicrawl.com/agents/mcp
• Docs for LLMs: docs.spicrawl.com/llms.txt
• Dashboard and API keys: app.spicrawl.com
• Issues: github.com/OfficialSpicrawl/mcp/issues
• Security: SECURITY.md. Never commit an API key; revoke a leaked one at app.spicrawl.com.

Publishing to the MCP Registry

server.json lists both the npm/stdio package and the hosted Streamable HTTP endpoint under io.github.OfficialSpicrawl/mcp. API keys are requested from the person installing the server; no key belongs in the manifest.

The existing release workflow publishes to the MCP Registry after its npm job succeeds, on pushes to main or manual runs from main. Registry authentication uses GitHub OIDC (id-token: write), so it needs no new secret, device login or DNS record. The npm release keeps using the existing NPM_TOKEN secret.

For a release, run npm version --no-git-tag-version X.Y.Z, update the changelog, and submit the changes to main. The npm version hook synchronizes both versions in server.json. If you edit package.json by hand, run npm run registry:sync. CI checks the manifest against its official $schema with Ajv and rejects name or version drift before npm publication. To run that check locally:

npm ci
npm run registry:validate
# Offline: pass a downloaded copy of the exact server.json $schema URL.
npm run registry:validate -- /path/to/server.schema.json

The registry job checks that the published npm package has the matching mcpName before publishing. An existing registry version is skipped; lookup failures stop the job. To retry a registry failure or list the already-published current npm version, run the release workflow on main again. Use a new version for metadata changes to an existing listing. The publisher binary is pinned and its download is checked against the release checksums; update its version in release.yml when upgrading the CLI.

Verify publication at https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.OfficialSpicrawl/mcp. The workflow must be merged into OfficialSpicrawl/mcp before OIDC can publish that namespace. Fork pull requests validate and test, but never run the registry job.

Official references: publishing quickstart, GitHub Actions, and remote servers.

License

Apache-2.0