com.scrapingbee/mcp (Model Context Protocol) Server

The com.scrapingbee/mcp MCP server provides capabilities to scrape web pages and to search major platforms—Google, Amazon, Walmart, and YouTube—then extract data using the ScrapingBee API. It is described as supporting both web scraping and platform-specific search with extraction.

🛠️ Key Features

Scrape any web page
Search Google, Amazon, Walmart, and YouTube
Extract data via the ScrapingBee API

🚀 Use Cases

Extract structured information from arbitrary web pages
Perform search-and-extract workflows across common shopping and video platforms
Collect data from Google, Amazon, Walmart, or YouTube results for downstream processing

⚡ Developer Benefits

Standardized access through an MCP server for scraping and extraction
Supports search + extraction flows for multiple target platforms using the same API

⚠️ Limitations

Only scraping and search/extraction via the ScrapingBee API are described
No additional tools, configuration details, or constraints are provided in the available data

ScrapingBee MCP Server

scrapingbee-mcp

A production-ready Model Context Protocol (MCP) server that enables AI agents to access structured web data using ScrapingBee as a tool.

This project demonstrates how to transform ScrapingBee into an AI-native data layer. Instead of embedding scraping logic directly inside prompts, AI agents register ScrapingBee as a tool and request structured web data on demand.

If you are building:

• AI agents that require live web access
• Autonomous research systems
• Ecommerce intelligence bots
• Market monitoring agents
• LLM workflows with tool execution
• AI-powered web scraping infrastructure

This repository provides a complete MCP integration blueprint.

Quick start

The server is hosted by ScrapingBee at https://mcp.scrapingbee.com/mcp (Streamable HTTP) — there is nothing to install. You need a ScrapingBee API key (get one here; trial credits included), sent as the header Authorization: Bearer <key>. Registry name: com.scrapingbee/mcp.

Claude Code

bash

claude mcp add --transport http scrapingbee https://mcp.scrapingbee.com/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

Codex CLI

bash

codex mcp add scrapingbee --url https://mcp.scrapingbee.com/mcp \
  --bearer-token-env-var SCRAPINGBEE_API_KEY

Cursor, VS Code and other clients with a JSON config

json

{
  "mcpServers": {
    "scrapingbee": {
      "url": "https://mcp.scrapingbee.com/mcp",
      "headers": { "Authorization": "Bearer YOUR_API_KEY" }
    }
  }
}

Also available through Glama, Smithery and the MCP Registry.

Tools

Tool	What it does
`get_page_text`	Fetch a URL and return its content as text or markdown (JavaScript rendered, blocks handled)
`get_page_html`	Fetch a URL and return the rendered HTML
`get_screenshot`	Fetch a URL and return a screenshot
`get_file`	Download a file (PDF, image, …) from a URL
`extract_page_data`	Fetch a URL and extract specific data with CSS or XPath selectors
`fast_search`	Fast web search returning clean organic results and top stories — the first choice for general searches
`get_google_search_results`	Google SERP as JSON for specialised searches: news, maps, Lens, shopping, images, AI mode
`get_amazon_search_results`	Amazon search results as JSON
`get_amazon_product_details`	Amazon product details by ASIN
`get_amazon_pricing`	Current Amazon price and seller offers by ASIN
`get_walmart_search_results`	Walmart search results as JSON
`get_walmart_product_details`	Walmart product details by ID
`get_youtube_search_results`	YouTube search results
`get_youtube_video_metadata`	Title, description, stats and more for a video
`get_youtube_video_subtitles`	Subtitles / transcript for a video
`ask_chatgpt`	Send a prompt to ChatGPT, optionally with web search
`ask_gemini`	Send a prompt to Gemini, with citations when available
`get_scrapingbee_usage`	Remaining credits and concurrency for your account

Each call consumes API credits exactly as the equivalent ScrapingBee API request would (see pricing and the API documentation). The server handles one page or query per call; for hundreds of URLs, crawling a site, writing results to files or scheduling recurring jobs, use the ScrapingBee CLI.

Relevant keywords: scrapingbee mcp
model context protocol server
mcp scraping
ai agent web scraping
mcp web scraping tool

What Is Model Context Protocol (MCP)?

Model Context Protocol (MCP) defines how AI systems interact with external tools in a structured way.

Instead of letting a language model guess how to scrape or browse the web, MCP allows it to:

Discover available tools
Call a tool using structured parameters
Receive normalized JSON responses
Inject results back into the reasoning loop

The ScrapingBee MCP server exposes web scraping capabilities as a formally registered tool that AI systems can use safely and predictably.

Why Use ScrapingBee with MCP?

AI models cannot reliably:

• Bypass anti-bot protections
• Manage proxy rotation
• Render JavaScript-heavy pages
• Handle CAPTCHA challenges
• Maintain scraping infrastructure

ScrapingBee handles all of that.

The MCP server acts as a bridge between your AI agent and ScrapingBee’s managed scraping infrastructure.

This enables true AI agent web scraping without fragile prompt-based hacks.

Architecture Overview

AI Agent
→ MCP Client
→ ScrapingBee MCP Server
→ ScrapingBee API
→ Target Website
→ Structured JSON Response
→ Injected Back Into Agent Context

The AI agent never scrapes directly.
It calls tools.
The MCP server executes them.

Installation

There is nothing to install or run: the server is operated by ScrapingBee at https://mcp.scrapingbee.com/mcp. The only setup is your API key, which every client below sends as a Authorization: Bearer header:

bash

export SCRAPINGBEE_API_KEY=your_api_key

Connecting to the ScrapingBee MCP Server

You can connect in two ways:

Via Claude Desktop (remote MCP)
Via a custom Python MCP client

Option 1: Connect to Claude Desktop

Add the ScrapingBee MCP server to your claude_desktop_config.json file:

json

{
  "mcpServers": {
    "scrapingbee": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.scrapingbee.com/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}"
      ],
      "env": {
        "AUTH_HEADER": "Bearer YOUR_API_KEY"
      }
    }
  }
}

(The AUTH_HEADER variable works around mcp-remote's handling of spaces in arguments.)

This registers ScrapingBee as an MCP tool provider inside Claude Desktop.

After restarting Claude, the tools become available automatically.

Option 2: Custom Python MCP Client

For advanced AI workflows, you can build a minimal async MCP client using Python.

Below is a complete working example.

custom_mcp_client.py

python

import os
import re
import json
import asyncio
import httpx
import google.generativeai as genai
from typing import Dict, Any, List

MCP_SERVER_URL = "https://mcp.scrapingbee.com/mcp"
SCRAPINGBEE_API_KEY = os.environ.get("SCRAPINGBEE_API_KEY")
GEMINI_API_KEY = os.environ.get("GEMINI_API_KEY")

MCP Client Implementation

python

class MCPClient:
    def __init__(self, base_url: str, api_key: str):
        self.base_url = base_url
        self.session_id = None
        self.http_client = httpx.AsyncClient(timeout=300)
        self.headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Accept": "application/json, text/event-stream",
            "User-Agent": "Minimal-MCP-Client/1.0",
        }

    async def _send_request(self, payload: Dict[str, Any]):
        headers = self.headers.copy()
        if self.session_id:
            headers["Mcp-Session-Id"] = self.session_id

        response = await self.http_client.post(self.base_url, json=payload, headers=headers)
        response.raise_for_status()

        text = (await response.aread()).decode("utf-8")

        if "data:" in text:
            match = re.search(r"data:\s*({.*})", text, re.DOTALL)
            if match:
                return json.loads(match.group(1))

        if response.status_code in [202, 204] or not text:
            return {"status": "ok"}

        return json.loads(text)

    async def initialize(self):
        init_payload = {
            "jsonrpc": "2.0",
            "id": "1",
            "method": "initialize",
            "params": {
                "protocolVersion": "2024-11-05",
                "capabilities": {},
                "clientInfo": {"name": "MinimalClient"},
            },
        }

        async with self.http_client.stream(
            "POST", self.base_url, json=init_payload, headers=self.headers
        ) as response:
            self.session_id = response.headers.get("Mcp-Session-Id")

        notify_payload = {
            "jsonrpc": "2.0",
            "method": "notifications/initialized",
            "params": {}
        }

        await self._send_request(notify_payload)

        list_payload = {
            "jsonrpc": "2.0",
            "id": "2",
            "method": "tools/list",
            "params": {}
        }

        return await self._send_request(list_payload)

    async def call_tool(self, tool_name: str, arguments: Dict[str, Any]):
        payload = {
            "jsonrpc": "2.0",
            "id": "3",
            "method": "tools/call",
            "params": {
                "name": tool_name,
                "arguments": arguments
            }
        }

        return await self._send_request(payload)

Converting MCP Tools for Gemini

python

def format_tools_for_gemini(tools: List[Dict[str, Any]]):
    type_map = {
        "string": genai.protos.Type.STRING,
        "integer": genai.protos.Type.INTEGER,
        "number": genai.protos.Type.NUMBER,
        "boolean": genai.protos.Type.BOOLEAN,
    }

    gemini_tools = []

    for tool in tools:
        properties = {}
        input_schema = tool.get("inputSchema", {})

        if "properties" in input_schema:
            for name, prop in input_schema["properties"].items():
                properties[name] = genai.protos.Schema(
                    type=type_map.get(prop.get("type", "string")),
                    description=prop.get("description", "")
                )

        func = genai.protos.FunctionDeclaration(
            name=tool["name"],
            description=tool["description"],
            parameters=genai.protos.Schema(
                type=genai.protos.Type.OBJECT,
                properties=properties,
                required=input_schema.get("required", [])
            ),
        )

        gemini_tools.append(
            genai.protos.Tool(function_declarations=[func])
        )

    return gemini_tools

Full AI → MCP → Tool Execution Flow

python

async def main():
    mcp_client = MCPClient(MCP_SERVER_URL, SCRAPINGBEE_API_KEY)
    tools_response = await mcp_client.initialize()

    genai.configure(api_key=GEMINI_API_KEY)
    gemini_tools = format_tools_for_gemini(tools_response["result"]["tools"])

    model = genai.GenerativeModel(
        model_name="gemini-2.5-pro",
        tools=gemini_tools
    )

    user_prompt = input("Ask the AI to perform a task:\n> ")

    response = model.generate_content(user_prompt)
    function_call = response.candidates[0].content.parts[0].function_call

    tool_name = function_call.name
    tool_args = dict(function_call.args)

    result = await mcp_client.call_tool(tool_name, tool_args)

    print(json.dumps(result, indent=2))

What This Enables

With this architecture you can:

• Build AI agents that reason about available tools
• Allow LLMs to dynamically choose scraping actions
• Execute structured web data extraction
• Inject real-world data into model context
• Create scalable AI-native web scraping systems

Conclusion

The ScrapingBee MCP Server transforms web scraping into a structured, tool-driven data layer for AI systems.

Instead of brittle prompt scraping, you get:

• Model Context Protocol compliance
• JSON-RPC structured execution
• AI agent web scraping
• Scalable tool orchestration

This is AI-native web data access.

To start building AI-powered agents today, Get your free API key and connect to the ScrapingBee MCP server in minutes.

query	string	required	The search query to perform.
add_html	boolean	optional	Whether to return the HTML along with the search results.
category_id	string	optional	The category ID to use for the search.
country	string	optional	The country code for localization (e.g., us, uk, de). Do not combine with a matching domain (e.g., country=fr&domain=fr).
currency	string	optional	The currency code (ISO 4217) to display results (e.g., USD, GBP, EUR).
domain	string	optional	The Amazon domain to use for the search (e.g., com, co.uk, de).
language	string	optional	The language code to display results (e.g., en, fr, de).
light_request	boolean	optional	Whether to use a light request or not.
merchant_id	string	optional	The merchant ID to use for the search.
pages	integer	optional	The number of pages to scrape.
sort_by	string	optional	The sort order to use for the search (most_recent, price_low_to_high, price_high_to_low, featured, average_review, bestsellers).
start_page	integer	optional	The page number to start scraping from.
zip_code	string	optional	The ZIP code to use for delivery localization.
autoselect_variant	boolean	optional	Automatically select the default/most-popular variant.
screenshot	boolean	optional	Force a browser screenshot (returns base64 image).
tag	string	optional	Response-header label only.

asin	string	required	Required 10-character ASIN from an Amazon product URL.
light_request	boolean	optional	Use a light request (default True) or browser-rendered (False).
domain	string	optional	Amazon domain (default com).
country	string	optional	ISO country code for localization. Do not combine with a matching domain (e.g., country=fr&domain=fr).
zip_code	string	optional	ZIP code for delivery localization.
language	string	optional	ISO language code.
currency	string	optional	ISO 4217 display currency.
add_html	boolean	optional	Include page HTML in the response.
tag	string	optional	Response-header label only.

prompt	string	required	The prompt to send to ChatGPT.
search	boolean	optional	Whether to enable web search capability for the prompt.
add_html	boolean	optional	Whether to include the full HTML of the page in the results.
country_code	string	optional	ISO country code the request should originate from (affects web results when search is enabled).
tag	string	optional	Response-header label only.

url	string	required	The URL of the page to scrape.
return_page_markdown	boolean	optional	Whether to return the page content in Markdown format.
return_page_text	boolean	optional	Whether to return the page content in Text format.
auto_mode	boolean	optional	On by default — ScrapingBee automatically picks the cheapest configuration that successfully fetches the page, escalating proxy strength only as needed (you are charged only for the winning config). To choose a configuration yourself instead, use one of: auto_mode=False for the classic tier (no proxy, cheapest), premium_proxy=True, or stealth_proxy=True — the two proxy flags override auto_mode on their own, so auto_mode=False is only needed for the classic tier. Setting render_js also switches to manual.
max_cost	integer	optional	Optional credit ceiling for auto_mode (integer >= 1). The request will not escalate to a configuration costing more than this many credits. 0 means no ceiling. Ignored unless auto_mode is True.
premium_proxy	boolean	optional	Manual option: use a premium proxy (middle tier of classic/premium/stealth). Setting this disables auto_mode.
stealth_proxy	boolean	optional	Manual option: use a stealth proxy (strongest tier of classic/premium/stealth). Setting this disables auto_mode.
custom_google	boolean	optional	Set to True if the url is a google domain in the following format: <subdomain>.google.<top-level-domain> (Example: translate.google.com, mail.google.com, www.google.co.in etc)
country_code	string	optional	ISO 3166-1 alpha-2 country code (e.g. "fr", "us") to route the request through an IP in that country. Only takes effect when premium_proxy or stealth_proxy is True.
render_js	any	optional	Whether to use headless-browser rendering. Omit to use the API default (true). Setting this explicitly switches the request to manual configuration (disables auto_mode).
wait	integer	optional	Milliseconds to wait after page load (0–35000).
wait_for	string	optional	CSS or XPath selector to wait for before returning.
wait_browser	string	optional	domcontentloaded (default), load, networkidle0, networkidle2.
block_resources	boolean	optional	Block images/CSS to speed up rendering.
window_width	integer	optional	Viewport width (default 1920).
window_height	integer	optional	Viewport height (default 1080).
session_id	any	optional	Integer 0–10000000 to reuse an IP for up to 5 minutes.
cookies	string	optional	Semicolon-separated cookies to send to the target.
timeout	integer	optional	Request timeout in milliseconds (1000–140000, default 140000).
device	string	optional	desktop (default) or mobile.
js_scenario	string	optional	Stringified JSON object of browser interaction instructions.
ai_query	string	optional	Natural-language question for AI extraction.
ai_selector	string	optional	CSS selector to focus AI extraction on part of the page.

url	string	required	The URL of the file to fetch.
premium_proxy	boolean	optional	Whether to use a premium proxy for the request. Use it only if you are getting blocked by the website.
custom_google	boolean	optional	Set to True if the url is a google domain in the following format: <subdomain>.google.<top-level-domain> (Example: translate.google.com, mail.google.com, www.google.co.in etc)
country_code	string	optional	ISO 3166-1 alpha-2 country code (e.g. "fr", "us") to route the request through an IP in that country. Only takes effect when premium_proxy is True.
render_js	boolean	optional	Whether to use headless-browser rendering (default False).
session_id	any	optional	Integer 0–10000000 to reuse an IP for up to 5 minutes.
cookies	string	optional	Semicolon-separated cookies to send to the target.
timeout	integer	optional	Request timeout in milliseconds (1000–140000, default 140000).
device	string	optional	desktop (default) or mobile.

product_id	string	required	The unique identifier for the Walmart product.
add_html	boolean	optional	Whether to return the HTML along with the product details.
delivery_zip	string	optional	The zip code to use for delivery localization.
device	string	optional	The device to use for the request (desktop, mobile, tablet).
domain	string	optional	The domain to use for the search (Example: com, ca, com.mx, etc.) for localization.
light_request	boolean	optional	Whether to use a light request or not.
store_id	string	optional	Specific Walmart store ID for localization.
screenshot	boolean	optional	Force a browser screenshot (returns base64 image).
tag	string	optional	Response-header label only.

video_id	string	required	The unique YouTube video ID.
language	string	optional	Language code (e.g., 'en', 'es', 'fr'). Default is 'en'.
subtitle_origin	string	optional	'auto_generated' (default) or 'uploader_provided'.
tag	string	optional	Response-header label only.

search	string	required	The search query.
upload_date	string	optional	Filter by date: 'today', 'last_hour', 'this_week', 'this_month', 'this_year'.
result_type	string	optional	Filter by type: 'video', 'channel', 'playlist', 'movie'.
duration	string	optional	Filter by duration: '<4' (short), '4-20' (medium), '>20' (long).
sort_by	string	optional	Sort order: 'relevance' (default), 'rating', 'view_count', 'upload_date'.
hd	boolean	optional	Return only HD videos.
is_4k	boolean	optional	Return only 4K videos.
subtitles	boolean	optional	Return only videos with subtitles/captions.
creative_commons	boolean	optional	Return only videos with Creative Commons license.
live	boolean	optional	Return only live streams.
is_360	boolean	optional	Return only 360-degree videos.
is_3d	boolean	optional	Return only 3D videos.
hdr	boolean	optional	Return only HDR videos.
location	boolean	optional	Return only videos with location metadata.
purchased	boolean	optional	Return only purchased movies.
vr180	boolean	optional	Return only VR180 videos.
tag	string	optional	Response-header label only.

com.scrapingbee/mcp (Model Context Protocol) Server

🛠️ Key Features

🚀 Use Cases

⚡ Developer Benefits

⚠️ Limitations

Topics

get_amazon_search_results

get_amazon_product_details

get_amazon_pricing

ask_chatgpt

fast_search

ask_gemini

get_google_search_results

get_page_text

get_page_html

extract_page_data

get_screenshot

get_file

get_scrapingbee_usage

get_walmart_search_results

get_walmart_product_details

get_youtube_search_results

get_youtube_video_metadata

get_youtube_video_subtitles

ScrapingBee MCP Server

Quick start

Tools

What Is Model Context Protocol (MCP)?

Why Use ScrapingBee with MCP?

Architecture Overview

Installation

Connecting to the ScrapingBee MCP Server

Option 1: Connect to Claude Desktop

Option 2: Custom Python MCP Client

custom_mcp_client.py

MCP Client Implementation

Converting MCP Tools for Gemini

Full AI → MCP → Tool Execution Flow

What This Enables

Conclusion

Install

Remote endpoint

Related servers