io.github.hshintelligence/agent-scrape

Official

by hshintelligence · TypeScript

AgentScrape

Pay-per-call web scraping for AI agents via x402 on Base USDC. Six tools, no signup.

Pay-per-call web scraping service for AI agents with Base USDC payment integration.

Topics

agent-nativeagentic-commerceai-agentsbase-networkbrowser-automationcloudflare-workerscoinbasehonomcpmcp-servermicropaymentsmodel-context-protocolpay-per-calltypescriptusdcweb-scrapingx402x402-protocol

README.md13 KBView on GitHub

AgentScrape

Pay-per-call web scraping for AI agents — no signup, no API keys, just USDC.

Live: agent-scrape.healingsunhaven.workers.dev

AI agents discover AgentScrape, pay per call in USDC on Base, and get clean structured data back — no accounts, no API keys, no human in the loop.

Two protocols, one service:

HTTP API with x402 v2 payment gate — for agents using raw HTTP
MCP server (Streamable HTTP transport) — for Claude Desktop, Cursor, Continue.dev, and any MCP-compatible framework

Why this exists

Traditional scraping APIs assume a human:

Sign up with email
Add a payment method
Manage an API key
Commit to a monthly subscription

AI agents can't do any of that. They have wallets. They have stablecoins. They need infrastructure built for them.

AgentScrape speaks two open standards:

x402 — HTTP-native payment protocol. The server returns 402 with payment requirements; the agent signs a USDC transfer; the facilitator settles on-chain; the data flows back. End-to-end in under 2 seconds.
MCP — the standard agent tool interface. Agents browse, discover, and invoke tools without per-vendor SDKs.

No signup. No API keys. No subscription. Just USDC.

Tools

All tools cost $0.001 USDC during the mainnet validation window. Pricing ramps after stability is proven (see Pricing).

Tool	Description
`scrape_webpage`	Scrape any URL to markdown, HTML, text, or JSON
`extract_structured_data`	AI-powered structured extraction with natural-language prompts and JSON schemas (Groq + Llama 4 Scout)
`screenshot_webpage`	PNG screenshot with viewport control (desktop/mobile/tablet, optional full-page)
`extract_metadata`	Title, description, Open Graph, Twitter cards, JSON-LD, canonical URL
`create_browser_session`	Stateful browser session with cookie + localStorage persistence
`run_workflow`	Multi-step atomic execution: navigate, click, type, wait, scroll, screenshot, extract, evaluate

Quick start — HTTP API

bash

# Free service discovery (no payment)
curl https://agent-scrape.healingsunhaven.workers.dev/

# Paid call — without payment, returns 402 with payment requirements
curl -X POST https://agent-scrape.healingsunhaven.workers.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'

# Free tier — first 10 calls per wallet, send your wallet in the X-402-Payer header
curl -X POST https://agent-scrape.healingsunhaven.workers.dev/scrape \
  -H "Content-Type: application/json" \
  -H "X-402-Payer: 0xYourBaseWalletAddress" \
  -d '{"url": "https://example.com", "format": "markdown"}'

Quick start — MCP

Add to any MCP-compatible client via Streamable HTTP:

json

{
  "mcpServers": {
    "agent-scrape": {
      "url": "https://agent-scrape.healingsunhaven.workers.dev/mcp",
      "transport": "streamable-http"
    }
  }
}

The x402 payment metadata is broadcast inside the MCP tools/list response. Your client handles the payment handshake automatically when configured with an x402-compatible wallet.

Discovery

For machine-driven discovery, four standard endpoints are exposed (free, no auth):

Endpoint	Purpose
`/`	Service profile JSON (tools, prices, network, payTo)
`/.well-known/x402` (and `.json` alias)	x402 manifest — payTo, network, facilitator, all paid routes
`/openapi.json`	OpenAPI 3.1 spec for traditional tooling
`/llms.txt`	Agent-friendly plaintext description

The 402 response itself carries the canonical payment requirements in the base64-encoded Payment-Required header, and the body includes a human-readable error with discovery links.

Architecture

code

                ┌──────────────────────────────────┐
                │      Cloudflare Workers          │
                │      (global edge, 330+ POPs)    │
                │                                  │
AI Agent ───────┼─▶  GET  /                  (free)│
(HTTP)          ├─▶  GET  /.well-known/x402  (free)│
                ├─▶  GET  /openapi.json      (free)│
                ├─▶  GET  /llms.txt          (free)│
                ├─▶  POST /scrape         ($0.001) │
                ├─▶  POST /extract        ($0.001) │
                ├─▶  POST /screenshot     ($0.001) │
                ├─▶  POST /metadata       ($0.001) │
                ├─▶  POST /workflow       ($0.001) │
                ├─▶  POST /session        ($0.001) │
                │                                  │
AI Agent ───────┼─▶  POST /mcp                     │
(MCP)           │      • initialize                │
                │      • tools/list                │
                │      • tools/call (paid)         │
                │                                  │
                └──────────────┬───────────────────┘
                               │
              ┌────────────────┴─────────────────┐
              │                                  │
   Coinbase CDP facilitator       Browser Rendering API
   (api.cdp.coinbase.com/         Groq API (Llama 4 Scout)
    platform/v2/x402)             KV (sessions, free tier)
   Ed25519 JWT auth
   Settles USDC on Base
              │
              ▼
   payTo wallet 0x3F33...9E9a

Stack:

Runtime: Cloudflare Workers (V8 isolates, global edge)
HTTP framework: Hono + @x402/hono v2 payment middleware
MCP framework: @modelcontextprotocol/sdk + agents/mcp with withX402 payment binding
Bazaar discovery: @x402/extensions — every paid route declares input schema + output example
Browser: Cloudflare Browser Rendering API + @cloudflare/puppeteer
AI extraction: Groq + Llama 4 Scout (17B, 16E)
Storage: Cloudflare KV (5-min response cache, browser sessions, free-tier counters)
Network: Base mainnet (eip155:8453)
Asset: Native USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913)
Facilitator: Coinbase CDP (api.cdp.coinbase.com/platform/v2/x402) with Ed25519 JWT auth

Free tier

Every wallet address gets 10 free calls per 30 days on the HTTP API. Pass your Base wallet in the X-402-Payer header. After 10 calls, payment via x402 is required.

Free tier is HTTP-only. MCP calls go through the standard x402 payment flow.

Pricing

Launch window (first 48h on mainnet): $0.001 flat per tool call.

After validation, pricing ramps to:

Tool	Production price
`scrape_webpage`	$0.003
`extract_structured_data`	$0.005
`screenshot_webpage`	$0.003
`extract_metadata`	$0.002
`run_workflow`	$0.008
`create_browser_session`	$0.001

Bulk discounts (20% at 1K calls/month, 40% at 10K) are planned.

Why x402 + MCP

Traditional API	x402 + MCP
Sign up at scraperapi.com	Discover via MCP `initialize`
Add credit card	Wallet holds USDC
Get API key	No keys
$49/month minimum	$0.001 per call
Manage rate limits	Pay-per-use, no limits
Human in the loop	Agent fully autonomous

Verifying the deployment

bash

# Service identity
curl https://agent-scrape.healingsunhaven.workers.dev/ | jq .

# x402 manifest
curl https://agent-scrape.healingsunhaven.workers.dev/.well-known/x402 | jq .

# 402 challenge with x402 v2 headers
curl -i -X POST https://agent-scrape.healingsunhaven.workers.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'
# → HTTP 402, Payment-Required header (base64 JSON), Cache-Control: private, no-store

# CORS preflight (browser-based agents)
curl -i -X OPTIONS https://agent-scrape.healingsunhaven.workers.dev/scrape \
  -H "Origin: https://example.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: Content-Type,X-Payment"
# → HTTP 204, allow-headers includes X-Payment and Payment-Signature

Roadmap

Q3 2026:

AgentParse — document/PDF/OCR-to-markdown service (same x402 + MCP pattern)
AgentSearch — multi-backend web search aggregation
LangChain integration package (langchain-agent-scrape)

Q4 2026:

Subscription bundles (1000 scrapes for $2.50)
Multi-asset support (USDT on Base, USDC on Arbitrum)
Webhook + async response option for long-running workflows

Run locally

Requires Node 22+, a Cloudflare account with Workers + Browser Rendering enabled, a Groq API key, and a Coinbase CDP API key.

bash

git clone https://github.com/hshintelligence/agent-scrape.git
cd agent-scrape
npm install

# Set secrets
npx wrangler secret put GROQ_API_KEY
npx wrangler secret put CDP_API_KEY_ID
npx wrangler secret put CDP_API_KEY_SECRET

# Local dev
npx wrangler dev

# Deploy
npx wrangler deploy

To use your own wallet as payTo, edit the PAY_TO constant in src/index.ts.

License

MIT — see LICENSE.

Contact

Built by HSH Intelligence, operated by Healing Sun Haven LLC (Wyoming, USA).

Service: https://agent-scrape.healingsunhaven.workers.dev
Repo: https://github.com/hshintelligence/agent-scrape
General inquiries: sales@healingsunhaven.com

Pay-per-call web scraping for AI agents — no signup, no API keys, just USDC.

Using AgentScrape from ElizaOS

ElizaOS supports remote MCP servers natively via the @elizaos/plugin-mcp plugin. AgentScrape works out of the box — no custom integration needed.

Install the MCP plugin

bash

bun add @elizaos/plugin-mcp
# or
npm install @elizaos/plugin-mcp

Configure your character to use AgentScrape

In your ElizaOS character JSON, add the plugin and point the MCP server config at AgentScrape's Streamable HTTP endpoint:

json

{
  "name": "ResearchAgent",
  "plugins": ["@elizaos/plugin-mcp"],
  "settings": {
    "mcp": {
      "servers": {
        "agentscrape": {
          "type": "streamable-http",
          "name": "AgentScrape",
          "url": "https://agent-scrape.healingsunhaven.workers.dev/mcp",
          "timeout": 60
        }
      }
    }
  }
}

Available tools (auto-discovered by ElizaOS)

Once configured, the agent automatically discovers all six AgentScrape tools:

Tool	Cost	Purpose
`scrape_webpage`	$0.003 USDC	Markdown/HTML/text/JSON scrape
`extract_structured_data`	$0.005 USDC	AI extraction via Groq + Llama 4 Scout
`screenshot_webpage`	$0.003 USDC	PNG screenshot with viewport control
`extract_metadata`	$0.002 USDC	Title, OG, Twitter, JSON-LD
`create_browser_session`	$0.001 USDC	Stateful browser session
`run_workflow`	$0.008 USDC	Multi-step atomic workflow up to 20 steps

Free tier

The first 10 calls per wallet in the first 30 days are free. After that, calls are paid in USDC on Base mainnet (eip155:8453) via the x402 v2 payment protocol. To pay, supply the X-PAYMENT-RESPONSE header with an x402 payment receipt on retry — the standard x402 flow.

Example prompt

Once the plugin is wired up, your ElizaOS agent can use AgentScrape transparently:

User: "Scrape https://www.x402.org and give me a two-sentence summary."

ElizaOS routes the request through @elizaos/plugin-mcp, which calls AgentScrape's scrape_webpage tool, and the agent answers based on the live page content.

Resources

AgentScrape Worker: https://agent-scrape.healingsunhaven.workers.dev
x402 manifest: https://agent-scrape.healingsunhaven.workers.dev/.well-known/x402.json
A2A Agent Card: https://agent-scrape.healingsunhaven.workers.dev/.well-known/agent.json
Source: https://github.com/hshintelligence/agent-scrape

Install

claude_desktop_config.json

{
  "mcpServers": {
    "agent-scrape": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://agent-scrape.healingsunhaven.workers.dev/mcp"
      ]
    }
  }
}

Desktop config is stdio-only; this bridges via mcp-remote. Native remote: Settings > Connectors.