io.github.JakeLiuMe/webpeel

The Problem

Every AI agent that touches the web rebuilds the same brittle stack: HTTP fetch → headless browser → anti-bot bypass → HTML cleanup → markdown conversion → token budgeting. Each layer fails differently. Sites change. Cloudflare rotates challenges. Your agent gets empty strings at 2 AM and your pipeline breaks.

WebPeel replaces that entire stack with one function call. It handles engine selection, anti-bot escalation, domain-specific extraction, and token optimization so your agent gets clean, structured data every time — without managing browsers, proxies, or parsing logic.

Quick Start

# Zero-install — just run it
npx webpeel "https://example.com"

# Search the web
npx webpeel search "latest AI agent frameworks"

# Crawl an entire site
npx webpeel crawl docs.example.com --max-pages 50

# Screenshot any page
npx webpeel screenshot "https://stripe.com/pricing" --full-page

# Ask a question about any page
npx webpeel ask "https://arxiv.org/abs/2401.00001" "What is the main contribution?"

Or install globally:

npm install -g webpeel

Use as a library:

import { peel } from 'webpeel';

const result = await peel('https://news.ycombinator.com');
console.log(result.markdown);   // Clean markdown, ready for your LLM
console.log(result.metadata);   // Title, tokens saved, timing, etc.

Use via API:

curl "https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing" \
  -H "Authorization: Bearer $WEBPEEL_API_KEY"

{
  "url": "https://stripe.com/pricing",
  "markdown": "# Stripe Pricing\n\n**Integrated per-transaction fees**...",
  "metadata": {
    "title": "Pricing & Fees | Stripe",
    "tokens": 420,
    "tokensOriginal": 8200,
    "savingsPct": 94.9
  }
}

Get your free API key → · No credit card required · 500 requests/week free

Why WebPeel

🧠 55+ Domain Extractors — Not Just HTML-to-Markdown

Generic scrapers convert raw HTML to markdown and call it a day. WebPeel has purpose-built extractors for 55+ domains — Reddit, GitHub, YouTube, Amazon, ArXiv, Hacker News, Wikipedia, StackOverflow, Zillow, Polymarket, ESPN, and more. Each extractor understands the site's structure and returns clean, structured data without browser rendering.

⚡ 65–98% Token Savings

Domain extractors strip navigation, ads, sidebars, and boilerplate before content reaches your agent. Less context consumed = lower costs, faster inference, and longer agent chains.

🔄 6-Layer Engine Escalation

WebPeel doesn't just try one method — it automatically escalates through 6 engines until it gets a good result:

Simple HTTP → Domain API → Browser render → Stealth browser → Cloaked browser → Search cache fallback

No manual --render flags for most sites. WebPeel knows which sites need JavaScript, which need stealth, and which have anti-bot protection — and picks the right engine automatically.

🔌 Firecrawl-Compatible Migration Path

Already using Firecrawl-style workflows? WebPeel supports compatible /v1/scrape, /v2/scrape, /v1/crawl, /v1/search, and /v1/map endpoints, which makes migration dramatically easier than rebuilding your pipeline from scratch.

Agent-Native Integrations

MCP Server (Claude, Cursor, Windsurf, VS Code)

Give any MCP-compatible AI the ability to browse, search, and extract from the web.

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["-y", "webpeel", "mcp"],
      "env": { "WEBPEEL_API_KEY": "wp_your_key_here" }
    }
  }
}

7 MCP tools exposed: webpeel_read · webpeel_find · webpeel_see · webpeel_extract · webpeel_monitor · webpeel_act · webpeel_crawl

Full MCP setup guide →

LangChain

import { WebPeelLoader } from 'webpeel/integrations/langchain';

const loader = new WebPeelLoader({ url: 'https://example.com', render: true });
const docs = await loader.load();

LlamaIndex

import { WebPeelReader } from 'webpeel/integrations/llamaindex';

const reader = new WebPeelReader();
const docs = await reader.loadData('https://example.com');

Python SDK

pip install webpeel

from webpeel import WebPeel

wp = WebPeel(api_key="wp_...")
result = wp.fetch("https://example.com")
print(result.markdown)

Full Feature Set

Use Cases for Agent Builders

RAG pipelines — Fetch docs, articles, or entire sites as clean markdown ready for chunking, embedding, and retrieval.

Price monitoring — Track product pages across major commerce sites with structured extraction and change detection.

Competitive intel — Monitor competitor pages, pricing tables, and job boards. Visual diff screenshots catch layout changes CSS selectors would miss.

Research agents — Give Claude, Codex, Cursor, or your own agent grounded web access through the API or MCP server.

Lead enrichment — Pull company details, public links, and page structure from business sites without writing per-site parsers.

Content aggregation — Crawl and extract from communities, docs sites, and publications with domain-native extractors that understand each site's structure.

Architecture

Your Agent
    ↓
 WebPeel (npm / API / MCP)
    ↓
┌─────────────────────────────────┐
│  Engine Ranker                  │
│  HTTP → Domain API → Browser   │
│  → Stealth → Cloaked → Cache   │
├─────────────────────────────────┤
│  55+ Domain Extractors          │
│  reddit · github · youtube      │
│  amazon · arxiv · zillow · ...  │
├─────────────────────────────────┤
│  Content Pipeline               │
│  Readability → Turndown →       │
│  Token budgeting → Chunking     │
└─────────────────────────────────┘
    ↓
 Clean markdown / structured JSON

Reliability

WebPeel is built for production agent workflows, not just one-off demos.

• Automated evals in-repo — smart search and fetch eval suites ship with the codebase
• Post-deploy gate — critical checks run before calling a deploy healthy
• Engine fallback chain — when one fetch method fails, WebPeel escalates instead of giving up
• Multiple surfaces, one core — CLI, API, SDK, and MCP all ride the same extraction pipeline

Security

• SSRF protection — blocks localhost, private IPs, metadata endpoints, file:// schemes
• Helmet.js — HSTS, X-Frame-Options, nosniff, XSS protection on all responses
• Webhook signing — HMAC-SHA256 on all outbound webhooks
• API key hashing — SHA-256 with granular scopes
• Rate limiting — sliding window, per-tier
• Audit logging — every API call logged with IP, key, and action
• GDPR compliant — DELETE /v1/account for full data erasure Security policy → · SLA (99.9% uptime) →

Why teams choose WebPeel instead of stitching a stack together

Links

📖 Documentation · 💰 Pricing · 🎮 Playground · 📝 Blog · 💬 Discussions · 🚀 Releases · 📊 Status · 🔒 Security · 📋 Changelog

Contributing

Pull requests welcome. Please open an issue first to discuss major changes.

git clone https://github.com/webpeel/webpeel.git
cd webpeel && npm install
npm run build && npm test

License

WebPeel SDK License — free for personal and commercial use with attribution.