Fetch any web page as clean, AI-ready markdown with smart HTTP-to-browser escalation.
io.github.JakeLiuMe/webpeel MCP Server
This MCP server (slug: io-github-jakeliume-webpeel) provides functionality to fetch web pages and return them as clean, AI-ready markdown. It includes “smart HTTP-to-browser escalation” to handle pages that require browser-like rendering. The project is positioned as a Firecrawl alternative and focuses on web data for RAG and AI agents.
🛠️ Key Features
Fetch web pages and convert to clean markdown
Smart escalation from HTTP to browser when needed
Content extraction / web scraping
Structured data output for downstream use
Screenshot support and browser automation topics
🚀 Use Cases
AI agents performing web crawling and web search workflows
RAG pipelines needing web content and extracted text
Token-efficient web data ingestion
Dataset building from public web pages
⚡ Developer Benefits
Better input quality for Claude Code / Codex style tooling
AI-agent-friendly “AI-ready markdown” format
Focus on web-data, content-extraction, and structured-data workflows
⚠️ Limitations
Intended for web pages; behavior depends on whether HTTP access is sufficient or browser escalation is required
WebPeel demo showing agent-friendly web fetch input, automatic engine selection, and clean JSON output
The Problem
Every AI agent that touches the web rebuilds the same brittle stack: HTTP fetch → headless browser → anti-bot bypass → HTML cleanup → markdown conversion → token budgeting. Each layer fails differently. Sites change. Cloudflare rotates challenges. Your agent gets empty strings at 2 AM and your pipeline breaks.
WebPeel replaces that entire stack with one function call. It handles engine selection, anti-bot escalation, domain-specific extraction, and token optimization so your agent gets clean, structured data every time — without managing browsers, proxies, or parsing logic.
Quick Start
bash
# Zero-install — just run it
npx webpeel "https://example.com"# Search the web
npx webpeel search "latest AI agent frameworks"# Crawl an entire site
npx webpeel crawl docs.example.com --max-pages 50
# Screenshot any page
npx webpeel screenshot "https://stripe.com/pricing" --full-page
# Ask a question about any page
npx webpeel ask "https://arxiv.org/abs/2401.00001""What is the main contribution?"
Or install globally:
bash
npm install -g webpeel
Use as a library:
typescript
import { peel } from'webpeel';
const result = awaitpeel('https://news.ycombinator.com');
console.log(result.markdown); // Clean markdown, ready for your LLMconsole.log(result.metadata); // Title, tokens saved, timing, etc.
🧠 55+ Domain Extractors — Not Just HTML-to-Markdown
Generic scrapers convert raw HTML to markdown and call it a day. WebPeel has purpose-built extractors for 55+ domains — Reddit, GitHub, YouTube, Amazon, ArXiv, Hacker News, Wikipedia, StackOverflow, Zillow, Polymarket, ESPN, and more. Each extractor understands the site's structure and returns clean, structured data without browser rendering.
⚡ 65–98% Token Savings
Domain extractors strip navigation, ads, sidebars, and boilerplate before content reaches your agent. Less context consumed = lower costs, faster inference, and longer agent chains.
Site
Raw HTML tokens
WebPeel tokens
Savings
News article
18,000
640
96%
Reddit thread
24,000
890
96%
Wikipedia page
31,000
2,100
93%
GitHub README
5,200
1,800
65%
E-commerce product
14,000
310
98%
🔄 6-Layer Engine Escalation
WebPeel doesn't just try one method — it automatically escalates through 6 engines until it gets a good result:
No manual --render flags for most sites. WebPeel knows which sites need JavaScript, which need stealth, and which have anti-bot protection — and picks the right engine automatically.
🔌 Firecrawl-Compatible Migration Path
Already using Firecrawl-style workflows? WebPeel supports compatible /v1/scrape, /v2/scrape, /v1/crawl, /v1/search, and /v1/map endpoints, which makes migration dramatically easier than rebuilding your pipeline from scratch.
Agent-Native Integrations
MCP Server (Claude, Cursor, Windsurf, VS Code)
Give any MCP-compatible AI the ability to browse, search, and extract from the web.
Research agents — Give Claude, Codex, Cursor, or your own agent grounded web access through the API or MCP server.
Lead enrichment — Pull company details, public links, and page structure from business sites without writing per-site parsers.
Content aggregation — Crawl and extract from communities, docs sites, and publications with domain-native extractors that understand each site's structure.