Scrape, crawl, and map websites to Markdown or JSON via local CLI.
io.github.kreuzberg-dev/kreuzcrawl MCP Server
This MCP server provides local CLI capabilities to scrape, crawl, and map websites, producing output in Markdown or JSON. It targets web-crawler and web-scraping workflows and exposes functionality under the io.github.kreuzberg-dev/kreuzcrawl slug, with development topics spanning multiple languages.
🛠️ Key Features
Scrape websites
Crawl websites
Map websites to Markdown or JSON
Local CLI execution
🚀 Use Cases
Converting crawled web content into Markdown
Exporting crawled results as JSON
Building or testing web-crawler and web-scraping pipelines
⚡ Developer Benefits
Output formats: Markdown and JSON
Supports integration within MCP-based toolchains
Language ecosystem coverage indicated by topics (csharp, elixir, golang, java, php, python, ruby, rust, typescript, wasm)
⚠️ Limitations
No tool count or specific tool interfaces are provided in the available data.
Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.
What and Why?
You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.
It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.
Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.
Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass
Batch & streaming
Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams
SSRF-safe by default
Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or CrawlConfig
Auth & rate limiting
HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling
MCP server & REST API
Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec
Supported Platforms
Precompiled binaries for glibc Linux (x86_64/aarch64), macOS, and Windows across every binding. On musl Linux (Alpine) coverage is deliberately partial — the CLI, Docker image, Rust, Ruby, Java, C#, and Elixir work there; the Node, Python, Go, and PHP bindings do not ship musl artifacts. See the platform support matrix for the full table and the Alpine workarounds.
⭐ Star this repo to show your support — it helps others discover Crawlberg.
Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, and works with every major coding agent — expand your harness below.
Xberg — the open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence. MIT.
Xberg Pro — a complete self-hosted content-intelligence backend in a single container. Commercial.
Xberg Enterprise — the distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.
crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
