Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.
MCP Server: io.github.Mingye-Lu/acrawl
This MCP server provides an Autonomous web crawler implemented as a single Rust binary. It includes 17 browser tools and a goal-driven run_goal agent for navigating, clicking, extracting, and inspecting browser activity such as network traffic and console errors. It can also profile performance and audit accessibility.
🛠️ Key Features
Autonomous web crawler
17 browser tools
Goal-driven run_goal agent
Network traffic inspection and debug console error review
Performance profiling and accessibility auditing
Single Rust binary
Stealth
🚀 Use Cases
Web scraping via navigation, clicks, and extraction
Debugging and inspecting page behavior (network + console)
Performance profiling and accessibility audits
⚡ Developer Benefits
Works as a single binary
Integrates with MCP via provided mcp-client/mcp-server topics
Built for browser automation workflows using Playwright
⚠️ Limitations
Focused on browser automation and crawling; no additional non-browser capabilities are described.
Browser automation agent that acts and observes. Navigate, click, and extract — or inspect network traffic, debug console errors, profile performance, and audit accessibility. One Rust binary.
Single Rust binary. Full DevTools observability. 43 tools. 25 LLM providers. MCP server built-in.
text
acrawl > get the name, price and rating of every book on books.toscrape.com, all 50 pages
acrawl reads the goal, opens the site, works out the pagination and hands back JSON. Along the way the agent can watch the network, read the console, profile performance and audit accessibility, so it can debug a page as well as scrape it. It also runs as an MCP server, so Claude Code, Cursor, VS Code and 14 other clients can drive the same browser.
The script downloads the latest release, verifies its SHA256 checksum and sets up CloakBrowser, the stealth browser acrawl uses for JavaScript-heavy pages. Browser features need Node.js 20+. To build from source, see the install guide.
Quick start
bash
acrawl auth anthropic # or openai, google, other (Ollama, LM Studio, ...), 25 providers in all
acrawl # interactive terminal UI
acrawl prompt "list the titles of the top 10 stories on news.ycombinator.com"# one goal, then exit
Then go further:
bash
acrawl mcp install # add acrawl to Claude Code, Cursor, VS Code and other MCP clients
acrawl run books_catalog -o books.json # replay a saved script with no LLM calls
The quick start walks through a first crawl, getting JSON out, and writing goals that work.
What you can do with it
Extract data from any site. Describe the fields you want; acrawl fetches static pages over HTTP and switches to a real browser only when a page needs JavaScript.
text
acrawl > get all repository names and star counts from github.com/trending
Debug a page like a developer would. Network requests, console logs, WebSocket frames, performance timings, cookies, storage, code coverage and axe-core accessibility audits are all agent tools.
text
acrawl > open localhost:3000/checkout, submit the form, and tell me which requests failed and why
Fill forms and work through flows. Click, type, select, hover, press keys and emulate phones. With the acrawl Bridge extension, it works inside your own logged-in Chrome, Edge or Brave.
Run it again for free. Save a crawl as a deterministic JSON script with loops, conditions and parallel branches, then replay it from cron or CI with acrawl run, with no model calls.
Split big jobs. The agent forks sub-agents onto separate tabs, and a URL-claiming registry stops two of them from crawling the same page.
Why acrawl
One binary. No Python environment, no Docker, no hosted service. Node.js is needed only for the bundled browser.
Acts and observes. Most browser agents navigate and click. acrawl's agent also sees the network, the console and performance, as first-class tools.
AI where you need it, scripts where you don't. Let the agent explore once; replay the result deterministically.
Your choice of model. 25 providers, including Anthropic, OpenAI, Gemini, DeepSeek, Bedrock, Azure, Vertex AI, GitHub Copilot, Groq, Mistral and OpenRouter, plus any OpenAI-compatible endpoint such as Ollama.
MCP both ways. Use other MCP servers as extra agent tools, or expose acrawl's 40 tools to your coding agent with acrawl mcp.
Open source. MIT-licensed, runs on your machine, with your own LLM key and no telemetry.
Bring your own model
Use it from your coding agent
acrawl mcp install sets acrawl up as an MCP server in any of these clients.
Driving acrawl from a coding agent? npx skills add Mingye-Lu/AgenticCrawler installs the acrawl-mcp skill, which teaches the agent when to use direct tools, scripts or run_goal.