Scrape-LE: Zero Hassle Scrapeability Checks

Load a URL in headless Chromium and see what will block your scraper — before you write it
Anti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots

Useful? A star or rating is how other developers find it — ★ GitHub · ★ Open VSX · ★ Marketplace

What it does

Run Scrape-LE: Check URL Scrapeability, enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).

One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).

Install

Use it from an AI agent

The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.

analyze_robots_txt(content, path, agent?, maxResults?)

Given robots.txt contents and a path, reports whether the rules permit crawling it: the group naming agent when one does, otherwise the generic (User-agent: *) rules, with agent in the answer saying which. Plus the crawl delay, disallowed patterns and any sitemaps.

The server takes content and returns data — it reads no files and makes no network requests of its own. Published as scrape-le-mcp on npm and as io.github.nolindnaidoo/scrape-le in the MCP registry.

{
  "mcpServers": {
    "scrape-le": {
      "command": "npx",
      "args": ["-y", "scrape-le-mcp"]
    }
  }
}

npm install -g scrape-le-mcp

{
  "mcpServers": {
    "scrape-le": { "command": "scrape-le-mcp" }
  }
}

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | npx -y scrape-le-mcp

The CLI

The same check runs from a terminal or an agent loop: a Rust CLI in crate/ of this repository, sharing one signature corpus with the extension — crate/signatures/ and crate/fixtures/ — so CI fails if the two ever disagree about a URL.

scrape-le https://example.com/search   # JSON on stdout, summary on stderr
scrape-le --input urls.txt             # a batch, streamed as it completes
scrape-le mcp                          # the same check over MCP on stdio

The exit code is the answer: 0 clear · 1 a real no · 2 the question was malformed. ## Detections

Honest limitations: signatures are best-effort fingerprints of public integration patterns — a detected widget means the page can challenge you, not that it will, and a clean result is not proof a site allows scraping. Agent-specific robots.txt groups are ignored (only the * rules are reported). Pages get up to 5 seconds to go network-idle after load, so content rendered later than that can be missed by the page-level detections.

Commands

No command is bound to a key by default. Give any of them one under Keyboard Shortcuts in the editor.

Settings

Languages

Twelve languages besides English:

German · Spanish · French · Indonesian · Italian · Japanese · Korean · Portuguese (Brazil) · Russian · Ukrainian · Vietnamese · Chinese (Simplified)

Both halves are covered — the manifest (command titles, setting names and descriptions) and everything shown while the extension runs (notifications, the status bar, quick-picks and prompts). The extension follows VS Code's display language, so it matches whatever the editor is already set to; no setting of its own.

Privacy & security

• Network access is the feature, and it is scoped. A check talks to exactly two things: the URL you enter (loaded in headless Chromium, which fetches that page's own resources like any browser) and that origin's /robots.txt. Nothing is sent anywhere else — no telemetry, no analytics.
• Screenshots stay local, written to the configured path inside your workspace.
• The MCP server makes no network request at all — unlike the extension, deliberately. fetchRobotsTxt builds a URL from an arbitrary origin, which inside an agent loop is an SSRF primitive: the caller supplying the URL is the model, not you. The server analyses robots.txt content you already fetched, and a test asserts no tool accepts a url argument.
• Error notifications redact home directories and credential-shaped fragments.
• Respect the sites you check: a scrapeability report is information, not permission.
• One rating prompt, at most twice. On the 3rd successful use the extension asks once whether you would rate it, and once more on the 20th if you chose Later or dismissed it. Don't Ask Again ends it. Setting notificationsLevel to important or silent yourself turns it off. The counts are kept in VS Code's extension storage and nothing is sent anywhere; Rate opens the listing you installed from — the VS Code Marketplace or Open VSX — in your browser.

Documentation

Performance

Median of 7 runs after warmup, on Apple M5 Pro, 24 GB RAM, Node 24.3.0. Inputs are generated by scripts/benchmark.ts rather than checked in, so the sizes above are exactly what was measured. Reproduce with bun run benchmark.

These are machine-specific and are not asserted in CI — a benchmark that gates a build only tells you how busy the runner was.

Testing

421 test cases across 33 files, plus an integration suite that runs in a real VS Code extension host and an end-to-end test that installs the built .vsix into a clean profile.

Generated from a real run — coverage/coverage-summary.json and coverage/test-results.json — by scripts/coverage-readme.js; CI fails if this section drifts. Reproduce with bun run test:coverage, and the case count is the one vitest prints.

More from the LE family

Sixteen single-purpose tools for the work in front of every model. Each ships a Rust CLI and an MCP server. One page: letools.dev

Get it out

• String-LE — Extract every string in a codebase, with its position, so a person can read them
• Numbers-LE — Extract every hardcoded number in a codebase, so a person can check them
• Units-LE — Extract every quantity with its unit, normalized, and refuse the ambiguous ones by name
• Dates-LE — Extract every date and timestamp, and the exact instant each one resolves to
• IDs-LE — Extract every UUID, ULID, NanoID, ObjectId and Snowflake, and decode the time inside
• IPs-LE — Extract every IP address, CIDR block and MAC, normalized and classified by scope
• URLs-LE — Extract every URL in a codebase, with its protocol and exact position
• Paths-LE — Extract every file path in a codebase, and say whether it still points at anything
• Colors-LE — Extract every color in a codebase, and say which ones are not in your palette

Check it

• Regex-LE — Find every regex in a codebase, and report which can be driven into catastrophic backtracking
• Versions-LE — Find where one dependency is constrained differently across a repository's manifests
• i18n-LE — Identify the i18n library a project uses, then audit its catalogs by that library's rules
• Scrape-LE — Check whether a page is scrapeable before the scraper is written, and say when it cannot tell

Guard it

• Secrets-LE — Find hardcoded credentials in a codebase, and never print one into the report
• EnvSync-LE — Compare the dotenv files in a tree, and say which keys are missing from which
• Unicode-LE — Find the Unicode that hides meaning — bidi controls, invisibles, homoglyphs, mixed scripts

Each stands on its own: no shared crate, no published core. Where two of them agree, it is because the same answer was right twice.

Contact — nolindnaidoo.com · GitHub · LinkedIn

Also by nolindnaidoo

Rust — pixelcoords and pixelactions are one loop: pixelcoords answers where, pixelactions acts there. Their own tools, their own voice — not part of the LE family.

• pixelcoords — Freeze your screen, mark regions, get pixel-exact coordinates and crops pixelcoords.dev · crates.io · docs.rs
• pixelactions — Consume human-verified coordinates, perform the interaction, confirm it landed pixelactions.dev · crates.io · docs.rs

License

MIT © nolindnaidoo