WebShift
What is WebShift
WebShift is a Rust library and MCP server that shifts noisy web pages into
clean, right-sized text for LLM consumption.
Raw HTML is mostly junk: scripts, ads, navigation menus, cookie banners,
tracking pixels. Feeding it directly to an LLM floods the context window
with tens of thousands of useless tokens and leaves no room for reasoning.
WebShift strips all that noise, sterilizes the text, and enforces strict
size budgets so the model receives only the content that matters.
What you get
Depending on the features you enable, WebShift can be four things:
| Use case | Crate | Feature flags | What it does |
|---|
| HTML denoiser | webshift | default-features = false | clean() — pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs. |
| HTML text rewriter | webshift | features = ["text-map"] | extract_text_nodes() + replace_text_nodes() — extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched. |
| Web content client | webshift | default or features = ["llm"] | fetch() + query() — streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results. |
| MCP server | webshift-mcp | all features | Native binary (mcp-webshift) that exposes webshift_query, webshift_fetch, and webshift_onboarding over MCP stdio. Single static binary, zero runtime dependencies. |
When to use WebShift
- You're building an AI agent that needs web search and you want clean,
budget-controlled text — not raw HTML.
- You're processing web pages in a Rust pipeline and need a reliable
HTML-to-text cleaner that strips noise without losing real content.
- You need an LLM to translate, rewrite, or simplify text inside HTML
without corrupting the markup — text-map gives you a safe round-trip.
- You want an MCP web search server that works as a single binary —
no Python, no pip, no venv, no Docker (unless you want it).
- You need hard guarantees on output size: per-page caps, total budget
caps, streaming download limits.
When NOT to use WebShift
- You need a headless browser that renders JavaScript-heavy SPAs.
WebShift parses static HTML — it doesn't execute JS.
- You need to render or screenshot a page preserving its visual layout.
WebShift processes HTML structure but does not render CSS or compute layout.
(Note:
text-map does preserve the DOM structure for text rewriting use cases.)
- You're building a web scraper that needs to navigate across pages,
fill forms, or handle authentication flows.
How it works
Question
|
+- (optional) LLM query expansion -> multiple search variants
|
+- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)
|
+- Deduplicate + filter binary URLs
|
+- Streaming fetch with per-page size cap
|
+- HTML cleaning -> plain text (noise elements, scripts, nav removed)
|
+- Unicode/BiDi sterilization
|
+- BM25 deterministic reranking
| +- (optional) LLM-assisted tier-2 reranking
|
+- Budget-aware truncation across all sources
|
+- (optional) LLM Markdown summary with inline citations
|
+- Structured JSON output
For a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see Under the Hood. For the full configuration reference (TOML, env vars, CLI args) see Configuration. For ready-to-use examples see Use Cases.
Installation
Binary (MCP server)
cargo install webshift-mcp
The binary is called mcp-webshift.
From source
cargo install --path crates/webshift-mcp
As a library
webshift = "0.2"
webshift = { version = "0.2", default-features = false }
webshift = { version = "0.2", default-features = false, features = ["text-map"] }
webshift = { version = "0.2", features = ["llm"] }
Quick start
1. Set up a search backend
The easiest option is SearXNG — free, self-hosted, no API key:
docker run -d -p 8080:8080 searxng/searxng
No Docker? Use a cloud backend — see Search backends.
{
"mcpServers": {
"webshift": {
"command": "mcp-webshift",
"args": ["--default-backend", "searxng"]
}
}
}
That's it. The agent now has webshift_query, webshift_fetch, and webshift_onboarding.
For client-specific setup see docs/integrations/.
| Tool | Description |
|---|
webshift_query | Full search pipeline: search + fetch + clean + rerank + (optional) summarize |
webshift_fetch | Single page fetch and clean |
webshift_onboarding | Returns a JSON guide for the agent (budgets, backends, tips) |
webshift_query parameters
| Parameter | Type | Default | Description |
|---|
queries | string or list | required | Search query or list of queries |
num_results_per_query | integer | 5 | Results per query |
lang | string | none | Language filter (e.g. "en") |
backend | string | config default | Override search backend |
Configuration
Resolution order (highest priority first):
- CLI args —
--default-backend, --brave-api-key, etc.
- Environment variables —
WEBSHIFT_* prefix
- Config file —
webshift.toml (current dir, then ~/webshift.toml)
- Built-in defaults
Config file
[server]
max_query_budget = 32000
max_result_length = 8000
max_total_results = 20
max_download_mb = 1
search_timeout = 8
results_per_query = 5
oversampling_factor = 2
adaptive_budget = "auto"
[backends]
default = "searxng"
[backends.searxng]
url = "http://localhost:8080"
[backends.brave]
api_key = "BSA-..."
[backends.tavily]
api_key = "tvly-..."
[backends.exa]
api_key = "..."
[backends.serpapi]
api_key = "..."
engine = "google"
[backends.google]
api_key = "..."
cx = "..."
[backends.bing]
api_key = "..."
market = "en-US"
[backends.http]
url = "https://my-search.example.com/api/search"
query_param = "q"
count_param = "limit"
results_path = "data.items"
title_field = "title"
url_field = "link"
snippet_field = "description"
[backends.http.headers]
"Authorization" = "Bearer my-token"
[llm]
enabled = false
base_url = "http://localhost:11434/v1"
api_key = ""
model = "gemma3:27b"
timeout = 60
expansion_enabled = true
summarization_enabled = true
llm_rerank_enabled = false
For every setting with all three config methods (TOML, env vars, CLI args)
and plain-language descriptions, see the full Configuration Reference.
Ready-to-use config examples are in Use Cases and examples/.
Key environment variables
WEBSHIFT_DEFAULT_BACKEND=searxng
WEBSHIFT_SEARXNG_URL=http://localhost:8080
WEBSHIFT_BRAVE_API_KEY=BSA-xxx
WEBSHIFT_GOOGLE_API_KEY=xxx
WEBSHIFT_GOOGLE_CX=xxx
WEBSHIFT_BING_API_KEY=xxx
WEBSHIFT_LLM_ENABLED=true
WEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1
WEBSHIFT_LLM_MODEL=gemma3:27b
Search backends
LLM features (optional)
All opt-in — disabled by default, no data leaves your machine unless enabled.
| Feature | What it does |
|---|
| Query expansion | Single query -> N complementary search variants |
| Summarization | Markdown report with inline [1] [2] citations |
| LLM reranking | Tier-2 reranking on top of deterministic BM25 |
Cross-language normalization (bonus): when BM25 reranking surfaces pages in
foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still
produces the final report in the prompt language. The agent receives clean,
readable output regardless of the language mix in the source pages.
Works with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):
[llm]
enabled = true
base_url = "http://localhost:11434/v1"
model = "gemma3:27b"
Anti-flooding protections
Always active — the core value proposition:
| Protection | Description |
|---|
max_download_mb | Streaming cap — never buffers full response |
max_result_length | Hard cap on characters per cleaned page |
max_query_budget | Total character budget across all sources |
max_total_results | Hard cap on results per call |
| Binary filter | .pdf, .zip, .exe, etc. filtered before any network request |
| Unicode sterilization | BiDi control chars, zero-width chars removed |
Library usage
use webshift::{Config, clean, fetch, query};
let result = clean("<html><body><p>Hello world</p></body></html>", 8000);
println!("{}", result.text);
let full = clean("<html><body><p>Hello world</p></body></html>", 0);
assert!(!full.truncated);
let config = Config::default();
let page = fetch("https://example.com", &config).await?;
let results = query(&["rust async programming"], &config).await?;
for source in &results.sources {
println!("[{}] {} — {} chars", source.id, source.title, source.content.len());
}
if results.sources.is_empty() && !results.warnings.is_empty() {
eprintln!("all backends failed: {:?}", results.warnings);
}
Text-map: rewrite HTML content without breaking markup
Extract text nodes, manipulate them (translate, rewrite, simplify), and rebuild
the HTML with structure, attributes, and links intact.
use webshift::{extract_text_nodes, replace_text_nodes, TextReplacement};
let html = r#"<p>Hello <a href="https://example.com">world</a></p>"#;
let map = extract_text_nodes(html);
let replacements = vec![
TextReplacement { id: 0, text: "Ciao".into() },
TextReplacement { id: 1, text: "mondo".into() },
];
let result = replace_text_nodes(html, &replacements).unwrap();
Requires features = ["text-map"]. See Use Cases #11 for a full translation example.
Feature flags
| Feature | Default | Enables |
|---|
backends | on | All search backends + query pipeline |
llm | off | LLM client, expander, summarizer, LLM reranking |
text-map | off | extract_text_nodes() + replace_text_nodes() — DOM round-trip for content rewriting |
Integrations
Beta Status
WebShift is in beta. Core functionality is stable and the server is used daily,
but the API surface may still change before 1.0.
Feedback is very welcome. If something doesn't work as expected, behaves oddly,
or you have a use case that isn't covered:
Open an issue on GitHub
Bug reports, configuration questions, and feature requests all help shape the roadmap.
Contributing
Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines on:
- Development setup and workflow
- Code style and conventions
- Testing requirements
- Documentation standards
- Pull request process
License
MIT License — see LICENSE for details.
Links
Need help? Check the documentation or open an issue on GitHub.