Paste a URL, get real results in seconds — webmetadataextractor.com runs the actual pipeline live, no API key needed. Try your own site, or one of these:

/demo/extract is the same no-key route the live demo page above uses (see "Honest Limitations" — it's rate-limited per IP, not a documented/stable API endpoint). For integrating in code, use /api/v1/extract with a RapidAPI key as shown below.

Or call it directly (swap in your key from RapidAPI for production use):

curl "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=github.com" \
  -H "X-RapidAPI-Key: YOUR_KEY" -H "X-RapidAPI-Host: web-metadata-and-contact-extractor.p.rapidapi.com"

🎯 Use Cases

import requests

RAPIDAPI_KEY = "YOUR_RAPIDAPI_KEY"
HEADERS = {
    "X-RapidAPI-Key": RAPIDAPI_KEY,
    "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com",
}
BASE = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"

for url in ["shop-a.com", "shop-b.com", "shop-c.com"]:
    data = requests.get(BASE, headers=HEADERS, params={"url": url}).json()
    if "Shopify" in data["detected_technologies"] and data["seo_score_percentage"] < 60:
        print(f"{url}: Shopify + {data['seo_score_percentage']}% SEO -> outreach candidate")

import requests

def read_url_as_markdown(url: str) -> str:
    """Tool function an LLM agent can call to read any public webpage."""
    resp = requests.get(
        "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/markdown",
        headers={"X-RapidAPI-Key": "YOUR_KEY", "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"},
        params={"url": url},
    ).json()
    return resp["markdown_content"]

🌟 Key Features

🎯 Performance SLA & Technical Architecture Notes

🎯 What This API Is — And Isn't — For

This API turns a URL into structured knowledge about that page: SEO/OpenGraph metadata, public contact signals, social profiles, tech stack, Schema.org/product data, graded security headers, link classification, and clean Markdown — consolidated into one fast HTTP request. It's a metadata/intelligence layer over ordinary server-rendered HTML, not a general-purpose scraping tool or a browser-automation replacement. Keep that framing in mind when deciding whether it fits your use case.

✅ Good fit:

• Company/website analysis — pull tech stack, CMS, socials, contact signals, and SEO health for a given site in one call.
• Input for a sales/outreach workflow — /api/v1/contacts finds public emails/phones/social links present on a page; it's a raw signal you feed into a lead-gen process you build (company identification, role verification, CRM enrichment), not a lead-enrichment product by itself — see "Honest Limitations" below.
• Automated SEO audits — the 14-point on-page score, structured seo_checks breakdown, and graded security-header audit are built for exactly this.
• Competitive/tech-stack monitoring — compare tech stack, metadata, or product data across a known set of URLs over time. You supply and re-check the URL list; there's no built-in crawler (see below).
• AI agents / RAG pipelines — markdown_content and the structured JSON fields are designed as clean LLM input, not raw HTML soup.
• E-commerce data extraction — product name/price/currency/availability/brand/SKU/reviews where a site exposes Schema.org JSON-LD, OpenGraph product tags, or Microdata, cross-checked with a per-field confidence score. Product responses use a short 3-minute cache specifically because price data goes stale faster than everything else.
• Link previews — title, description, image, favicon for chat apps, social cards, and bookmarking tools.

❌ Not a good fit:

• Complex/interactive scraping — logins, multi-step forms, button clicks, infinite scroll, CAPTCHA-gated content. This API issues one HTTP GET and parses the HTML it gets back; it does not drive a browser or simulate user interaction.
• Heavily JS-rendered sites — if the meaningful content only exists after client-side hydration, this API sees close to the same empty shell a plain HTTP client would. The adaptive byte limit improves coverage of what the server already sent; it doesn't execute JavaScript.
• Mass or distributed crawling — there's no queue, frontier, or "give me a domain and I'll find its pages" mode. It's one URL in, one page's data out, by design — kept stateless and fast rather than growing into a crawler.
• Penetration testing or vulnerability scanning — the security-header audit grades the presence and quality of headers like HSTS, CSP, and X-Frame-Options. It does not probe for vulnerabilities, scan for misconfigurations beyond those headers, or constitute a security assessment. A high security_score means "this page sends good security headers," not "this site is secure."
• "Extract anything from any site" scraping — this is a fixed set of well-defined extractors (SEO, contacts, socials, tech, product, links, Markdown), not a programmable scraper for arbitrary custom fields or page structures.

For the technical reasoning behind these boundaries — why JS execution, crawling, and bot-protection bypass are out of scope, and what the confidence/quality scores do and don't mean — see "Honest Limitations" below.

Wondering how this compares to ScrapingBee, Firecrawl, or BuiltWith? See COMPETITOR_COMPARISON.md for an honest, pricing-verified breakdown of when each one actually wins.

⚠️ Honest Limitations

This API fetches raw HTTP responses and parses HTML — it is not a browser. That's a deliberate trade-off for speed (no browser startup, no JS execution wait, minimal memory per request), and it comes with real ceilings that no amount of additional regex or extractors can fully remove:

• No JavaScript execution. Content that only exists after client-side rendering (many SPAs, some login-gated pages, content behind "load more" interactions) will be missing or incomplete. A static/SSR page (most blogs, e-commerce product pages, marketing sites, GitHub, Wikipedia) works great; a client-rendered single-page app whose initial HTML is just <div id="root"></div> will return mostly empty results — the API cannot execute the JS that would fill it in.
• Detecting a JS framework doesn't mean the JS problem is solved. The adaptive byte limit (see below) expanding to 256 KB when React/Next.js/Vue/etc. signatures are spotted improves coverage of the initial HTML — it does not run a browser, execute hydration, or wait for client-side API calls. Next.js -> more bytes -> a bigger parse is not Next.js -> Chromium -> JS execution -> hydration -> final DOM. Treat it as "we tried harder to read what the server already sent," not "we solved SPAs."
• Bot-protection / CAPTCHAs are detected, not bypassed. bot_protection_detected (see above) tells you that Cloudflare/Akamai/PerimeterX/a CAPTCHA blocked the request — it does not solve the challenge. There is no workaround for this short of running a real browser.
• Extraction is heuristic evidence, not ground truth — including when it looks confident. Technology detection, contact info, social links, and product data are pattern/signature-based. A tech signature match (e.g. one CDN hostname in an image URL) proves the string was present somewhere the fetcher looked, not that the technology is meaningfully in use — it could be dead code, a shared/third-party asset, a transitive dependency, or an accidental substring. technology_details.confidence reflects how many independent signals agreed, not a probability that the detection is correct. See the live GitHub example above, where "Contentful" is detected at 0.75 confidence from a single CDN hostname in an og:image URL — plausible evidence, not proof GitHub runs on Contentful.
• A quality.score near 1.0 means the evidence available to this API was internally consistent — not that the data is verified accurate. When JSON-LD, OpenGraph, and Microdata all report the same price, that's three encodings, not necessarily three independent sources — a site frequently generates all three from the same underlying product record, so agreement can mean "one data source, expressed three ways" rather than three independently-verified facts. Read confidence: 0.98 as "nothing on the page contradicts this," not "there's a 98% chance this is correct."
• The adaptive 64/256 KB byte limit is a real trade-off, not just a performance knob. It's what keeps typical requests fast, but it means content positioned late in a large page — JSON-LD placed near the closing </body>, a product block far down a long CMS-generated page — can be missed purely because of where it sits in the HTML, independent of how "complex" the page seems. Larger/more script-heavy pages are paradoxically more likely to have relevant data pushed past the cutoff. Always check quality.warnings for a CONTENT_TRUNCATED entry before treating a missing field as "this page doesn't have it" rather than "this API didn't read that far" — there's no separate top-level content_truncated boolean, that signal lives inside quality.
• Markdown conversion is not semantic understanding. markdown_content is a reasonably clean HTML→Markdown conversion, not a "read and understand what's actually the article" model — it doesn't reliably distinguish primary content from navigation, related-content widgets, paywalled teasers, or boilerplate on every layout.
• The cache trades freshness for speed, and that trade-off matters more for some fields than others. A 15-minute-stale <title> is harmless; a 15-minute-stale price is not — which is why product-bearing responses get a separate, much shorter TTL (PRODUCT_CACHE_TTL_SECONDS, 3 minutes by default) rather than sharing the general 15-minute cache. Everything else still follows the general TTL: don't build a real-time price-monitoring product directly on the general cache path without accounting for that window.
• Arbitrary URL fetching is inherently unreliable, independent of this API: pages can be slow, redirect repeatedly, be enormous, block automated clients, have TLS quirks, serve different content by geography/User-Agent, be temporarily down, or return unexpected status codes. SSRF protections, timeouts, redirect limits, byte caps, and the circuit breaker (see above) bound the damage but can't make an unreliable target reliable.
• "~150–300ms" describes the live-fetch network leg, not a ceiling on total response time. Server-side processing genuinely is single-digit milliseconds (see the benchmark methodology in benchmarks/), but the live end-to-end time is dominated by however long the target site takes to respond — if the target takes 2.5s, so does your request. Don't read the headline numbers as "this API responds in 200ms regardless of target."
• There is no crawler. The unit of work is one URL in, one page's data out — not "give me a domain and I'll discover/prioritize/crawl its pages for you." /api/v1/links classifies up to 100 hyperlinks on the page you gave it; it doesn't follow them. Analyzing a whole site (homepage + /about + /pricing + /blog, deduplicated, prioritized, crawled) is something you'd orchestrate externally, one /extract call per URL — this project deliberately doesn't grow a queue/worker/crawl-frontier system into what's meant to stay a fast, stateless per-request API.
• /api/v1/contacts finds public contact signals, not company or people intelligence. It returns emails/phones/social links it found in the page's HTML — it does not identify who those belong to, their role, or verify anything. Turning website -> public contact signals into real lead enrichment (domain -> company identification -> people -> roles -> verification -> confidence) is out of scope here; the endpoint is intentionally named/tagged "Contact Discovery," not "Lead Enrichment," to avoid implying more than it does.
• Prefer the specific endpoint over /api/v1/extract — or use fields=. /extract without fields runs the complete pipeline. /extract?fields=detected_technologies now runs exactly the same narrow extraction as GET /api/v1/tech-stack (and shares its cache entry) — fields genuinely skips unneeded work rather than computing everything and filtering afterward.
• This project's API surface is still evolving. New fields ship frequently (quality, product_field_confidence, bot_protection_detected, and others were all added recently) — additive changes only, existing fields never change type (see the versioning notes below), but if you're integrating deeply, treat the live /openapi.json schema as the source of truth over any single README snippet, which can lag by a commit or two.

None of this means the underlying approach is flawed for its intended job (fast metadata/SEO/contact/tech extraction from ordinary web pages) — it means "extract everything from any URL via one HTTP request" has a ceiling that only browser automation (Playwright/Puppeteer) can raise, at the cost of the speed and low resource footprint that make this API fast in the first place.

⚡ Quick Start

Python

import requests

url = "https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract"
headers = {
    "X-RapidAPI-Key": "YOUR_RAPIDAPI_KEY_HERE",
    "X-RapidAPI-Host": "web-metadata-and-contact-extractor.p.rapidapi.com"
}
params = {"url": "https://github.com"}

data = requests.get(url, headers=headers, params=params).json()

print(f"Title:        {data['metadata']['title']}")
print(f"OG Type:      {data['metadata']['og_type']}")
print(f"Robots:       {data['metadata']['robots']}")
print(f"hreflang:     {data['metadata']['hreflang_tags']}")
print(f"Product:      {data['product_data']}")
print(f"Emails:       {data['contacts']['emails']}")
print(f"Tech Stack:   {data['detected_technologies']}")
print(f"Time:         {data['execution_time_ms']} ms")

JavaScript / Node.js

const url = 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com';
const response = await fetch(url, {
  headers: {
    'X-RapidAPI-Key': 'YOUR_RAPIDAPI_KEY_HERE',
    'X-RapidAPI-Host': 'web-metadata-and-contact-extractor.p.rapidapi.com'
  }
});
const data = await response.json();
console.log('Title:', data.metadata.title);
console.log('OG Type:', data.metadata.og_type);
console.log('Product:', data.product_data);
console.log('Time:', data.execution_time_ms, 'ms');

cURL

curl --request GET \
  --url 'https://web-metadata-and-contact-extractor.p.rapidapi.com/api/v1/extract?url=https%3A%2F%2Fgithub.com' \
  --header 'X-RapidAPI-Host: web-metadata-and-contact-extractor.p.rapidapi.com' \
  --header 'X-RapidAPI-Key: YOUR_RAPIDAPI_KEY_HERE'

📦 Client Libraries

For Python, prefer the official client over raw requests calls — same 11 endpoints, typed exceptions, and both sync and async clients:

pip install webmetadata-extractor

from webmetadata_extractor import WebMetadataClient

client = WebMetadataClient(api_key="YOUR_RAPIDAPI_KEY_HERE")
data = client.extract("https://github.com")

Prefer the terminal? The same package installs a web-inspect CLI (or grab it standalone via Homebrew, no Python required):

brew tap josejux/webmetadata && brew install web-inspect   # or: pip install webmetadata-extractor

web-inspect example.com --security
web-inspect example.com --markdown > article.md

Building an agent? There are tool packages for the three most common frameworks — each ships tools (extract, markdown/reader, contacts, seo_audit) ready to hand to an agent:

pip install langchain-webmetadata-extractor    # LangChain / LangGraph
pip install crewai-webmetadata-extractor       # CrewAI
pip install llamaindex-webmetadata-extractor   # LlamaIndex (reader + tools)

from langchain_webmetadata_extractor import get_tools

tools = get_tools(api_key="YOUR_RAPIDAPI_KEY_HERE")

LlamaIndex also gets a dedicated WebMarkdownReader for RAG ingestion (loads a URL straight into a Document, not just an agent tool call):

from llamaindex_webmetadata_extractor import WebMarkdownReader

documents = WebMarkdownReader(api_key="YOUR_RAPIDAPI_KEY_HERE").load_data(["https://example.com"])

All four packages are MIT-licensed and open source: webmetadata-extractor, langchain-webmetadata-extractor, crewai-webmetadata-extractor, llamaindex-webmetadata-extractor.

Want a full working example instead of a snippet? See agent-cookbooks/ for three real workflows built on these packages: a CrewAI competitor analyzer, a LlamaIndex RAG pipeline fed with clean Markdown, and a LangGraph price/stock monitor that alerts only on real changes.

Want to try it without installing anything? Live demo on Hugging Face Spaces — paste a URL, see the full response in your browser.

🤖 MCP / AI Agent Support

This API is available as an MCP server out of the box, via RapidAPI's hosted gateway — no extra setup or self-hosting needed. Add it to Claude Desktop, Claude Code, Cursor, or any MCP-compatible client:

{
  "mcpServers": {
    "RapidAPI Hub - Web Metadata and Contact Extractor": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://mcp.rapidapi.com",
        "--header",
        "x-api-host: web-metadata-and-contact-extractor.p.rapidapi.com",
        "--header",
        "x-api-key: YOUR_RAPIDAPI_KEY_HERE"
      ]
    }
  }
}

Once connected, an agent can call the extractor directly as a tool — e.g. pulling markdown_content into a RAG pipeline, or checking detected_technologies and seo_score_percentage as part of an autonomous research/outreach workflow. Get a free key from the RapidAPI listing first.

🧩 Use in Dify, Flowise, or Langflow

No plugin install needed for any of these — visual agent builders can call this API directly.

Dify or Flowise, in about 60 seconds: both support importing a custom tool straight from an OpenAPI schema. Get a free key from the RapidAPI listing, then:

1. Dify: Tools → Custom Tool → Create from OpenAPI. Flowise: add an "OpenAPI Toolkit" node and point it at the schema.
2. Paste in dify-flowise-openapi.json — a curated 3-endpoint slice of the full API (extract, markdown, schema) pre-configured to call through the RapidAPI gateway.
3. In the tool's auth settings, set header X-RapidAPI-Key to your key. X-RapidAPI-Host is already fixed to the right value in the spec — nothing else to configure.

Need a different endpoint than those 3? The full OpenAPI spec has all 11 — same import flow, just point at that URL instead and add the same two headers.

Langflow: zero setup beyond what's already above — Langflow speaks MCP natively (both as client and server as of v1.8.4). Add an MCP Tools component, connect it to https://mcp.rapidapi.com with the same x-api-host / x-api-key headers shown in the "MCP / AI Agent Support" section above, and every endpoint shows up as a tool.

📊 Sample API Response (JSON)

{
  "url": "https://github.com",
  "final_url": "https://github.com",
  "status_code": 200,
  "execution_time_ms": 130.43,
  "bot_protection_detected": false,
  "metadata": {
    "title": "GitHub · Change is constant. GitHub keeps you ahead. · GitHub",
    "description": "Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.",
    "og_image": "https://images.ctfassets.net/8aevphvgewt8/4pe4eOtUJ0ARpZRE4fNekf/f52b1f9c52f059a33170229883731ed0/GH-Homepage-Universe-img.png",
    "og_type": "object",
    "canonical_url": "https://github.com",
    "language": "en",
    "favicon": "https://github.githubassets.com/favicons/favicon.svg",
    "theme_color": "#1e2327",
    "hreflang_tags": [
      {"lang": "en-us", "url": "https://github.com"},
      {"lang": "pt-br", "url": "https://github.com?locale=pt-br"}
    ],
    "h1_tags": [],
    "links_count": 51,
    "content_length_bytes": 109075
  },
  "social_links": { "github": "https://github.com/features/copilot", "twitter": null, "linkedin": null, "...": "17 more platforms, all null here" },
  "contacts": { "emails": [], "phones": [] },
  "detected_technologies": ["Contentful"],
  "technology_details": [
    { "name": "Contentful", "confidence": 0.75, "evidence": ["ctfassets.net"], "category": "cms" }
  ],
  "product_data": null,
  "quality": {
    "score": 0.85,
    "rendered": false,
    "sources_used": ["meta"],
    "warnings": [{ "field": null, "type": "CONTENT_TRUNCATED" }]
  },
  "rss_feeds": ["https://github.com/opensearch.xml", "https://github.githubassets.com/favicons/favicon.svg"],
  "json_ld_schemas": [],
  "security_score_percentage": 64.2,
  "seo_score_percentage": 75.0,
  "seo_passed_checks": ["Title tag present with optimal length (10-70 chars)", "Canonical link tag present", "OpenGraph image present for social sharing", "Favicon icon present"],
  "seo_warnings": ["Meta description present but sub-optimal length (186 chars)", "Missing <h1> primary heading", "No structured data (JSON-LD) found"],
  "internal_links": ["https://github.com/", "https://github.com/login", "https://github.com/features/copilot"],
  "total_internal_count": 44,
  "total_external_count": 5,
  "word_count": 0,
  "reading_time_minutes": 0.0,
  "markdown_content": ""
}

This response is exactly MetadataResponse (app/models/responses.py) serialized, trimmed for length — every field shown above is real. Four model fields aren't shown: phone_details and product_field_confidence (both empty here — no phone numbers or product_data on this page), seo_checks (the structured, per-check version of the seo_passed_checks/seo_warnings already shown above), and external_links (5 URLs, per total_external_count, omitted the same way internal_links is truncated). There's no separate content_truncated/bytes_downloaded at the top level; that signal lives inside quality.warnings.

Since v4.0.0, metadata also includes viewport, twitter_card, and h1_count; /api/v1/tech-stack additionally returns technology_details (confidence score, matched evidence, category per technology); and product_data (when present) includes sku, mpn, gtin/isbn, seller, condition, and price-range fields. All additions are purely additive — no existing field was removed or changed type. Fields that were previously untyped Dict/List[Dict] blobs (metadata, technology_details, seo_checks, phone_details) now have documented Pydantic models in the OpenAPI schema — with product_data, previously missing from /api/v1/extract despite being shown here, now actually returned. Also new: product_field_confidence and a top-level quality object — e.g. for a page where JSON-LD says €39.99 and OpenGraph says €29.99: "product_field_confidence": {"price": {"value": "39.99", "confidence": 0.5, "source": "json_ld", "agreement": ["json_ld", "microdata", "opengraph"]}} and "quality": {"score": 0.9, "rendered": false, "sources_used": ["json_ld", "microdata", "opengraph", "meta"], "warnings": [{"field": "product.price", "type": "SOURCE_CONFLICT", "values": {"json_ld": "39.99", "microdata": "39.99", "opengraph": "29.99"}, "chosen_source": "json_ld", "chosen_value": "39.99"}]}.

Newest additions: every endpoint's response now also includes redirect_count (HTTP redirect hops followed to reach final_url) and is_shortened_url (true if the URL you passed in — not final_url — is on a known shortener domain like bit.ly). /api/v1/seo-audit and /api/v1/extract add a readability object (sentence_count, paragraph_count, avg_words_per_sentence, heading_structure). Three brand-new endpoints: POST /api/v1/batch (up to 10 URLs, lightweight link-preview extraction per URL), GET /api/v1/domain (DNS/WHOIS lookups, no page fetch), and /api/v1/security?include_tls_details=true (opt-in live TLS certificate inspection, off by default). All additive — no existing field changed shape or type.

Also new: /api/v1/security (and security_headers on /api/v1/extract) now detects Content-Security-Policy-Report-Only as its own "report-only" grade instead of treating it as missing, and every graded header ships a security_header_explanations entry explaining why it matters.

📖 API Endpoint Documentation

Query Parameters

🔧 Self-Hosting & Local Development

# 1. Clone
git clone https://github.com/JosejuX/rapidapi-metadata-extractor.git
cd rapidapi-metadata-extractor/rapidapi_service

# 2. Install
pip install -r requirements.txt

# 3. Fixture-based test suite (SSRF matrix, circuit breaker, single-flight, rate-limit atomicity, ...)
pip install pytest
pytest tests/ -q

# 3b. Live-network smoke suite (14 SSRF vectors + 12 global domains)
python test_api.py

# 4. Load test (concurrent requests benchmark)
python load_test.py

# 5. Dev server
uvicorn main:app --reload --port 8000

Open http://localhost:8000/docs for the interactive Swagger UI.

Docker

# Pull the published image (built + pushed by .github/workflows/docker-publish.yml on every version tag)
docker pull ghcr.io/josejux/rapidapi-metadata-extractor:latest
docker run -p 8000:8000 ghcr.io/josejux/rapidapi-metadata-extractor:latest

# Or build it yourself
cd rapidapi_service
docker build -t rapidapi-metadata-extractor .
docker run -p 8000:8000 rapidapi-metadata-extractor

🏷️ Keywords

metadata-extractor • opengraph-parser • email-scraper • contact-extractor • social-links-finder • tech-stack-detector • seo-parser • fastapi • rapidapi • python-web-scraper • link-preview-generator • lead-generation-api • hreflang • schema-org • product-data-extractor • ssrf-protection

🤝 Built With

This API was built and iterated on in collaboration with AI coding assistants — Google's Gemini 3.6 Flash and Anthropic's Claude Sonnet 5 — alongside human design, review, and testing decisions throughout.

📄 License

Distributed under the MIT License. See LICENSE for more information.