Sitemap URL Extractor for AI Agents

Official5 tools

by industrial-platform-ai · JavaScript

x402 sitemap discovery and URL extraction for agents. $0.0001 per emitted URL.

io.github.industrial-platform-ai/sitemap (MCP Server)

The io.github.industrial-platform-ai/sitemap MCP server performs sitemap discovery and URL extraction for agents. It is described as an “x402” sitemap discovery and URL extraction service, and it reports pricing of $0.0001 per emitted URL, indicating a usage-based cost tied to extracted URLs.

🛠️ Key Features

Sitemap discovery
URL extraction for agents
Emits extracted URLs with a usage-based price of $0.0001 per emitted URL

🚀 Use Cases

Agent-driven discovery of URLs from sitemaps
Automated URL collection based on sitemap sources

⚡ Developer Benefits

Designed for agent workflows that need sitemap-derived URLs
Cost scales per emitted URL ($0.0001 per emitted URL)

⚠️ Limitations

Pricing is specified per emitted URL, but no other operational constraints (e.g., limits, supported sitemap types) are provided

Topics

agent-toolsagentic-aiai-agentsapifyautonomous-agentsdeveloper-toolsmcpmodel-context-protocolresearch-apiweb-research

Captured live from the server via tools/list.

get-actor-run

Get detailed information about a specific Actor run. Returns run result: status, storages (datasets/keyValueStores alias map), stats, summary, nextStep. - summary describes the past (e.g. "SUCCEEDED in 22s. 47 items; 3 fields available."). - nextStep prescribes one primary follow-up action with identifiers interpolated (e.g. "Use get-dataset-items with datasetId=..."). - waitSecs (0–45, default 30) waits up to that many seconds for terminal status before returning. USAGE: - Use to check the status of a run started by any Actor-running tool. - Pass waitSecs > 0 to block until terminal (or until the cap elapses). USAGE EXAMPLES: - user_input: Show details of run y2h7sK3Wc - user_input: Wait for run y2h7sK3Wc to finish This tool requires an x402 payment. Include a valid x402 payment signature in the request metadata (_meta["x402/payment"]). Your MCP client must support the x402 payment protocol.

Parameters2

runId	string	required	The ID of the Actor run.
waitSecs	integer	optional	Maximum seconds to wait for the run to reach a terminal state (SUCCEEDED, FAILED, ABORTED, TIMED-OUT). 0 returns immediately with the current status. Cap: 45. Default: 30.

Raw schema

{
  "type": "object",
  "properties": {
    "runId": {
      "type": "string",
      "minLength": 1,
      "description": "The ID of the Actor run."
    },
    "waitSecs": {
      "default": 30,
      "description": "Maximum seconds to wait for the run to reach a terminal state (SUCCEEDED, FAILED, ABORTED, TIMED-OUT).\n0 returns immediately with the current status. Cap: 45. Default: 30.",
      "type": "integer",
      "minimum": 0,
      "maximum": 45
    }
  },
  "required": [
    "runId"
  ],
  "additionalProperties": false
}

get-dataset-items

Get items (rows) from a dataset — the output/results produced by an Actor run. Returns the rows themselves, not dataset metadata, counts, or a schema. When the user provides a datasetId and asks to retrieve results, output, data, or rows, call this tool directly. Default limit is 20. Use clean=true to skip empty items and hidden fields. USAGE: - Use when you need to read data from a dataset (all items or only selected fields). USAGE EXAMPLES: - user_input: Retrieve results from dataset abc123 - user_input: Get only metadata.url and title from dataset username~my-dataset This tool requires an x402 payment. Include a valid x402 payment signature in the request metadata (_meta["x402/payment"]). Your MCP client must support the x402 payment protocol.

Parameters8

datasetId	string	required	Dataset ID or username~dataset-name.
clean	boolean	optional	If true, returns only non-empty items and skips hidden fields (starting with #). Shortcut for skipHidden=true and skipEmpty=true.
offset	number	optional	Number of items to skip at the start. Default is 0.
limit	integer	optional	Maximum number of items to return. Default is 20.
fields	string	optional	Comma-separated list of fields to include in results. Fields in output are sorted as specified. Use dot notation for nested objects (e.g. "metadata.url"); the server auto-flattens parent prefixes.
omit	string	optional	Comma-separated list of fields to exclude from results.
desc	boolean	optional	If true, results are returned in reverse order (newest to oldest).
flatten	string	optional	Comma-separated list of fields to flatten (e.g. flatten="metadata" turns {"metadata":{"url":"x"}} into {"metadata.url":"x"}). Normally derived automatically from dot-notation in `fields`; specify only as a diagnostic override.

Raw schema

{
  "type": "object",
  "properties": {
    "datasetId": {
      "type": "string",
      "minLength": 1,
      "description": "Dataset ID or username~dataset-name."
    },
    "clean": {
      "description": "If true, returns only non-empty items and skips hidden fields (starting with #). Shortcut for skipHidden=true and skipEmpty=true.",
      "type": "boolean"
    },
    "offset": {
      "description": "Number of items to skip at the start. Default is 0.",
      "type": "number"
    },
    "limit": {
      "description": "Maximum number of items to return. Default is 20.",
      "type": "integer",
      "minimum": 1,
      "maximum": 9007199254740991
    },
    "fields": {
      "description": "Comma-separated list of fields to include in results. Fields in output are sorted as specified. Use dot notation for nested objects (e.g. \"metadata.url\"); the server auto-flattens parent prefixes.",
      "type": "string"
    },
    "omit": {
      "description": "Comma-separated list of fields to exclude from results.",
      "type": "string"
    },
    "desc": {
      "description": "If true, results are returned in reverse order (newest to oldest).",
      "type": "boolean"
    },
    "flatten": {
      "description": "Comma-separated list of fields to flatten (e.g. flatten=\"metadata\" turns {\"metadata\":{\"url\":\"x\"}} into {\"metadata.url\":\"x\"}). Normally derived automatically from dot-notation in `fields`; specify only as a diagnostic override.",
      "type": "string"
    }
  },
  "required": [
    "datasetId"
  ],
  "additionalProperties": false
}

get-key-value-store-record

Get the value stored under a specific key in a key-value store — a single record, not a listing of all keys. Requires the exact key name. The response preserves the original Content-Encoding; most clients handle decompression automatically. USAGE: - Use when you need to retrieve a specific record (JSON, text, or binary) from a store. USAGE EXAMPLES: - user_input: Get record INPUT from store abc123 - user_input: Get record data.json from store username~my-store This tool requires an x402 payment. Include a valid x402 payment signature in the request metadata (_meta["x402/payment"]). Your MCP client must support the x402 payment protocol.

Parameters2

keyValueStoreId	string	required	Key-value store ID or username~store-name.
recordKey	string	required	Key of the record to retrieve.

Raw schema

{
  "type": "object",
  "properties": {
    "keyValueStoreId": {
      "type": "string",
      "minLength": 1,
      "description": "Key-value store ID or username~store-name."
    },
    "recordKey": {
      "type": "string",
      "minLength": 1,
      "description": "Key of the record to retrieve."
    }
  },
  "required": [
    "keyValueStoreId",
    "recordKey"
  ],
  "additionalProperties": false
}

abort-actor-run

Abort an Actor run that is currently starting or running. For runs with status SUCCEEDED, FAILED, ABORTING, ABORTED, or TIMED-OUT, this call has no effect. The results will include the updated run details after the abort request. USAGE: - Use when you need to stop a run that is taking too long or misconfigured. USAGE EXAMPLES: - user_input: Abort run y2h7sK3Wc - user_input: Gracefully abort run y2h7sK3Wc This tool requires an x402 payment. Include a valid x402 payment signature in the request metadata (_meta["x402/payment"]). Your MCP client must support the x402 payment protocol.

Parameters2

runId	string	required	The ID of the Actor run to abort.
gracefully	boolean	optional	If true, the Actor run will abort gracefully with a 30-second timeout.

Raw schema

{
  "type": "object",
  "properties": {
    "runId": {
      "type": "string",
      "minLength": 1,
      "description": "The ID of the Actor run to abort."
    },
    "gracefully": {
      "description": "If true, the Actor run will abort gracefully with a 30-second timeout.",
      "type": "boolean"
    }
  },
  "required": [
    "runId"
  ],
  "additionalProperties": false
}

industrial_platform--sitemap-intelligence

This tool calls the Actor "industrial_platform/sitemap-intelligence" and retrieves its output results. Actor description: Discover and expand sitemaps into crawl URLs. Price: $0.0001 per emitted URL. This tool requires an x402 payment. Include a valid x402 payment signature in the request metadata (_meta["x402/payment"]). Your MCP client must support the x402 payment protocol.

Parameters5

start_urls	array	required	REQUIRED Domains, website URLs, or direct sitemap URLs. Domains use robots.txt discovery plus /sitemap.xml fallback. Example values: ["https://www.google.com/gmail/sitemap.xml"]
timeout_seconds	integer	optional	Maximum time allowed for each robots.txt or sitemap request. Example values: 30
max_sitemaps	integer	optional	Maximum sitemap and sitemap-index documents processed during one run. Example values: 100
max_urls	integer	optional	Maximum unique URL rows emitted and charged during one run. Example values: 50000
waitSecs	integer	optional	Max seconds (0–45, default 30) to cap the wait for the Actor run to reach terminal state. For long-running Actors the response returns at the cap with the current run status; follow `nextStep` to poll via get-actor-run. Set to 0 to fire-and-forget.

Raw schema

{
  "type": "object",
  "properties": {
    "start_urls": {
      "title": "Domains or sitemap URLs",
      "description": "**REQUIRED** Domains, website URLs, or direct sitemap URLs. Domains use robots.txt discovery plus /sitemap.xml fallback.\nExample values: [\"https://www.google.com/gmail/sitemap.xml\"]",
      "type": "array",
      "prefill": [
        "https://www.google.com/gmail/sitemap.xml"
      ],
      "items": {
        "type": "string",
        "maxLength": 2048
      },
      "examples": [
        "https://www.google.com/gmail/sitemap.xml"
      ]
    },
    "timeout_seconds": {
      "title": "Fetch timeout",
      "description": "Maximum time allowed for each robots.txt or sitemap request.\nExample values: 30",
      "type": "integer",
      "default": 30,
      "examples": [
        30
      ]
    },
    "max_sitemaps": {
      "title": "Maximum sitemap files",
      "description": "Maximum sitemap and sitemap-index documents processed during one run.\nExample values: 100",
      "type": "integer",
      "default": 100,
      "examples": [
        100
      ]
    },
    "max_urls": {
      "title": "Maximum URLs",
      "description": "Maximum unique URL rows emitted and charged during one run.\nExample values: 50000",
      "type": "integer",
      "default": 50000,
      "examples": [
        50000
      ]
    },
    "waitSecs": {
      "type": "integer",
      "minimum": 0,
      "maximum": 45,
      "default": 30,
      "description": "Max seconds (0–45, default 30) to cap the wait for the Actor run to reach terminal state. For long-running Actors the response returns at the cap with the current run status; follow `nextStep` to poll via get-actor-run. Set to 0 to fire-and-forget."
    }
  },
  "required": [
    "start_urls"
  ],
  "title": "Sitemap Intelligence input",
  "additionalProperties": false,
  "$id": "https://apify.com/mcp/industrial_platform--sitemap-intelligence/schema.json"
}

README.md13 KBView on GitHub

Industrial Platform Agent Tools

Network trust governance

Industrial Platform uses the Industrial Platform Trust Protocol (IPTP) to separate Base USDC settlement, optional wallet-control verification, independent commercial attribution, and demand-gated factory activation. Identity verification is never a prerequisite to an ordinary x402 purchase. Current implementation is partial; see the protocol for evidence and enforcement boundaries.

Direct recurring x402 MCP

For recurring autonomous workloads, use the direct x402-native MCP first:

text

https://x402-mcp-gateway-production.up.railway.app/mcp

Literal recurring tool names are optimized for auto-import and delegation:

monitor_wallet, transaction_status, watch_transaction, treasury_snapshot, pretrade_context, crypto_candles, crypto_price, crypto_book, market_snapshot, and monitor_webpage_change.

These map to the same canonical paid HTTP workloads, so importer compatibility increases without splitting HTTP settlement history across duplicate routes.

Open-source integration toolkit for connecting AI agents, MCP clients, autonomous workflows, agent orchestrators, and developer applications to Industrial Platform machine-callable services.

Industrial Platform provides specialized paid utilities that other agents can discover and invoke instead of rebuilding research, monitoring, extraction, and document-processing capabilities inside every workflow.

Install in your agent IDE

For the lowest-friction recurring workflow, install Web Metadata + Web Change first. Both currently complete in about 3 seconds median and cost $0.01 per successful unit.

Cursor: one-click install
VS Code / GitHub Copilot: one-click install
GitHub Copilot CLI: copilot mcp add --transport http industrial-platform-fast-web 'https://mcp.apify.com?tools=industrial_platform/web-metadata-intelligence,industrial_platform/web-change-intelligence&payment=x402'
Claude Code: claude mcp add --transport http industrial-platform-fast-web 'https://mcp.apify.com?tools=industrial_platform/web-metadata-intelligence,industrial_platform/web-change-intelligence&payment=x402'
Cline: cline mcp install industrial-platform-fast-web --transport http 'https://mcp.apify.com?tools=industrial_platform/web-metadata-intelligence,industrial_platform/web-change-intelligence&payment=x402'

See INSTALL.md for the full installer matrix, recurring-use patterns, and the full portfolio endpoint.

One MCP endpoint for the full portfolio

text

https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402

This hosted MCP endpoint exposes the six currently runnable Industrial Platform services through one direct x402 integration. Tool discovery is free; autonomous callers pay only when invoking a paid Actor.

Current services

Service	Actor	Primary purpose	Billing event	Current price
AI Research Brief	`industrial_platform/research-brief-agent`	Current, cited multi-source research	`research-brief-completed`	$1.99
Web Change Intelligence	`industrial_platform/web-change-intelligence`	Detect deterministic changes in public web content	`page-comparison`	$0.01
Web Metadata Intelligence	`industrial_platform/web-metadata-intelligence`	Extract page metadata, OG/Twitter, JSON-LD, headings	`metadata-extraction`	$0.01
Article Content Intelligence	`industrial_platform/article-content-intelligence`	Extract clean article text for RAG/research pipelines	`article-extraction`	$0.002
PDF Text Intelligence	`industrial_platform/pdf-text-intelligence`	Extract PDF text, per-page text, and metadata	`pdf-extraction`	$0.0015
Sitemap Intelligence	`industrial_platform/sitemap-intelligence`	Discover and expand sitemaps into clean URL rows	`sitemap-url`	$0.0001

Live pricing and execution eligibility should be verified against Apify Store before paid execution.

Official MCP Registry

Industrial Platform is published in the official MCP Registry as:

text

io.github.industrial-platform-ai/industrial-platform

Registry publication is automated on merge.

The Registry listing points to the same hosted six-service Apify MCP bundle used by this repository.

Individual MCP Registry entries

For search-oriented discovery, each paid service also has its own active MCP Registry entry:

Service	MCP Registry name
Research Brief	`io.github.industrial-platform-ai/research-brief`
Web Change	`io.github.industrial-platform-ai/web-change`
Web Metadata	`io.github.industrial-platform-ai/web-metadata`
Article Extractor	`io.github.industrial-platform-ai/article-extractor`
PDF Text	`io.github.industrial-platform-ai/pdf-text`
Sitemap	`io.github.industrial-platform-ai/sitemap`

These individual entries point to the same Apify-hosted Actor tools as the portfolio bundle and improve discovery for capability-specific MCP searches.

Machine-readable discovery

The repository publishes:

text

catalog.json
manifests/mcp-bundle.json
examples/mcp/remote-server.json

Service catalog

catalog.json is the portfolio-level machine-readable catalog. It contains Actor IDs, Store URLs, billing events, current prices, and common delegation targets.

MCP bundle manifest

manifests/mcp-bundle.json defines the hosted MCP endpoint and the Actors included in the bundle.

MCP client configuration

examples/mcp/remote-server.json can be used as a starting point for MCP clients that accept remote server configuration.

Direct x402 framework integrations

Industrial Platform also exposes a direct Base x402 gateway for autonomous callers that do not need the Apify MCP layer:

text

https://x402-gateway-production-1f21.up.railway.app

Machine-readable discovery:

text

https://x402-gateway-production-1f21.up.railway.app/.well-known/x402
https://x402-gateway-production-1f21.up.railway.app/openapi.json
https://x402-gateway-production-1f21.up.railway.app/llms.txt

Direct x402-native MCP:

text

https://x402-mcp-gateway-production.up.railway.app/mcp

The MCP server exposes the same paid tool catalog over Streamable HTTP and performs x402 negotiation at tool-call time. Tool discovery is free; a caller wallet signs the advertised Base USDC requirement when it invokes a paid tool.

Ready-to-copy adapters and workflows are included for:

Minimal official x402 caller — official x402 client, automatic 402 → payment → retry, with a hard per-payment ceiling
Coinbase AgentKit — Base wallet + AgentKit's native x402 provider, including a dry-run-by-default 1-atomic-USDC clean-room canary
ElizaOS — native @elizaos/plugin-wallet x402 client with explicit budget controls and payment disabled by default
LangChain / LangGraph
CrewAI
LlamaIndex
SEO crawler workflow
Crypto intelligence workflow

The examples use the caller's own funded Base wallet, negotiate x402 automatically, and preserve framework-level tool choice rather than forcing Industrial Platform calls.

Agent-to-agent paid quick start

bash

npm install -g @apify/mcpc
mcpc x402 init
mcpc connect "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence,industrial_platform/sitemap-intelligence&payment=x402" @industrial-platform --x402
mcpc @industrial-platform tools-list

Fund the generated wallet with USDC on Base before paid calls. Product #7 (Link Intelligence) is not included here until its public/agentic publication is confirmed.

Quick start

MCP

Configure one remote MCP server:

json

{
  "mcpServers": {
    "industrial-platform": {
      "url": "https://mcp.apify.com?tools=industrial_platform/research-brief-agent,industrial_platform/web-change-intelligence,industrial_platform/web-metadata-intelligence,industrial_platform/article-content-intelligence,industrial_platform/pdf-text-intelligence"
    }
  }
}

Compatible clients can then discover the exposed Actor tools from that server.

REST / Apify API

Every service can also be invoked directly through Apify's Actor API.

General Actor identifier form:

text

industrial_platform~ACTOR-NAME

Example:

text

industrial_platform~pdf-text-intelligence

Do not commit API tokens, wallet credentials, secrets, or private keys to this repository.

When should an agent delegate?

A calling agent should prefer a specialized Industrial Platform service when delegation is cheaper, simpler, or operationally safer than rebuilding the capability internally.

Examples:

text

Need current multi-source research with citations
→ research-brief-agent

text

Need to know whether a monitored page changed
→ web-change-intelligence

text

Need Open Graph, JSON-LD, canonical, robots, and headings
→ web-metadata-intelligence

text

Need clean article text for a RAG or research pipeline
→ article-content-intelligence

text

Need text and metadata from a public PDF
→ pdf-text-intelligence

Portfolio architecture

text

developer / agent / orchestrator
             ↓
   Industrial Platform MCP
             ↓
  specialized machine service
             ↓
   structured result / event
             ↓
calling system continues workflow

Execution and monetization currently run through Apify.

Authentication and machine payments

Paid execution requires a supported authenticated or agentic-payment path.

Depending on the integration, this may use:

Apify OAuth
Apify API tokens
Apify-supported agentic payment infrastructure

Store eligibility and pricing can change. Calling systems should inspect current Actor metadata before executing paid services.

Repository structure

text

industrial-platform-agent-tools/
│
├── README.md
├── catalog.json
├── AUTONOMOUS_REVENUE_INITIATIVE.md
│
├── actors/
│   ├── web-change-intelligence/
│   ├── web-metadata-intelligence/
│   ├── article-content-intelligence/
│   ├── pdf-text-intelligence/
│   ├── sitemap-intelligence/
│   └── link-intelligence/
│
├── packages/
│   ├── research-mcp/
│   └── change-mcp/
│
├── manifests/
│   └── mcp-bundle.json
│
├── examples/
│   ├── mcp/
│   ├── curl/
│   ├── python/
│   ├── javascript/
│   ├── openai-agents/
│   └── microsoft-agent-framework/
│
└── scripts/
    └── apify/

Revenue initiative

The repository contains automation for:

testing and deployment
Apify runtime configuration
pay-per-event pricing
Store / agentic-payment eligibility
external paid-event telemetry
milestone issues at 1, 10, and 100 external paid events

See:

text

AUTONOMOUS_REVENUE_INITIATIVE.md

Security principles

Industrial Platform services should:

block private/local-network fetch targets when accepting arbitrary URLs
bound response sizes, timeouts, and concurrency
use structured machine-readable outputs
avoid silently fabricating unavailable data
charge successful results rather than failed fetches where practical
keep credentials in secret stores rather than source code

License

Integration examples and open-source code in this repository are provided under the MIT License unless otherwise noted.

Industrial Platform

Organization:

text

industrial-platform-ai

Apify namespace:

text

industrial_platform

Machine-readable catalog:

text

catalog.json

Install

Tools (5)

get-actor-runget-dataset-itemsget-key-value-store-recordabort-actor-runindustrial_platform--sitemap-intelligence

Remote endpoint

Streamable HTTP

Hosted server - connect over the network, no local install.

url

https://mcp.apify.com?tools=industrial_platform/sitemap-intelligence&payment=x402

Add to VS Code Add to Cursor

claude_desktop_config.json

json

{
  "mcpServers": {
    "sitemap": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.apify.com?tools=industrial_platform/sitemap-intelligence&payment=x402"
      ]
    }
  }
}

Desktop config is stdio-only; this bridges via mcp-remote. Native remote: Settings > Connectors.

io.github.industrial-platform-ai/sitemap (MCP Server)

🛠️ Key Features

🚀 Use Cases

⚡ Developer Benefits

⚠️ Limitations

Topics

get-actor-run

get-dataset-items

get-key-value-store-record

abort-actor-run

industrial_platform--sitemap-intelligence

Industrial Platform Agent Tools

Network trust governance

Direct recurring x402 MCP

Install in your agent IDE

One MCP endpoint for the full portfolio

Current services

Official MCP Registry

Individual MCP Registry entries

Machine-readable discovery

Service catalog

MCP bundle manifest

MCP client configuration

Direct x402 framework integrations

Agent-to-agent paid quick start

Quick start

MCP

REST / Apify API

When should an agent delegate?

Portfolio architecture

Authentication and machine payments

Repository structure

Revenue initiative

Security principles

License

Industrial Platform

Install

Remote endpoint

Related servers