Agentโ™ฅ๏ธŽAge
Catalog

GenPark Bloom Filter URL Dedup Crawler

Official

by alphaparkinc ยท Python

In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge

Model Context Protocol (MCP) Server: io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill

This MCP server provides an in-memory Bloom filter used for URL and document hash deduplication. It prevents redundant web crawler ingestion by recording previously seen values, supporting deduplication in data-pipeline workflows.

๐Ÿ› ๏ธ Key Features

  • In-memory Bloom filter
  • URL deduplication
  • Document hash deduplication
  • Probabilistic data structure approach

๐Ÿš€ Use Cases

  • Web crawler ingestion deduplication
  • RAG ingestion pipelines
  • Data pipelines requiring fast, in-memory duplicate detection

โšก Developer Benefits

  • Supports high-performance deduplication via a Bloom filter
  • Fits into MCP-based agent skills for search and ingestion flows

โš ๏ธ Limitations

  • Bloom filters are probabilistic (deduplication is not guaranteed exact)

Topics

agent-skillsbloom-filterdata-pipelinehigh-performancemcpprobabilistic-data-structurepython-standard-libraryrag-ingestionsearchurl-deduplicationweb-crawler