Model Context Protocol (MCP) Server: io.github.alphaparkinc/genpark-bloom-filter-url-dedup-crawler-skill
This MCP server provides an in-memory Bloom filter used for URL and document hash deduplication. It prevents redundant web crawler ingestion by recording previously seen values, supporting deduplication in data-pipeline workflows.
๐ ๏ธ Key Features
- In-memory Bloom filter
- URL deduplication
- Document hash deduplication
- Probabilistic data structure approach
๐ Use Cases
- Web crawler ingestion deduplication
- RAG ingestion pipelines
- Data pipelines requiring fast, in-memory duplicate detection
โก Developer Benefits
- Supports high-performance deduplication via a Bloom filter
- Fits into MCP-based agent skills for search and ingestion flows
โ ๏ธ Limitations
- Bloom filters are probabilistic (deduplication is not guaranteed exact)