Golden Suite

Zero-config entity resolution that scales — dedupe & match messy records from a laptop CSV to 100M+ rows. No training data, no tuning.

The headline package, GoldenMatch, does the matching — fuzzy + exact + probabilistic (Fellegi-Sunter) + LLM — and beats hand-tuned Splink out of the box (96.4% F1 on DBLP-ACM), identical in Python, edge-safe TypeScript, and SQL. Around it sits a full data-quality suite: GoldenCheck profiles, GoldenFlow standardizes, GoldenAnalysis reports, GoldenPipe orchestrates, and InferMap maps schemas — with a Rust extension layer for Postgres / DuckDB and optional WebAssembly acceleration behind the TS ports.

Made for GraphRAG, too — entity resolution is the stage knowledge-graph pipelines do worst (the same entity scatters across documents as duplicate surface forms). GoldenMatch drops into neo4j-graphrag / LlamaIndex / Graphiti as the resolution stage (goldenmatch-kg), or builds a knowledge graph straight from text with that resolution at its core (goldengraph). Both in early access. → Knowledge graphs

Verified at scale: 100,000,000 records deduped in 9.2 min on a Ray cluster — recall-complete across any partitioning, 0.36 GB driver footprint.

# Dedupe a CSV in 30 seconds — zero config, writes <timestamp>_golden.csv to the
# current directory. Add --tui to review interactively, --output-all for every artifact.
pip install goldenmatch && goldenmatch dedupe customers.csv

# From Python — zero-config, returns golden records
python -c "import goldenmatch as gm; gm.dedupe('customers.csv').golden.write_csv('deduped.csv')"

# TypeScript / Edge runtimes
npm install goldenmatch

# The WHOLE suite (Check + Flow + Match + Analysis + Pipe + InferMap) + native acceleration, one line:
pip install golden-suite

v2.4.0 — The healing loop, now default-on across every surface — every dedupe_df run surfaces ranked, self-verified config-suggestions on result.suggestions when there's headroom (free on a healthy run, no second pipeline pass). dedupe_df(suggest=True) returns verified suggestions; heal=True applies them and re-runs, returning the healed result.config + result.heal_trail. Available across Python, CLI (--suggest / --heal), MCP, A2A, REST, web, the TUI, and the edge-safe TypeScript port via WebAssembly. Needs goldenmatch[native]; degrades gracefully without it. Kill-switch GOLDENMATCH_SUGGEST_ON_DEDUPE=0.

v2.3.0 — Auto-enabled semantic blocking, now default-on — text-heavy data automatically routes to SimHash-over-embeddings blocking when an embedder is reachable (a byte-identical no-op otherwise). Plus pluggable pgvector / DuckDB-HNSW vector-index backends and opt-in Fellegi-Sunter routing for no-strong-identifier datasets (GOLDENMATCH_AUTOCONFIG_ROUTE_PROBABILISTIC=1).

v2.2.0 — Semantic blocking — an opt-in recall lever for abbreviations and aliases. dedupe_df(semantic_blocking=...) unions extra candidate sources (initialism/abbreviation blocking, a business-alias canonical-form table, and an embedding ANN pass) into the pipeline. Off by default; on the abbreviation-heavy benchmark it adds +5.3pp recall at zero precision cost.

Why a suite?

Each tool stands alone, but they compose into a single pipeline:

flowchart LR
    raw([raw rows])
    golden([golden records])

    subgraph orchestration ["GoldenPipe orchestrates"]
        direction LR
        infermap[InferMap]
        goldencheck[GoldenCheck]
        goldenflow[GoldenFlow]
        goldenmatch[GoldenMatch]
        infermap --> goldencheck --> goldenflow --> goldenmatch
    end

    raw --> infermap
    goldenmatch --> golden

• Zero-config defaults that admit when they're unsure — every step has a self-verifying preflight + postflight; results carry an inspectable report instead of failing silently.
• 96.4% F1 on DBLP-ACM out of the box for entity resolution — and the opt-in Fellegi-Sunter engine beats hand-rolled, expert-tuned Splink head-to-head on every dataset Splink scores (historical_50k pairwise F1 0.778 vs 0.757, cluster-level B³ 0.844 vs 0.789; one shared evaluator, reproducible bake-off).
• Learning Memory — corrections persist across runs and re-anchor across row reorders, so the system stops needing the same correction twice (GoldenMatch v1.6.0; off by default).
• Identity Graph — a durable graph layer above run-local clusters: stable entity_ids that survive across runs, an append-only event log, and create / absorb / merge / split semantics, surfaced on the CLI, REST, MCP, and SQL interfaces (the Identity Graph v2 feature, shipped in GoldenMatch v1.15).
• Privacy-preserving record linkage — match across organizations without sharing raw data (PPRL, 92.4% F1 on FEBRL4).
• AI-native by design — every package ships an MCP server, a REST API, and an A2A agent surface. 70+ MCP tools across the suite, including auto_configure + controller_telemetry for auto-config introspection.
• AutoConfigController visible everywhere (v1.7-v1.12 surface-parity arc) — web ControllerPanel, TUI Ctrl+A, CLI goldenmatch autoconfig, REST /autoconfig + /controller/telemetry, Postgres goldenmatch_autoconfig + gm_telemetry, DuckDB UDFs, MCP/A2A telemetry tools. One JSON shape across every interface.
• Polyglot parity — the full suite ships on npm (goldenmatch, goldencheck, goldenflow, goldenanalysis, infermap, goldenpipe) alongside PyPI; the TypeScript and Python implementations track the same outputs to 4-decimal precision via a cross-language parity harness.
• Edge-safe, with optional native speed — the TypeScript cores are dependency-free and node:*-free, so they run in browsers, Cloudflare Workers, Vercel Edge, and Deno. An opt-in WebAssembly backend (await enableWasm() / enableAnalysisWasm() / enableSuggestWasm() — the last bringing the config-suggestion "healer" to TS) swaps in the same pyo3-free Rust kernels the Python wheels and the SQL UDFs use — pure-TS stays the default and the byte-identical fallback, so default users download zero wasm bytes.
• SQL-native, both engines at parity — the same functions run inside PostgreSQL (pgrx extension) and DuckDB: dedupe / match / score / auto-config + telemetry / identity graph, plus data profiling, evaluate, Fellegi-Sunter probabilistic scoring, and GoldenFlow transforms.
• Production paths — Postgres sync, daemon mode, lineage tracking, review queues, dbt integration, GitHub Actions, and a Rust extension layer for Postgres / DuckDB.

The healing loop

GoldenMatch's core workflow is a loop, not a one-shot:

1. Zero-config first pass — dedupe_df(df) runs with no rules and no training data. Auto-config picks a defensible config and you get good results immediately.
2. You get the config it chose — the chosen config comes back on result.config: inspectable, diffable, versionable. Never a black box.
3. The healer suggests tweaks — every dedupe_df(df) run checks a free signal and, when there's headroom, attaches ranked, explainable, self-verified edits to result.suggestions (no extra cost on a healthy result). Each suggestion is kept only if it doesn't worsen an unsupervised health proxy, so a tweak never makes results worse.
4. You apply the tweaks — dedupe_df(df, heal=True) applies them and re-runs in one call (returning the healed result.config + a result.heal_trail); or take the wheel with apply_suggestion.
5. Results improve. Repeat — loop until the healer goes quiet.

Zero-config gets you most of the way in one pass; the healing loop closes the gap to an expert-tuned config without you having to be the expert.

Status: the healer is wired into the default pipeline on every surface — Python (dedupe_df(df, suggest=True) for verified suggestions, heal=True for the full apply-and-re-run loop, or review_config for direct control), CLI (--suggest / --heal), MCP & A2A (review_config), REST, web, and the TUI — plus the edge-safe TypeScript port via WebAssembly (enableSuggestWasm()). It needs goldenmatch[native] and degrades gracefully without it (attaches nothing, never errors). Kill-switch: GOLDENMATCH_SUGGEST_ON_DEDUPE=0. Full details: config-suggestions.

The Suite

Headline pitch and the deepest docs live in packages/python/goldenmatch/README.md (~1,300 lines, full feature list, CLI, architecture, benchmarks).

Knowledge graphs

Entity resolution is the stage most GraphRAG pipelines do badly — duplicate surface forms of the same entity scatter across documents. Two new packages put GoldenMatch's resolution there:

Choose your path

Quick examples

Python — dedupe in 30 seconds

import goldenmatch as gm

# Zero-config
result = gm.dedupe("customers.csv")
print(result)  # DedupeResult(records=5000, clusters=847, match_rate=12.0%)
result.golden.write_csv("deduped.csv")

# Or be explicit
result = gm.dedupe("customers.csv",
    exact=["email"],
    fuzzy={"name": 0.85, "zip": 0.95},
    blocking=["zip"],
    threshold=0.85)

TypeScript — edge-safe core

import { dedupe } from "goldenmatch";

const result = dedupe(rows, {
  fuzzy: { name: 0.85 },
  blocking: ["zip"],
  threshold: 0.85,
});
console.log(result.stats);  // { totalRecords, totalClusters, matchRate, ... }

Runs in browsers, Vercel Edge, Cloudflare Workers, Deno — and optionally swaps in the Rust score-core kernel via await enableWasm(). ~940 tests, strict TypeScript (noUncheckedIndexedAccess, exactOptionalPropertyTypes).

Web workbench — browser UI for matching

pip install 'goldenmatch[web]'
goldenmatch serve-ui my-project   # opens http://localhost:5050

Edit rules with live validation, preview against a sampled slice, label pairs (mirrored into Learning Memory automatically), compare runs (CCMS), sweep parameters, browse the corrections store. Single-process localhost workbench shipped as the optional [web] extra.

Composed pipeline

import goldenpipe as gp

pipeline = gp.Pipeline.from_yaml("pipeline.yaml")  # check → flow → match
result = pipeline.run("customers.csv")
result.report.write_html("report.html")

More: examples/ has runnable demos for every Suite scenario: Python (quickstart, full pipeline, customer 360, PPRL, review workflow, MCP client) · TypeScript (quickstart, Vercel Edge route, MCP client) · Airflow DAGs (12 production-shaped pipelines).

Use cases (real-world pipelines)

Reproducible end-to-end pipelines running GoldenMatch on public data at scale, each with measured headline numbers vs baselines:

• goldenmatch-shell-company-network — investigative ER across ICIJ Offshore Leaks + OpenSanctions + GLEIF + UK PSC + UK disqualified-directors. Confidence-weighted graph, structure mining, named investigative candidates. −62.5% analyst-hours to triage vs single-source baselines; +133% adversarial perturbation recovery.
• goldenmatch-vuln-attribution — cross-database ER on 6.1M OSS vulnerability records across 40 sources (OSV, GHSA, PyPA, RustSec, Go vulndb, EPSS, CISA KEV, CVE Project bulk). 6,126,895 records → 847,475 canonical vulns in ~5 minutes end-to-end on a single 64GB runner via the full Golden Suite (Check + Flow + Match + Pipe).
• goldenmatch-sanctions-reconciliation — cross-list coverage analysis on 85 public sanctions lists across 50+ jurisdictions via OpenSanctions, plus 10-year OFAC SDN history and PEP/crypto cross-analysis. Coverage-gap benchmark for any sanctions-screening vendor.

The whole suite in one line — golden-suite

Want everything, configured for speed, without hand-picking packages? The golden-suite meta-package pulls in the entire suite plus the native (Rust) acceleration in a single install:

pip install golden-suite

That one command installs GoldenMatch (entity resolution), GoldenCheck (data quality), GoldenFlow (transforms), GoldenAnalysis (reporting), GoldenPipe (orchestration), InferMap (schema mapping), and the goldenmatch-native / goldencheck-native / goldenflow-native / goldenanalysis-native compiled kernels — all pinned to compatible versions and defaulted to the perf-optimized configuration (native paths on, no env vars to set). The native wheels are hard dependencies on purpose: on a platform without a wheel the install fails loudly rather than silently running the slow pure-Python path.

golden-suite doctor      # verify every package + native kernel is importable and healthy
golden-suite optimize    # repair / re-enable the perf-optimized config if doctor finds a gap

pip install golden-suite[mcp]     # + the aggregator MCP server (every tool, one endpoint)
pip install golden-suite[agent]   # + GoldenPipe serving surfaces (A2A + REST + TUI)
pip install golden-suite[all]     # everything above

Prefer to install just one tool, or pick your own extras? Use the per-package installs below.

Install variants

GoldenMatch ships fat optional extras so you only pay for what you use:

pip install goldenmatch                    # core (CSV in, CSV out) + native acceleration on common platforms
pip install goldenmatch[native]            # back-compat alias; native is already default on common platforms
pip install goldenmatch[embeddings]        # + sentence-transformers, FAISS
pip install goldenmatch[llm]               # + Claude / OpenAI for LLM boost
pip install goldenmatch[postgres]          # + Postgres sync
pip install goldenmatch[snowflake]         # + Snowflake connector
pip install goldenmatch[bigquery]          # + BigQuery connector
pip install goldenmatch[databricks]        # + Databricks connector
pip install goldenmatch[salesforce]        # + Salesforce connector
pip install goldenmatch[duckdb]            # + DuckDB out-of-core backend
pip install goldenmatch[ray]               # + Ray distributed backend (50M+ rows)
pip install goldenmatch[quality]           # + GoldenCheck integration
pip install goldenmatch[transform]         # + GoldenFlow integration
pip install goldenmatch[mcp]               # + MCP server for Claude Desktop
pip install goldenmatch[agent]             # + A2A agent (aiohttp)
pip install goldenmatch[web]               # + localhost browser workbench (FastAPI + React)

goldenmatch setup    # interactive wizard: GPU, API keys, database

Sister packages compose: pip install goldenpipe[full] brings in Check + Flow + Match together.

Remote MCP Server

GoldenMatch is hosted as an MCP server on Smithery — connect from any MCP client without installing anything.

{
  "mcpServers": {
    "goldenmatch": {
      "url": "https://goldenmatch-mcp-production.up.railway.app/mcp/"
    }
  }
}

70+ MCP tools across the suite: deduplicate, match, explain, review, link privately, configure, scan quality, transform, synthesize golden records, analyze trends and regressions, and manage Learning Memory corrections.

Container images

Every Suite package ships as a multi-arch container image (linux/amd64 + linux/arm64) on GitHub Container Registry. Pull anonymously, no auth needed:

# One container, every Suite tool — the convenience option
docker run -p 8300:8300 ghcr.io/benseverndev-oss/goldensuite-mcp:latest

# Per-package containers — narrower deployments
docker run -p 8200:8200 ghcr.io/benseverndev-oss/goldenmatch-mcp:latest
docker run -p 8100:8100 ghcr.io/benseverndev-oss/goldencheck-mcp:latest
docker run -p 8150:8150 ghcr.io/benseverndev-oss/goldenflow-mcp:latest
docker run -p 8250:8250 ghcr.io/benseverndev-oss/goldenpipe-mcp:latest
docker run -p 8400:8400 ghcr.io/benseverndev-oss/infermap-mcp:latest

# Postgres + extension preinstalled
docker run -e POSTGRES_PASSWORD=secret ghcr.io/benseverndev-oss/goldenmatch-extensions:latest

Tags:

• :latest — current main
• :main-<sha7> — every push to main, immutable
• :vX.Y.Z and :vX.Y — pushed when a <package>-vX.Y.Z tag is created

See packages/python/goldensuite-mcp/README.md for the aggregator's tool-collision behaviour.

Airflow

13 drop-in DAGs at examples/airflow/, grouped by lifecycle stage:

TaskFlow API, Airflow 2.7+ (compatible with 3.x). Each DAG has tunable knobs at the top, idempotent retries, and is marker-protected against double-processing. Drop the file you want into your Airflow dags/ folder.

Repository layout

goldenmatch/
├── packages/
│   ├── python/
│   │   ├── goldenmatch/      # entity resolution — headline package
│   │   ├── goldencheck/      # data quality scanning
│   │   ├── goldenflow/       # transforms & standardizers
│   │   ├── goldenpipe/       # orchestrator
│   │   ├── infermap/         # schema mapping
│   │   └── goldenanalysis/   # cross-cutting analysis & reporting
│   ├── typescript/
│   │   ├── goldenmatch/      # full TS port (edge-safe core)
│   │   ├── goldencheck/      # TS implementation
│   │   ├── goldencheck-types/ # shared TS types
│   │   ├── goldenflow/       # TS transforms
│   │   ├── infermap/         # TS schema mapping
│   │   └── goldenanalysis/   # TS analysis & reporting (edge-safe + WASM)
│   ├── rust/
│   │   └── extensions/       # Postgres pgrx + DuckDB UDFs (own Cargo workspace)
│   ├── python/goldensuite-mcp/ # aggregator MCP server (one container, all tools)
│   ├── dbt/goldensuite/      # dbt package (materializations, tests, macros)
│   └── actions/goldencheck/  # GitHub Action
├── examples/
│   ├── python/               # 6 runnable Python scripts (quickstart → MCP)
│   ├── typescript/           # 3 TS scripts (quickstart, Vercel Edge, MCP)
│   └── airflow/              # 12 drop-in Airflow DAGs
├── docs/superpowers/         # design specs and implementation plans
├── justfile                  # install / test / lint / build, all languages
├── pyproject.toml            # uv workspace (root)
├── pnpm-workspace.yaml       # TypeScript pnpm workspace (Turborepo)
├── package.json              # root scripts + pnpm workspace root
└── .github/workflows/ci.yml

Workspaces (Cargo vs pnpm)

• Cargo — no root workspace. packages/rust/extensions/ is itself a Cargo workspace (the postgres crate is excluded for pgrx-specific build requirements). Cargo doesn't allow nested workspaces sharing members, so Cargo commands run from inside packages/rust/extensions/.
• TypeScript — a single pnpm workspace. packages/typescript/* form one pnpm + Turborepo workspace (see TypeScript dev setup). .npmrc pins node-linker=hoisted, giving a flat node_modules that avoids the Windows symlink issues an earlier per-package layout hit.

Build / test / lint everything

just install   # uv sync + per-package npm install + cargo fetch
just test      # all languages
just lint
just build

Reproducing benchmarks

Published GoldenMatch numbers (DQbench composite 91.04, DBLP-ACM 0.9641 F1, Febrl3 0.9443 F1, NCVR 0.9719 F1) map back to a single committed runner: scripts/run_benchmarks.py. See docs/reproducing-benchmarks.md for per-number commands, dataset URLs, expected output (with tolerance), variance notes (deterministic vs LLM-augmented), and a copy-pasteable one-click reproduction snippet for the DQbench composite. The same runner powers the weekly benchmarks.yml workflow.

Scale envelope

"How big can this handle?" is answered in docs/scale-envelope.md: per-backend ranges (Polars in-memory < 500K, DuckDB out-of-core 500K - 50M, Ray distributed >= 50M), block-size failure modes, candidate-pair math, and a single-page decision tree for picking a backend.

Verified at the top end: a full 100,000,000-row GoldenMatch dedupe on a 5-node Ray cluster (e2-standard-16, 80 CPU) in 9.2 min (554 s), 20,000,000 golden records recovered exactly, driver process peak 0.36 GB RSS — the default distributed path is now recall-complete (blocking-key shuffle scoring + a distributed randomized-contraction WCC), so duplicates merge correctly no matter how the input is partitioned, and it stays driver-collect-free end to end (#844). A faster per-partition path is available via GOLDENMATCH_DISTRIBUTED_BLOCK_SHUFFLE=0 (driver-collect-free, ~213 s on a 4-worker run) for inputs where duplicates already co-locate within partitions — but it under-merges when a cluster's members land in different input partitions, which is why recall-complete is the default. Recipe in packages/python/goldenmatch/configs/distributed-100m.yaml.

Contributing

• Feature work goes on feature/<name> branches; merge via squash PR.
• PR title format: feat: <description>, fix: <description>, docs: <description>.
• Tests must pass on all three languages where the change applies; the parity harness in packages/typescript/goldenmatch/tests/parity/ enforces 4-decimal-tolerance Python ↔ TypeScript scorer parity.
• See docs/superpowers/specs/ for design rationale on architectural decisions.

TypeScript dev setup (pnpm + Turborepo)

The TypeScript packages live in a single pnpm workspace orchestrated by Turborepo. From the repo root:

corepack enable                               # one-time, picks up pnpm@9.15.0 from package.json
pnpm install                                  # installs all workspace packages
pnpm turbo run build test typecheck lint      # full pipeline (cached after first run)
pnpm --filter goldenmatch test                # single package

Windows: enable Developer Mode for pnpm. pnpm install creates symlinks under node_modules/. Settings → For Developers → Developer Mode → On. If you see EPERM: operation not permitted, symlink ... during install, Dev Mode is off.

If corepack enable fails (often needs an admin shell on Windows), the fallback is npm i -g pnpm@9.15.0 — functionally equivalent.

History

This repository was formed on 2026-05-01 by folding 8 sibling repos into the existing goldenmatch repo using git filter-repo. Full commit history is preserved for every source. See docs/superpowers/specs/2026-05-01-goldenmatch-monorepo-fold-in-design.md for the design rationale and docs/superpowers/plans/2026-05-01-goldenmatch-monorepo-fold-in.md for the step-by-step migration plan.

Author & License

Built by Ben Severn.

MIT — see LICENSE.