This MCP server provides tools for duplicate detection and entity resolution, targeting fast performance (“Find duplicate records in 30 seconds”) and out-of-the-box quality (“97.2% F1 out of the box”). It is described as “zero-config,” with focus on generating stable “golden entities” for customer identity.
Profile data, detect domain, recommend ER strategy
Parameters4
file_path
string
optional
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
auto_configure
Run AutoConfigController on a CSV; return the committed GoldenMatchConfig (incl. negative_evidence / Path Y when chosen) plus telemetry — stop_reason, health, decision trace, indicator column priors. Programmatic equivalent of `goldenmatch autoconfig`.
Parameters6
file_path
string
optional
constraints
object
optional
exclude_columns
array
optional
Column names to skip across GoldenMatch + GoldenFlow + auto-config. Optional. Layered with config.exclude_columns when both are set. force_include (env var) rescues from any opt-out path.
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string"
},
"constraints": {
"type": "object"
},
"exclude_columns": {
"type": "array",
"items": {
"type": "string"
},
"description": "Column names to skip across GoldenMatch + GoldenFlow + auto-config. Optional. Layered with config.exclude_columns when both are set. force_include (env var) rescues from any opt-out path.",
"default": []
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
controller_telemetry
Return the AutoConfigController telemetry from the most recent `auto_configure` or `agent_deduplicate` call in this MCP session. Same JSON shape as the web /api/v1/controller/telemetry endpoint.
Parameters
No parameters.
Raw schema
{
"type": "object",
"properties": {}
}
agent_deduplicate
Run full ER pipeline with confidence gating and reasoning
Parameters6
file_path
string
optional
config
object
optional
exclude_columns
array
optional
Column names to skip across GoldenMatch + GoldenFlow + auto-config. Optional. Layered with config.exclude_columns when both are set. force_include (env var) rescues from any opt-out path.
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string"
},
"config": {
"type": "object"
},
"exclude_columns": {
"type": "array",
"items": {
"type": "string"
},
"description": "Column names to skip across GoldenMatch + GoldenFlow + auto-config. Optional. Layered with config.exclude_columns when both are set. force_include (env var) rescues from any opt-out path.",
"default": []
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
agent_match_sources
Match two files with intelligent strategy selection
Parameters9
file_a
string
optional
file_b
string
optional
config
object
optional
exclude_columns
array
optional
Column names to skip across GoldenMatch + GoldenFlow + auto-config. Optional. Layered with config.exclude_columns when both are set. force_include (env var) rescues from any opt-out path.
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
ground_truth_content
string
optional
Alternative to ground_truth: base64/text bytes
ground_truth_name
string
optional
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string"
},
"ground_truth": {
"type": "string"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"ground_truth_content": {
"type": "string",
"description": "Alternative to ground_truth: base64/text bytes"
},
"ground_truth_name": {
"type": "string"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
suggest_pprl
Check if data needs privacy-preserving matching
Parameters4
file_path
string
optional
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
scan_quality
Run GoldenCheck data quality scan on a CSV file. Returns issues found (encoding errors, Unicode problems, format violations) without applying fixes. Requires goldencheck: pip install goldenmatch[quality]
Parameters5
file_path
string
optional
Path to the CSV file to scan
domain
string
optional
Optional domain hint (healthcare, finance, ecommerce)
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "Path to the CSV file to scan"
},
"domain": {
"type": "string",
"description": "Optional domain hint (healthcare, finance, ecommerce)"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
fix_quality
Run GoldenCheck scan and apply fixes to a CSV file. Returns the fixed data summary and a manifest of all fixes applied. Requires goldencheck: pip install goldenmatch[quality]
Parameters7
file_path
string
optional
Path to the CSV file to fix
fix_mode
string
optional
Fix aggressiveness: safe (conservative) or moderate (balanced). Default: safe
domain
string
optional
Optional domain hint (healthcare, finance, ecommerce)
output_path
string
optional
Optional path to save the fixed CSV. If omitted, returns summary only.
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "Path to the CSV file to fix"
},
"fix_mode": {
"type": "string",
"enum": [
"safe",
"moderate"
],
"description": "Fix aggressiveness: safe (conservative) or moderate (balanced). Default: safe",
"default": "safe"
},
"domain": {
"type": "string",
"description": "Optional domain hint (healthcare, finance, ecommerce)"
},
"output_path": {
"type": "string",
"description": "Optional path to save the fixed CSV. If omitted, returns summary only."
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
run_transforms
Run GoldenFlow data transforms on a CSV file. Normalizes phone numbers (E.164), dates (ISO), categorical spelling, and Unicode issues. Returns a manifest of transforms applied. Requires goldenflow: pip install goldenmatch[transform]
Parameters5
file_path
string
optional
Path to the CSV file to transform
output_path
string
optional
Optional path to save the transformed CSV. If omitted, returns summary only.
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "Path to the CSV file to transform"
},
"output_path": {
"type": "string",
"description": "Optional path to save the transformed CSV. If omitted, returns summary only."
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
sensitivity
Parameter-sensitivity analysis: sweep one or more config parameters across a range and report how stable the clustering is at each value (CCMS unchanged %). Use it to find robust thresholds. Auto-configures the file if no config is given.
Parameters7
file_path
string
optional
CSV/Parquet to analyze
sweep
array
required
Sweep specs as 'field:start:stop:step', e.g. 'threshold:0.70:0.95:0.05'. One or more.
config
string
optional
Optional config YAML path
sample_size
integer
optional
Optional: randomly sample N records before sweeping
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "CSV/Parquet to analyze"
},
"sweep": {
"type": "array",
"items": {
"type": "string"
},
"description": "Sweep specs as 'field:start:stop:step', e.g. 'threshold:0.70:0.95:0.05'. One or more."
},
"config": {
"type": "string",
"description": "Optional config YAML path"
},
"sample_size": {
"type": "integer",
"description": "Optional: randomly sample N records before sweeping"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": [
"sweep"
]
}
incremental
Match a batch of new records against an existing base dataset (without re-running the whole base). Returns matched (new_row_id, base_row_id, score) pairs plus counts. Auto-configures from the base file if no config is given.
Estimate match RECALL without ground truth (unsupervised). Treats each auto-configured matchkey/pass as a decorrelated system and uses capture-recapture over their overlaps to estimate how many true matches were missed. Returns a point estimate (a safe lower bound additionally needs a small labelled audit; see `goldenmatch evaluate --certify --audit-out`). Needs >=3 decorrelated systems.
Parameters4
file_path
string
optional
Dataset to dedupe + certify
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "Dataset to dedupe + certify"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": []
}
retrieve_similar
Semantic retrieval (#1089): return the records in a CSV most similar to a free-text query, ranked by cosine similarity. Embeds the chosen column and the query with the zero-config in-house embedder (no cloud/torch by default) and runs ANN search. The read side of the RAG entity-canonicalization epic -- fetch candidate records by query without running a full dedupe.
Parameters10
file_path
string
optional
CSV/Parquet corpus to search
query
string
required
Free-text query to search for
column
string
required
Column of the corpus to embed + search
k
integer
optional
Max records to return (default 20)
threshold
number
optional
Minimum cosine similarity in [-1, 1] (default 0.0)
model
string
optional
Embedder id (default 'inhouse' -- local, deterministic, no cloud/torch). Also 'all-MiniLM-L6-v2', a Vertex/OpenAI model, etc.
filters
object
optional
Optional {column: value} equality pre-filter applied before embedding
file_content
string
optional
Alternative to file_path: file bytes (base64 default, or raw with encoding='text')
filename
string
optional
Original filename when using file_content
encoding
string
optional
Encoding of *_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_path": {
"type": "string",
"description": "CSV/Parquet corpus to search"
},
"query": {
"type": "string",
"description": "Free-text query to search for"
},
"column": {
"type": "string",
"description": "Column of the corpus to embed + search"
},
"k": {
"type": "integer",
"description": "Max records to return (default 20)"
},
"threshold": {
"type": "number",
"description": "Minimum cosine similarity in [-1, 1] (default 0.0)"
},
"model": {
"type": "string",
"description": "Embedder id (default 'inhouse' -- local, deterministic, no cloud/torch). Also 'all-MiniLM-L6-v2', a Vertex/OpenAI model, etc."
},
"filters": {
"type": "object",
"description": "Optional {column: value} equality pre-filter applied before embedding"
},
"file_content": {
"type": "string",
"description": "Alternative to file_path: file bytes (base64 default, or raw with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename when using file_content"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of *_content (default base64)"
}
},
"required": [
"query",
"column"
]
}
upload_dataset
Upload a local file's bytes to the server and get back a server-side path to reuse across other tools (analyze_data, auto_configure, agent_deduplicate, ...). No hosting needed. Send base64 (default) or raw text via `encoding`. Uploaded files are ephemeral scratch, reaped after GOLDENMATCH_MCP_UPLOAD_TTL (default 24h); re-upload if you need a path older than that. Max size GOLDENMATCH_MCP_MAX_UPLOAD_BYTES (default 64MB) -- above it, pass a public http(s) URL as file_path instead.
Parameters3
file_content
string
required
File bytes, base64-encoded (or raw text with encoding='text')
filename
string
required
Original filename (extension preserved for format sniffing)
encoding
string
optional
Encoding of file_content (default base64)
Raw schema
{
"type": "object",
"properties": {
"file_content": {
"type": "string",
"description": "File bytes, base64-encoded (or raw text with encoding='text')"
},
"filename": {
"type": "string",
"description": "Original filename (extension preserved for format sniffing)"
},
"encoding": {
"type": "string",
"enum": [
"base64",
"text"
],
"description": "Encoding of file_content (default base64)"
}
},
"required": [
"file_content",
"filename"
]
}
list_corrections
List stored Learning Memory corrections, optionally filtered by dataset. Returns id_a, id_b, decision, source, trust, reason, matchkey_name, dataset, original_score, created_at.
Parameters2
dataset
string
optional
Optional dataset filter (e.g. file path).
path
string
optional
SQLite memory DB path. Default: .goldenmatch/memory.db
Add a Learning Memory correction. Two shapes:
- pair-level: decision='approve' or 'reject', requires id_a + id_b
- field-level (v1.18.2+): decision='field_correct', requires cluster_id + field_name + corrected_value
Source is 'agent' with trust=0.5 (lower than human steward 1.0). Pair (id_a, id_b) is canonicalized to (min, max) before storage.
Parameters11
id_a
integer
optional
Pair-level: first row id. Field-level: ignored.
id_b
integer
optional
Pair-level: second row id. Field-level: ignored.
cluster_id
integer
optional
Field-level: cluster_id the correction targets.
decision
string
required
field_name
string
optional
Field-level: the column being corrected.
original_value
string
optional
Field-level: the value build_golden_record chose.
corrected_value
string
optional
Field-level: the value the reviewer changed it to.
List all registered goldenmatch plugins by category. Includes the 22 v1.18.2 predefined plugins (numeric/format/business/aggregation) plus any user-registered plugins via entry-points or PluginRegistry.register_*(). Each entry includes name, source (builtin or user), category, and the first line of the merge docstring.
Force a MemoryLearner pass over accumulated corrections. Returns the list of LearnedAdjustments produced (matchkey_name, threshold, sample_size, learned_at). Requires >= 10 corrections per matchkey before threshold tuning fires; otherwise returns an empty list.
Parameters2
matchkey_name
string
optional
Optional: learn only for this matchkey.
path
string
optional
SQLite memory DB path. Default: .goldenmatch/memory.db
Raw schema
{
"type": "object",
"properties": {
"matchkey_name": {
"type": "string",
"description": "Optional: learn only for this matchkey."
},
"path": {
"type": "string",
"description": "SQLite memory DB path. Default: .goldenmatch/memory.db"
}
}
}
memory_stats
Return Learning Memory status: total correction count, last learn time, and current learned adjustments. Cheap; safe for status checks.
Parameters1
path
string
optional
SQLite memory DB path. Default: .goldenmatch/memory.db
Import corrections from a list of dicts (the exact shape memory_export returns). Upserts into the store: higher trust wins, same trust = latest wins. Returns the count imported.
Parameters2
corrections
array
required
Correction dicts, as returned by memory_export.
path
string
optional
SQLite memory DB path. Default: .goldenmatch/memory.db
Raw schema
{
"type": "object",
"properties": {
"corrections": {
"type": "array",
"items": {
"type": "object"
},
"description": "Correction dicts, as returned by memory_export."
},
"path": {
"type": "string",
"description": "SQLite memory DB path. Default: .goldenmatch/memory.db"
}
},
"required": [
"corrections"
]
}
identity_resolve
Resolve a record_id to its durable identity. Returns the full identity view (members, evidence edges, recent events) or null when no identity exists for that record.
Manually merge two identities. All records from `absorb_entity_id` are reassigned to `keep_entity_id`. The merge events are stamped with `actor`/`trust` provenance so the audit log records who merged these and on what authority.
Parameters6
keep_entity_id
string
required
absorb_entity_id
string
required
reason
string
optional
actor
string
optional
Principal making the change, e.g. 'agent:claude' or 'steward:alice'. Defaults to 'agent'.
trust
number
optional
Trust of the actor in [0,1]. Defaults by actor prefix (steward 1.0, agent 0.5).
path
string
optional
Raw schema
{
"type": "object",
"properties": {
"keep_entity_id": {
"type": "string"
},
"absorb_entity_id": {
"type": "string"
},
"reason": {
"type": "string"
},
"actor": {
"type": "string",
"description": "Principal making the change, e.g. 'agent:claude' or 'steward:alice'. Defaults to 'agent'."
},
"trust": {
"type": "number",
"description": "Trust of the actor in [0,1]. Defaults by actor prefix (steward 1.0, agent 0.5)."
},
"path": {
"type": "string"
}
},
"required": [
"keep_entity_id",
"absorb_entity_id"
]
}
identity_split
Split a subset of records off an identity into a brand-new identity. The original keeps the remaining records. The split events carry `actor`/`trust` provenance.
Parameters6
entity_id
string
required
record_ids
array
required
reason
string
optional
actor
string
optional
Principal making the change, e.g. 'agent:claude'. Defaults to 'agent'.
trust
number
optional
Trust of the actor in [0,1]. Default by actor prefix.
path
string
optional
Raw schema
{
"type": "object",
"properties": {
"entity_id": {
"type": "string"
},
"record_ids": {
"type": "array",
"items": {
"type": "string"
}
},
"reason": {
"type": "string"
},
"actor": {
"type": "string",
"description": "Principal making the change, e.g. 'agent:claude'. Defaults to 'agent'."
},
"trust": {
"type": "number",
"description": "Trust of the actor in [0,1]. Default by actor prefix."
},
"path": {
"type": "string"
}
},
"required": [
"entity_id",
"record_ids"
]
}
identity_claim
Claim a record into an identity, moving it out of any prior entity ('this record belongs to that identity'). Emits a provenance-stamped `claimed` event on both the gaining and losing entities.
Parameters6
entity_id
string
required
Entity to claim the record into
record_id
string
required
record id in `{source}:{source_pk}` form
reason
string
optional
actor
string
optional
Principal, e.g. 'agent:claude'. Defaults to 'agent'.
trust
number
optional
Trust in [0,1]. Default by actor prefix.
path
string
optional
Raw schema
{
"type": "object",
"properties": {
"entity_id": {
"type": "string",
"description": "Entity to claim the record into"
},
"record_id": {
"type": "string",
"description": "record id in `{source}:{source_pk}` form"
},
"reason": {
"type": "string"
},
"actor": {
"type": "string",
"description": "Principal, e.g. 'agent:claude'. Defaults to 'agent'."
},
"trust": {
"type": "number",
"description": "Trust in [0,1]. Default by actor prefix."
},
"path": {
"type": "string"
}
},
"required": [
"entity_id",
"record_id"
]
}
identity_resolve_conflict
Adjudicate a `conflicts_with` pair: 'same' keeps the entity intact, 'distinct' splits the second record out into a new identity, 'defer' only logs. Records a durable mediation verdict + event with actor/trust provenance, and stops the conflict re-surfacing in the open-conflicts queue.
Parameters9
record_a_id
string
required
record_b_id
string
required
resolution
string
required
reason
string
optional
dataset
string
optional
apply
boolean
optional
Act on the verdict (split on 'distinct'); false = log only.
actor
string
optional
Principal, e.g. 'steward:alice'. Defaults to 'agent'.
Export the append-only identity audit log in commit order: every event with actor / trust / timestamp / reason, so a reviewer can reconstruct exactly which actor changed what, when, and why. Optionally filtered by dataset / actor.
Anchor the append-only audit log with a tamper-evidence seal: a chained sha256 root over every event since the last seal. Cheap and idempotent (a no-op when nothing new has been logged). Run it periodically (or after a batch of stewardship actions) so the history becomes provably untampered. Optionally scoped to a dataset. Publish/mirror the returned root_hash to make tampering detectable by an external party.
Verify the append-only audit log against its seal chain. Replays the per-event content hashes and the seal roots to detect content edits, deletion, reordering, and insertion of any sealed event. Returns {ok, events_checked, seals_checked} plus the ids of any content mismatches / broken seals / missing sealed events. Optionally scoped to a dataset.
Fetch the full detail of one identity by entity_id: its member records, evidence edges, and recent event log. Returns {found: false} when no such entity exists.
MDM profile of one entity: record count + per-source breakdown, golden record, confidence, conflict count, canonical version (structural-event count), and first/last activity. Returns {found: false} when no such entity exists.
Graph-level summary / health stats: entities by status, total records, records-per-entity distribution, conflict total, source mix, and the largest entities. Optionally scoped to a dataset.
Flag config/env overrides that force a slow path (e.g. CLUSTERING_THRESHOLD=0 when the edge set fits driver RAM). ERROR at scale; would_refuse mirrors the runtime guard.
Match a single record against the loaded dataset in real-time. Paste a record's fields and instantly see if it matches any existing record. Uses the configured matchkeys, scorers, and thresholds. Example: {"name": "John Smith", "email": "john@test.com", "zip": "10001"}
Parameters3
record
object
required
Record fields to match against the dataset
threshold
number
optional
Minimum score to consider a match (default: use config threshold)
top_k
integer
optional
Max matches to return (default 5)
Raw schema
{
"type": "object",
"properties": {
"record": {
"type": "object",
"description": "Record fields to match against the dataset"
},
"threshold": {
"type": "number",
"description": "Minimum score to consider a match (default: use config threshold)"
},
"top_k": {
"type": "integer",
"description": "Max matches to return (default 5)",
"default": 5
}
},
"required": [
"record"
]
}
unmerge_record
Remove a record from its cluster. The record becomes a singleton. Remaining cluster members are re-clustered using stored pair scores. Use this to fix bad merges.
Parameters1
record_id
integer
required
Row ID of the record to unmerge
Raw schema
{
"type": "object",
"properties": {
"record_id": {
"type": "integer",
"description": "Row ID of the record to unmerge"
}
},
"required": [
"record_id"
]
}
shatter_cluster
Break an entire cluster into individual records. All members become singletons. Use when a cluster is completely wrong.
Analyze bad merges and suggest config changes. Provide examples of incorrect merges (pairs that should NOT have matched) and GoldenMatch will identify which fields/thresholds to tighten. Example: [{"record_a": {...}, "record_b": {...}, "reason": "different people"}]
Parameters1
bad_merges
array
required
List of bad merge examples with record_a, record_b, and optional reason
Run the config healer over the loaded dataset: analyze the dedupe run and return ranked, self-verified suggestions for improving the matching config (thresholds, scorers, negative evidence, blocking). Each suggestion carries an id, kind, target, rationale, and a machine-applicable patch. Requires the native kernel (pip install goldenmatch[native]); returns an empty list otherwise.
Parameters
No parameters.
Raw schema
{
"type": "object",
"properties": {}
}
profile_data
Get data quality profile: column types, null rates, unique counts, sample values.
Run privacy-preserving record linkage between two parties' data. Computes bloom filters, matches records without sharing raw data. Specify fields, threshold, and security level.
Parameters10
file_a
string
optional
Path to party A's CSV file
file_b
string
optional
Path to party B's CSV file
file_a_content
string
optional
Alternative to file_a: base64/text bytes
file_a_name
string
optional
file_b_content
string
optional
Alternative to file_b: base64/text bytes
file_b_name
string
optional
encoding
string
optional
Encoding of *_content (default base64)
fields
array
required
Field names to match on (e.g. ['first_name', 'last_name', 'zip_code'])
Score the loaded run against ground-truth pairs. Loads a ground-truth CSV (id_a,id_b columns) and returns precision, recall, and F1 for the current clustering.
Parameters3
ground_truth_path
string
required
CSV of true match pairs (columns id_a,id_b or idA,idB).
Diagnose blocking on the loaded dataset: returns ranked blocking key candidates with block counts, max block size, total candidate comparisons, and estimated recall. Use it to explain why matching is slow or produces too many candidate pairs.
Compare two ER clustering outcomes on the same dataset without ground truth (CCMS): classifies each cluster as unchanged / merged / partitioned / overlapping and returns the Talburt-Wang Index. Both inputs are JSON cluster files (as written by export-style output).
Parameters7
clusters_a_path
string
optional
Baseline clusters JSON
clusters_b_path
string
optional
Comparison clusters JSON
clusters_a_content
string
optional
Alternative to clusters_a_path: base64/text bytes (JSON, use encoding='text')
clusters_a_name
string
optional
clusters_b_content
string
optional
Alternative to clusters_b_path: base64/text bytes (JSON, use encoding='text')
Auto-map columns between two files with different schemas. Returns proposed (col_a, col_b) mappings with a confidence score and method (synonym / name_sim / composite). Useful before matching two sources.
Field-level provenance for the loaded run: for each scored pair, the per-field scores that produced the match, plus cluster id. Optionally write a lineage JSON to a directory.
Parameters3
max_pairs
integer
optional
natural_language
boolean
optional
output_dir
string
optional
If set, write lineage JSON here and return the path instead of inline records.
Undo a previous run by DELETING its output files (looked up by run_id in the run log). Destructive: removes the files that run wrote. Use list_runs first to find the run_id.
Diagnose weaknesses in the loaded run's auto-config: columns admitted that shouldn't be (source/provenance labels, per-row IDs), oversized or shared-value blocks, null sinks, low-signal matchkeys, and over-merging. Returns ranked findings, each with a plain-English explanation + a concrete fix, plus a one-paragraph summary.
Parameters2
max_findings
integer
optional
Max findings to return, ranked by severity (default 6).
Alias for `match_record`. Match a single record against the loaded dataset in real-time. Paste a record's fields and instantly see if it matches any existing record. Uses the configured matchkeys, scorers, and thresholds. Example: {"name": "John Smith", "email": "john@test.com", "zip": "10001"}
Parameters3
record
object
required
Record fields to match against the dataset
threshold
number
optional
Minimum score to consider a match (default: use config threshold)
top_k
integer
optional
Max matches to return (default 5)
Raw schema
{
"type": "object",
"properties": {
"record": {
"type": "object",
"description": "Record fields to match against the dataset"
},
"threshold": {
"type": "number",
"description": "Minimum score to consider a match (default: use config threshold)"
},
"top_k": {
"type": "integer",
"description": "Max matches to return (default 5)",
"default": 5
}
},
"required": [
"record"
]
}
explain_pair
Alias for `explain_match`. Explain why two records match or don't match. Shows per-field score breakdown.
Your customer data lives in a CRM, a billing system, and three spreadsheets nobody owns. Some records are duplicates. Some are the same company spelled four different ways. Nobody can answer how many customers do we actually have, and every dashboard built on top inherits the doubt.
Splink-beating entity resolution, Arrow-native and Rust-fast with zero tuning, feeding a durable identity layer so messy records from every source become stable golden entities with whole-record, Customer-360 provenance.
Zero-config matching that beats expert-tuned Splink head-to-head on messy customer records, in an Arrow-native, Rust-authoritative engine verified from a laptop CSV to a 250M-row dedupe in 11.2 minutes. The identities it produces live in a transaction-native control plane carrying stable entity_ids, per-field provenance, merge/split, and a tamper-evident audit log, all one call away as a Customer 360. It even owns its primitives: byte-identical, faster-than-rapidfuzz / jellyfish / FAISS Rust kernels, not rented dependencies.
Python · TypeScript · SQL, at 4-decimal parity · native in Postgres + DuckDB · edge WASM · 70+ MCP tools · beats hand-tuned Splink · 250M rows in 11.2 min
Pair drilldown in the web workbench: cluster members, field-level diff, and a one-line NL explanation per pair. pip install goldenmatch[web] then goldenmatch serve-ui <project>. More screenshots →
v3.17.1: The polars-free first run actually works now. 3.17.0 claimed this and did not
deliver it: auto-config puts negative evidence on the exact matchkey by default,
and that path still bridged to polars, so goldenmatch dedupe customers.csv
exited 3 on a default install. Verified the way it should have been the first
time -- pip install into a clean polars-free venv, then the documented command.
v3.17.0: The documented first run works on a default install.pip install goldenmatch
followed by goldenmatch dedupe customers.csv -- the quickstart on every doc
surface -- exited 3 on a polars-free install, which is what a plain install has
produced since polars became an optional extra. Three separate polars imports on
the zero-config path (auto-config ingest, the Arrow lane's preflight decline, and
the csv writer) are gone, with polars' exact csv bytes reproduced and
parity-pinned.
v3.13.0: Fellegi-Sunter training runs distributed on Spark. The E-step reads only the
comparison vector, so identical vectors collapse to one counted row and the whole
step becomes a Spark GROUP BY over agreement patterns -- the cluster counts, the
driver only fits. Training cost tracks DISTINCT vectors (bounded by
prod(levels + 1)), not pairs: 1M -> 5M rows grew candidate pairs 5.00x and the
distributed counting stage 5.25x, while distinct patterns grew 3.0% (433 -> 446)
and driver-side EM stayed at 0.01s. Runs on jar-only executors via
goldenmatch-spark, off the same Rust kernel every other surface uses.
What you get: the golden entity
Most entity-resolution tools hand you clusters and stop. GoldenMatch keeps going: it resolves messy records into a durable golden entity, one per real-world customer, that survives re-runs, carries provenance on every field, and answers "who is this, and where did each value come from?" in a single call.
A stable identity, not a throwaway cluster id. Every entity gets a stable entity_id (UUIDv7) that persists across runs as new data arrives. Records are absorbed, entities merge or split, but the id an entity earns is the id downstream systems can rely on. Run-local cluster numbers reshuffle on every run; these don't.
Whole-record provenance. Every field of the golden record traces back to the source record that won it: which source, when it was last seen, and which survivorship strategy picked it. The values it didn't pick stay visible rather than being silently dropped. Provenance is on the whole unified record, not just the match decision.
Governed by construction. Conflicting values collapse to one best value by an explicit survivorship policy (most-complete · source-priority · most-recent · majority-vote); every identity change lands in an append-only event timeline; and the log is sealed with a hash-chained, tamper-evident audit that a reviewer (or the other language port) can independently verify.
customer_360(entity_id) composes it into one read: golden record, per-field provenance, every linked source record, the event timeline, and the entity's relationship neighborhood:
What ships today vs. what's emerging. The identity spine is production-grade and in main: stable entity_ids, per-field provenance, survivorship, merge/split, the append-only log + audit chain, cross-channel stitching, the relationship overlay, and incremental resolution against a persisted index (a new record resolves without a full re-run). The customer_360() serving view above and the source-registry layer that keeps it fresh from live systems are the newer, actively-landing pieces. The source connectors (Snowflake, BigQuery, Salesforce, HubSpot) ship today; the registry that wires them into the spine is emerging. See the Customer 360 design + ADR. We label the seam rather than blur it.
The golden entity lives in the control plane; the matching that builds it runs in the compute engine. That split is the next section.
One product, two engines
The golden entity above is produced by two engines that optimize for genuinely different things, and keeping them distinct is the architecture, not an implementation detail (ADR 0047).
flowchart LR
src([source records])
e360([golden entities · Customer 360])
subgraph compute ["Identity Compute Engine: Arrow-native, Rust-authoritative"]
match[block · score · cluster]
end
subgraph control ["Identity Control Plane: transaction-native state machine"]
spine[stable ids · survivorship · merge/split · provenance · audit]
end
src --> compute -->|resolution batch + evidence| control --> e360
control -.->|persisted index| compute
Identity Compute Engine
Identity Control Plane
Shape
Arrow at bulk boundaries, Rust-authoritative kernels
Transaction-native state machine (SQLite default · Postgres)
Job
Block, score, cluster: throughput, vectorized, deterministic per run
Stateless per call; measurement-driven kernelization
Durable, transactional, replayable, auditable
Backends
DataFusion · Ray · Sail · Spark are replaceable execution backends, none synonymous with GoldenMatch
Storage backends conform to one externally-observable semantics
Many surfaces, one answer. The same capabilities reach Python, edge-safe TypeScript (with an opt-in WASM backend running the same Rust kernels), SQL inside PostgreSQL and DuckDB, and MCP / REST / A2A, all governed by specification + conformance, not copy-paste. There is one authoritative owner per capability; pure-Python / standalone-TS paths are classified, conformance-tested fallbacks. Where a boundary can't cross byte-for-byte, we measure and label it rather than claim parity.
Why a platform engineer should care:
The compute layer isn't framework lock-in. It's Arrow-native and backend-replaceable, so you can push the heavy matching to a query engine that plans, spills, and distributes (verified to 250M rows) without the identity state coming along for the ride.
The identity layer is a real state machine, not a columnar rebuild-every-time batch. Durable ids, transactional merge/split, provenance and audit are first-class operations you can integrate against.
Behavior is consistent where it's shared. SQL, Python, and TypeScript track the same answers to a conformance spec, so the surface you build on isn't quietly inventing its own semantics.
Resolution that beats the expert
The identity layer is only as good as the matching underneath it, and the matching starts at zero config. dedupe_df(df) runs with no rules and no training data: it profiles the data, picks a defensible configuration, and returns golden records immediately. The config it chose comes back on result.config: inspectable, diffable, versionable. Never a black box.
Beats the expert, out of the box. On messy customer records, the opt-in Fellegi-Sunter engine beats hand-tuned Splink head-to-head, with historical_50k pairwise F1 0.827 vs 0.757, cluster B³ 0.862 vs 0.788, one shared evaluator, reproducible bake-off. Fuzzy, exact, probabilistic (Fellegi-Sunter), and LLM scorers, with EM-trained weights and calibrated scores.
A healing loop, not a one-shot. Zero-config gets you most of the way; then every run checks a free unsupervised signal and, when there's headroom, attaches ranked, self-verified config tweaks to result.suggestions. Each is kept only if it doesn't worsen a health proxy, so a suggestion never makes results worse. dedupe_df(df, heal=True) applies and re-runs in one call. You close the gap to expert-tuned without being the expert.
Privacy-preserving record linkage. Match across organizations without sharing raw data: Bloom-filter PPRL, 92.4% F1 on FEBRL4, with HMAC-salted encodings.
Self-verifying. Every step runs preflight + postflight checks and returns an inspectable report instead of failing silently. That is the "advanced, never black-box" contract that makes an automated identity layer safe to build on.
Runs on unstructured input, too: extract records from PDFs and images, then resolve them like any other source (pip install goldenmatch[documents]).
Runs where your stack is
The engine and the identity layer reach your stack through the surface you already use, with the same capabilities governed by conformance (one product, two engines) rather than re-implemented thinly per surface.
SQL-native, at parity. The same functions run inside PostgreSQL (pgrx extension) and DuckDB: dedupe · match · score · auto-config + telemetry · identity-graph reads · profiling · evaluate · Fellegi-Sunter scoring · GoldenFlow transforms. Resolve without moving data out of the warehouse.
Python and edge-safe TypeScript. The full suite ships on npm alongside PyPI. The TS cores are dependency-free and node:*-free (browsers, Cloudflare Workers, Vercel Edge, Deno); an opt-in WebAssembly backend (await enableWasm()) swaps in the same pyo3-free Rust kernels the Python wheels and SQL UDFs use, with pure-TS as the byte-identical default.
AI-native by default. Every package ships an MCP server, a REST API, and an A2A agent surface (70+ MCP tools across the suite), all exposing the same JSON telemetry shape across web, TUI, CLI, Postgres, DuckDB, and MCP.
Spark, with no Python on the executors. The Rust kernels ride into a Spark cluster in one jar (spark.addArtifact("goldenmatch-spark.jar")) and are called over JNI, so executors need no goldenmatch virtualenv, no packed env, nothing installed. Fellegi-Sunter training runs distributed on that path -- the E-step is a Spark GROUP BY over agreement patterns, so the cluster does the counting and the driver only fits the model. Deployment story, not a throughput one: the JVM scoring path measured ~2.4x slower than the Python-worker path, and the reason to use it is that there is nothing to install.
Pipeline-native. A dbt package (dedupe/match materializations, quality tests, identity-graph reads), a GitHub Action (fail PRs on data-quality regressions), and 13 drop-in Airflow DAGs (Deploy).
Production paths. Postgres sync, daemon mode, lineage tracking, review queues.
Cross-language parity
Surface parity is not the same as handing any pipeline phase from one language to the other byte-for-byte. Each verdict below is measured by a conformance harness, not assumed:
Boundary
Verdict
Identity graph DB
✅ byte-safe + cryptographically cross-verifiable (a seal written by one toolkit validates under the other)
Rule of thumb: hand off at the cluster or identity boundary and it's seamless; don't split across standardize/dates, embeddings, or the controller and expect bit-exact reproduction. Full detail + the runnable harness that keeps these verdicts honest: Cross-language parity & phase-handoff limits.
The suite: the pipeline into the spine
GoldenMatch is the headline, but resolution is only as good as what feeds it. Five sibling tools clean, standardize, and map records before they reach the identity layer. Each stands alone, but they compose into one pipeline, orchestrated declaratively by GoldenPipe:
flowchart LR
raw([raw rows])
golden([golden entities])
subgraph orchestration ["GoldenPipe orchestrates"]
direction LR
infermap[InferMap] --> goldencheck[GoldenCheck] --> goldenflow[GoldenFlow] --> goldenmatch[GoldenMatch]
end
raw --> infermap
goldenmatch --> golden
One MCP server exposing every suite tool under a single endpoint (stdio or Streamable HTTP)
pip install goldensuite-mcp
Owned libraries (standalone)
The suite owns its string-matching primitives instead of renting them: byte-identical drop-in replacements, published on their own so they're usable outside the suite too.
Pure-Rust HNSW approximate-nearest-neighbor index (zero C deps). Powers embedding-based blocking across Python, Rust, and TS/WASM.
pip install goldenmatch-hnsw
Knowledge graphs
Entity resolution is the stage most GraphRAG pipelines do worst: duplicate surface forms of one entity scatter across documents. Two packages put GoldenMatch's resolution there:
Build-your-own-KG from text: text → LLM extraction → GoldenMatch resolution → durable bi-temporal store. Rust engine; ER is the differentiator.
in-repo · first PyPI release pending
Measured, not asserted (ER-KG-Bench): resolution scores F1 0.602 on the labelled set, ahead of Neo4j-KGBuilder (0.456), neo4j-graphrag (0.403), and MS-GraphRAG / LightRAG / Cognee / mem0 (0.066). A resolved graph also does two things passage-window RAG structurally can't: exact aggregation (size-invariant where RAG recall collapses 0.99 → 0.64 across cluster-size buckets) and temporal as-of (1.000 vs 0.002 on past-date queries).
Scale & benchmarks
Every headline number maps back to a single committed runner (scripts/run_benchmarks.py); see docs/reproducing-benchmarks.md for per-number commands, dataset URLs, and expected output with tolerance.
Accuracy on customer-shaped data. NC Voter 0.9719 F1 (real-data sample), Febrl3 0.9912 F1; the opt-in Fellegi-Sunter path beats hand-tuned Splink head-to-head on every dataset Splink scores (bake-off). (Bibliographic DBLP-ACM lands 96.4% F1 for the record-linkage crowd, but customer identity is the focus.)
Privacy-preserving. PPRL 92.4% F1 on FEBRL4, matching across parties with no shared raw data.
Scale envelope (docs/scale-envelope.md): per-backend ranges (in-memory/bucket to a few M · DuckDB out-of-core to ~50M · Ray distributed ≥ 50M), block-size failure modes, and a decision tree for picking a backend.
Verified on the Ray tier: a full 100M-row dedupe on a 5-node Ray cluster in 9.2 min (554 s), 20,000,000 golden records recovered exactly, driver peak 0.36 GB RSS. (The larger 250M / 11.2 min headline is the Spark tier on a different cluster shape -- the two are separate lanes, not one number superseding the other, and this one is what carries the recall-complete guarantee below.) The default distributed path is recall-complete: duplicates merge correctly no matter how the input is partitioned (blocking-key shuffle scoring + distributed randomized-contraction WCC), and it stays driver-collect-free end to end. Recipe: configs/distributed-100m.yaml.
Fellegi-Sunter training, distributed on Spark: the E-step collapses to one Spark GROUP BY over agreement patterns, so training cost tracks the number of DISTINCT comparison vectors (bounded by prod(levels + 1)), not the pair count. Measured on a real 2-worker Spark cluster (jar-only executors, no Python installed), 1M -> 5M rows: candidate pairs grew 5.00x and the distributed counting stage 5.25x, while distinct patterns grew 3.0% (433 -> 446) and driver-side EM stayed at 0.01s. That is the property the tier rests on -- the cluster absorbs the data, the driver's work stays flat.
Head-to-head vs Splink on a real Spark cluster, at 50M rows. Both engines on the same 5-node cluster, the same fixture, the same shared metric implementation, and Splink configured the way its own performance guide prescribes -- break_lineage_method="parquet" onto a real distributed filesystem, shuffle partitions at 5x cluster cores, and identical 48 GB executors. Over 463,923,179 candidate pairs, scored identically by both:
GoldenMatch
Splink
ratio
wall
552s
1,054s
1.91x
u / estimate
3.5s
30.8s
8.78x
shuffle write
86.8 GB
212.1 GB
2.44x
stages
119
394
3.31x
executor CPU
31,300s
51,572s
1.65x
Reported with it, because a benchmark that only publishes its wins is not evidence: the margin still narrows with scale (2.54x at 1M -> 1.91x at 50M), single runs on this lane move ~16% so no one ratio should carry much weight, zero spill was scale-bounded on the build that curve measured (true at 50M, 56.4 GB at 100M, 201.3 GB at 250M -- since taken to zero at 250M by #2698, so the caveat describes the curve rather than the shipped path), the fixture is synthetic, and the accuracy figures in it are not an accuracy verdict -- for that, see the bake-off. Measured further since:250M rows / 2.32 billion pairs in 670s on the same 5-node cluster, no executor deaths and zero failed tasks, at 0.289 seconds per million pairs -- 4.13x faster than the 2,766s that curve first measured, with 3.1x less executor CPU, zero spill and a byte-identical trained model. Cost stays linear in PAIRS rather than rows; the constant got four times smaller. Four earlier attempts at this comparison were invalid because we had misconfigured Splink; each defect and its effect is documented alongside the results. Full method, caveats and reproduce command.
Three reproducible real-world pipelines run this on public data at scale:
shell-company-network: investigative ER across ICIJ Offshore Leaks + OpenSanctions + GLEIF + UK PSC. −62.5% analyst-hours to triage vs single-source baselines.
vuln-attribution: 6,126,895 OSS-vulnerability records → 847,475 canonical vulns across 40 sources in ~5 minutes on one 64GB runner.
sanctions-reconciliation: cross-list coverage on 85 public sanctions lists across 50+ jurisdictions.
Install & quick start
Dedupe a CSV in 30 seconds, zero config, writes <timestamp>_golden.csv:
The whole suite, configured for speed.golden-suite pulls in every package plus the native (Rust) kernels, pinned and defaulted to the perf-optimized config. Native wheels are hard dependencies on purpose: a platform without a wheel fails loudly rather than silently running the slow pure-Python path.
bash
pip install golden-suite
golden-suite doctor # verify every package + native kernel is importable and healthy
golden-suite optimize # repair / re-enable the perf-optimized config
pip install golden-suite[mcp] # + aggregator MCP server (every tool, one endpoint)
pip install golden-suite[all] # everything
Just GoldenMatch. Fat optional extras, so you pay only for what you use (native acceleration is default on common platforms):
Web workbench.pip install 'goldenmatch[web]' then goldenmatch serve-ui my-project (opens http://localhost:5050): edit rules with live validation, preview against a sampled slice, label pairs (mirrored into Learning Memory), compare runs.
Remote MCP. The hosted endpoint requires a bearer token -- it exposes tools that read and write files on the server, so it is not open. Ask the maintainer for one, or self-host: goldenmatch mcp-serve --transport http is the same server.
Cargo: no root workspace.packages/rust/extensions/ is itself a Cargo workspace (the postgres crate is excluded for pgrx); Cargo commands run from inside it.
TypeScript: one pnpm workspace.packages/typescript/* form a single pnpm + Turborepo workspace.
bash
just install # uv sync + per-package npm install + cargo fetch
just test# all languages · just lint · just build
Contributing
Feature work on feature/<name> branches; merge via squash PR. Titles: feat: / fix: / docs:.
Tests must pass on all three languages where the change applies; the parity harness in packages/typescript/goldenmatch/tests/parity/ enforces 4-decimal Python ↔ TypeScript scorer parity.
Architecture changes conform to (or amend) the one-product-two-engines frame in the same PR. Design rationale lives in context-network/decisions/ and docs/superpowers/specs/.
bash
corepack enable# one-time, picks up pnpm@9.15.0
pnpm install
pnpm turbo run build test typecheck # full pipeline (cached after first run)
Windows: enable Developer Mode so pnpm install can create symlinks; if corepack enable needs admin, npm i -g pnpm@9.15.0 is equivalent.
This repo was formed on 2026-05-01 by folding 8 sibling repos into goldenmatch via git filter-repo (full history preserved). Built by Ben Severn. MIT, see LICENSE.
