ConKurrence MCP Server (io.github.AlligatorC0der/conkurrence)

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. As an MCP server, it can be run in any MCP-compatible client (including Claude Desktop) via npx conkurrence mcp, to measure whether an AI agrees with itself using statistical consensus metrics.

🛠️ Key Features

Uses multiple AI models as independent raters
Measures inter-rater reliability with Fleiss’ kappa
Provides bootstrap confidence intervals
Routes contested items to human experts

🚀 Use Cases

AI evaluation pipelines that require consensus and disagreement detection
Workflows that need reliability metrics (Fleiss’ kappa) and uncertainty estimates (bootstrap intervals)
Human review for items where models disagree

⚡ Developer Benefits

Can be used as an MCP server in Claude Desktop or other MCP-compatible clients
Command-line invocation: npx conkurrence mcp

⚠️ Limitations

The provided MCP documentation excerpt does not specify available tools or tool count.

ConKurrence

Multi-model rating for AI evaluation: measures agreement among model raters (Fleiss' κ, Kendall's W) to find contested items for expert review.

What agreement tells you: where your raters and your criteria are consistent, and where they are not — which items are contested, and whether disagreement comes from the criteria or the raters.

What it does not tell you: that the raters are right. Agreement is reliability, not validity. Models can agree confidently on a wrong answer. Use ConKurrence to decide where an expert's judgment is needed and to diagnose your evaluation criteria — not to replace expert-labelled ground truth.

Status: maintenance mode. ConKurrence is developed by JE Vectors LLC as the evaluation instrument behind Inqura. It is not accepting purchases and carries no support commitment.

MCP Server

Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:

bash

npx conkurrence mcp

Claude Desktop Configuration

Add to your claude_desktop_config.json:

json

{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": ["-y", "conkurrence", "mcp"]
    }
  }
}

Claude Code Plugin

code

/plugin marketplace add AlligatorC0der/conkurrence

Features

Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously

Agreement statistics — Fleiss' kappa with bootstrap confidence intervals; Kendall's W against expert-labelled anchor items

Self-consistency mode — No API keys needed; uses the host model via MCP Sampling

Schema suggestion — AI-powered schema design from your data

Trend tracking — Compare runs over time, detect agreement degradation

Cost estimation — Know the cost before running

MCP Tools

Tool

Description

conkurrence_run

Execute an evaluation across multiple AI raters

conkurrence_report

Generate a detailed markdown report

conkurrence_compare

Side-by-side comparison of two runs

conkurrence_trend

Track agreement over multiple runs

conkurrence_suggest

AI-powered schema suggestion from your data

conkurrence_validate_schema

Validate a schema before running

conkurrence_estimate

Estimate cost and token usage

io.github.AlligatorC0der/conkurrence