tracehub-mcp

Also listed on the official MCP registry as io.github.mcpsmiths/tracehub-mcp (the registry's own ?search= endpoint can surface an outdated version first; this exact-name endpoint always reflects the current isLatest release), and on mcprush.

Give your AI assistant a direct line into your observability backend. tracehub-mcp is an MCP (Model Context Protocol) server that lets Claude, Cursor, Windsurf, Gemini CLI, or any MCP client query OpenTelemetry traces from your LLM/GenAI application and reason about them — find expensive calls, debug errors, compare model performance, track token usage — without you copy-pasting trace JSON into a chat window.

It speaks OpenTelemetry's gen_ai.* semantic conventions natively, so it understands prompts, completions, token usage, and finish reasons as first-class concepts, not just generic span attributes.

tracehub-mcp started as a fork of traceloop/opentelemetry-mcp-server (Apache 2.0) — full attribution and fork history are in NOTICE. It's grown into an 8-backend, security-hardened server maintained independently under mcpsmiths; see What's Different From Upstream below for the parts that are new here.

Table of Contents

• Quick Start
• Supported Backends
• What's Different From Upstream
• Installation
• Configuration
• Diagnostics
• Security Considerations
• Privacy Policy
• MCP Client Setup
• Tools Reference
• Generic Filter System
• Example Queries
• Common Workflows
• Development
• Troubleshooting
• Roadmap
• License
• Support

Quick Start

tracehub-mcp is on PyPI. No install step needed — uvx fetches and runs it in one shot:

// claude_desktop_config.json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

Or from Claude Code directly:

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

That's it. Ask your assistant: "Show me traces with errors from the last hour."

See MCP Client Setup for Cursor, Windsurf, VS Code, and Gemini CLI, and Installation for pip/pipx/from-source alternatives.

Supported Backends

• Jaeger — local/self-hosted, the most common open-source trace backend. No auth required (a Bearer token can be sent via BACKEND_API_KEY; credentials embedded in BACKEND_URL are rejected).
• Grafana Tempo — local or Grafana Cloud, TraceQL-native search.
• Traceloop — cloud LLM observability platform, API-key auth.
• Datadog — cloud APM, requires an API key and an Application key.
• Sentry — cloud or self-hosted, requires an auth token and an organization slug.
• AWS X-Ray — SigV4-signed via boto3, requires an AWS region and standard AWS credentials (not an API key). Because OTel span attributes land in unindexed X-Ray segment metadata by default (only annotations are queryable), native server-side filtering is narrower here than for the other backends — see the Backend Support Matrix.
• New Relic — NerdGraph GraphQL API, requires a User API key (not an Ingest/License key) and an account ID. Built without a live account to verify against — see the New Relic setup section below and the module docstring in backends/newrelic.py for exactly which assumptions are unverified.
• Honeycomb — Query Data API, requires a Configuration Key and a dataset slug. Built without a live account to verify against, and the research behind it found the programmatic Query Data API (the only way to actually run a query and get results back) is documented as Enterprise-plan exclusive — see the Honeycomb setup section below and the module docstring in backends/honeycomb.py.

All eight implement the same BaseBackend interface, so every MCP tool works identically regardless of which one you point the server at. See Configuration for per-backend setup.

What's Different From Upstream

Upstream opentelemetry-mcp-server shipped Jaeger, Tempo, and Traceloop. tracehub-mcp adds Datadog and Sentry as full backends — not thin wrappers, but complete implementations of every tool (search, span search, trace hydration, service discovery, health checks). Along the way, all five backends — including the three inherited from upstream — were hardened to a consistent bar:

• HTTPS-only enforcement on cloud backends. Datadog and Sentry both refuse to start against a plain http:// URL, because their auth is a bearer token / API+App key pair that has no business going out over plaintext.
• Query-injection-safe escaping. Every value spliced into a Datadog span-search query or a Sentry Discover query is escaped and exact-quoted; field names (which are less obviously untrusted, since they come from the MCP tool's filters parameter) are validated against an allowlist pattern before being spliced into the query string, closing off structural injection through a crafted field name.
• Bounded pagination on every backend that paginates via cursor (Datadog, Sentry) — a search stops at the requested limit or when the backend stops returning a continuation cursor, whichever comes first, so a single tool call can't degrade into an unbounded crawl.
• Exact-ID re-verification. Where a backend's search API can return neighbors instead of an exact match (notably Datadog's trace reconstruction from grouped spans), every result is re-checked against the exact ID that was asked for before being returned.
• No fabricated data on malformed responses, in the backends we built. Datadog and Sentry reject a span outright — rather than substituting a placeholder like now() for a missing timestamp or a literal "unknown" for a missing service_name/operation_name — since a fabricated value would silently corrupt trace ordering, duration aggregation, and any tool that groups by service or operation. (The three backends inherited from upstream — Jaeger, Tempo, Traceloop — predate this discipline and haven't been retrofitted; that's deliberate scope discipline, not an oversight, mirroring this project's own precedent of not reaching into shared/inherited code without full regression coverage for it.)

All of this is backed by 458 passing tests (2 skipped, zero regressions), a clean ruff check and mypy --strict run, and two rounds of adversarial CodeRabbit review on the new backends.

Installation

tracehub-mcp is on PyPI. Pick whichever of these your workflow already uses — they're equivalent.

Option 1: uvx (no install step)

uvx tracehub-mcp --backend jaeger --url http://localhost:16686

This is what the Quick Start config above uses — uv fetches the package and runs the tracehub-mcp entry point in one shot, nothing left behind on disk between runs.

Option 2: pip / pipx

pipx install tracehub-mcp
# or: pip install tracehub-mcp

tracehub-mcp --backend jaeger --url http://localhost:16686

Option 3: Clone and run from source

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync

uv run tracehub-mcp --backend jaeger --url http://localhost:16686

Use this if you're developing locally, want to pin to a specific commit, or want the dev tooling installed (uv sync --group dev).

Option 4: Docker

# The image's CMD binds 0.0.0.0 inside the container, so a caller token is
# mandatory: clients send it as `Authorization: Bearer $MCP_HTTP_BEARER_TOKEN`.
export MCP_HTTP_BEARER_TOKEN=$(openssl rand -hex 32)
docker run --rm -p 8000:8000 \
  -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://host.docker.internal:16686 \
  -e MCP_HTTP_BEARER_TOKEN \
  -e MCP_ALLOWED_HOSTS=localhost \
  ghcr.io/mcpsmiths/tracehub-mcp:latest

Backend responses are capped at BACKEND_MAX_RESPONSE_BYTES (64 MiB decoded by default) before they are buffered, but a container that serves many sessions still benefits from Docker's own guard rails: add --memory 1g --restart unless-stopped (or the Compose equivalents) so a process that does exhaust memory is contained and comes back without an operator.

Runs HTTP transport by default (the image's CMD) and refuses to start without MCP_HTTP_BEARER_TOKEN - or the explicit -e MCP_ALLOW_UNAUTHENTICATED_HTTP=true opt-out, which hands every peer that can reach the published port your backend's full read scope. Clients connect to http://localhost:8000/mcp with Authorization: Bearer <token>; /health and /ready (the image's HEALTHCHECK) never need the token. Because the CMD binds 0.0.0.0, the container validates the Host header only when MCP_ALLOWED_HOSTS is set - always set it to the hostnames clients use (localhost in the example above, the public name in a real deployment); * turns Host validation off and may not be mixed with names (the server refuses to start). This is also the form to use for MCP clients whose config takes a command/args pair pointing at docker directly (Cursor, Windsurf) instead of a local binary - pass the same -e MCP_HTTP_BEARER_TOKEN=... in args.

Prerequisites: Python 3.11+, plus uv for Options 1 and 3; Docker for Option 4.

Configuration

Configuration comes from environment variables, CLI flags, or both. Precedence: CLI arguments > environment variables > defaults.

# .env (see .env.example)
BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

# Equivalent via CLI flags
tracehub-mcp --backend jaeger --url http://localhost:16686

All Configuration Options

Every backend-related CLI flag has a matching env var (--backend/BACKEND_TYPE, --url/BACKEND_URL, --api-key/BACKEND_API_KEY, --app-key/BACKEND_APP_KEY, --tempo-instance-id/BACKEND_TEMPO_INSTANCE_ID, --sentry-org/BACKEND_SENTRY_ORG, --sentry-project/BACKEND_SENTRY_PROJECT, --aws-region/BACKEND_AWS_REGION, --environments/BACKEND_ENVIRONMENTS). --disable-tools <name1,name2,...> / --enabled-tools <name1,name2,...> (CLI-only, no env var) remove/allowlist tools for reduced-trust deployments - --enabled-tools is applied first, --disable-tools on top of whatever it kept. --allowed-hosts, --forwarded-allow-ips, --rate-limit-max-requests and --rate-limit-window-seconds (and their env vars) are validated when the server starts: a value that cannot mean what was written exits 2 with a usage error that names both the flag and the env var - for --allowed-hosts and --forwarded-allow-ips the offending entry is echoed as a bounded repr; the two rate-limit options are checked by click's own integer/float ranges, which echo the value as written - and their --help text shows the env var ([env var: ...). Run tracehub-mcp --help for the full list.

Known third-party egress dependency: the underlying FastMCP framework checks PyPI (https://pypi.org/pypi/fastmcp/json) for a newer FastMCP release once every 12 hours when it prints its startup banner - this is FastMCP's own behavior, not tracehub-mcp's, and unrelated to the OTel self-instrumentation above. It fails silently if there's no network access. For network-restricted/air-gapped deployments, disable it with FASTMCP_CHECK_FOR_UPDATES=off, or suppress the banner entirely with FASTMCP_SHOW_SERVER_BANNER=false.

Backend-Specific Setup

BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686

BACKEND_TYPE=tempo
BACKEND_URL=http://localhost:3200

BACKEND_TYPE=tempo
BACKEND_URL=https://tempo-prod-XX-prod-XX-XXXX.grafana.net
BACKEND_TEMPO_INSTANCE_ID=your_stack_instance_id
BACKEND_API_KEY=your_cloud_access_policy_token

BACKEND_TYPE=traceloop
BACKEND_URL=https://api.traceloop.com
BACKEND_API_KEY=your_api_key_here

BACKEND_TYPE=datadog
# US site (default): https://api.datadoghq.com
# EU site:            https://api.datadoghq.eu
BACKEND_URL=https://api.datadoghq.com
BACKEND_API_KEY=your_api_key_here
BACKEND_APP_KEY=your_application_key_here

BACKEND_TYPE=sentry
# SaaS (may be region-specific, e.g. https://us.sentry.io):
BACKEND_URL=https://sentry.io
BACKEND_API_KEY=your_auth_token_here
BACKEND_SENTRY_ORG=your-org-slug
# Optional: narrow queries to one project
BACKEND_SENTRY_PROJECT=your-project-slug

BACKEND_TYPE=xray
# Decorative only - never dereferenced for a live request. Keep it
# consistent with BACKEND_AWS_REGION by convention.
BACKEND_URL=https://xray.us-east-1.amazonaws.com
BACKEND_AWS_REGION=us-east-1

BACKEND_TYPE=newrelic
# US datacenter (default): https://api.newrelic.com/graphql
# EU datacenter:            https://api.eu.newrelic.com/graphql
BACKEND_URL=https://api.newrelic.com/graphql
BACKEND_API_KEY=your_user_api_key_here
BACKEND_NEWRELIC_ACCOUNT_ID=1234567

BACKEND_TYPE=honeycomb
# US instance (default): https://api.honeycomb.io
# EU instance:            https://api.eu1.honeycomb.io
BACKEND_URL=https://api.honeycomb.io
BACKEND_API_KEY=your_configuration_key_here
BACKEND_HONEYCOMB_DATASET=your-dataset-slug

Transport Modes

# stdio (default) — local use, Claude Desktop, single process
uvx tracehub-mcp
tracehub-mcp                      # pipx/pip install
uv run tracehub-mcp               # from-source install

# HTTP — remote access, multiple clients, network deployment, sample applications
# Loopback (the default --host) needs no token: same-machine clients use http://127.0.0.1:8000/mcp
uvx tracehub-mcp --transport http --port 8000
# Every interface: a caller token is mandatory (or the explicit --allow-unauthenticated-http opt-out)
export MCP_HTTP_BEARER_TOKEN=$(openssl rand -hex 32)
uvx tracehub-mcp --transport http --host 0.0.0.0 --port 8000
tracehub-mcp --transport http --host 0.0.0.0 --port 8000              # pipx/pip install
uv run tracehub-mcp --transport http --host 0.0.0.0 --port 8000       # from-source install

With HTTP transport, clients connect to http://<host>:<port>/mcp (streamable-HTTP, for compatibility across MCP clients) and, whenever MCP_HTTP_BEARER_TOKEN is set, send Authorization: Bearer <token> on every request - a missing or wrong token answers 401 with a WWW-Authenticate: Bearer challenge and never opens a session. uvicorn's per-request access log is off by default (--access-log / MCP_ACCESS_LOG=true re-enables it with query strings, fragments, matrix parameters (;, which uvicorn logs as %3B) and percent-encoded ?/# stripped from every line and the logged path cut at 2 KiB; a token placed in a path segment is still logged): it would otherwise write the full request target - including a token a misconfigured client puts in the query string - to the server log, and nothing in it is needed; request-level diagnostics come from the server's own bounded Host/Origin rejection and rate-limit lines, the slow-request WARNING and, when enabled, OTel spans.

A loopback bind also validates the Host header automatically (loopback names and the bound address pass, anything else answers 421), closing the DNS-rebinding shape where a browser page's Origin-less GET arrives with the attacker's name in Host. If a same-host reverse proxy forwards to 127.0.0.1 and preserves the public Host (nginx's default proxy_set_header Host $proxy_host does not), list that name in --allowed-hosts/MCP_ALLOWED_HOSTS. Network binds get the same protection once --allowed-hosts names their public hostnames; --allowed-hosts '*' opts out and must stand alone (a list that mixes * with names is refused at startup).

Fly.io: the one-click fly mcp launch command only supports stdio-transport servers — it can't deploy this server's --transport http mode. Running tracehub-mcp on Fly.io with HTTP transport needs the manual fly.toml + fly deploy path instead.

Diagnostics

Validate a config before wiring it into a real MCP client:

tracehub-mcp doctor --backend jaeger --url http://localhost:16686

[OK] Configuration loaded and validated
[OK] Backend constructed: jaeger @ http://localhost:16686/
[OK] Health check: healthy
[OK] Connectivity probe (list_services): 2 service(s)

doctor accepts the same --backend/--url/etc. flags as the main command and exits non-zero if any step fails - unlike normal server startup, which lazily initializes the backend and deliberately keeps running even if the initial health check fails.

To see the fully-resolved configuration (env vars + CLI overrides merged) without starting the server:

tracehub-mcp --print-config --backend jaeger --url http://localhost:16686

Secrets (--api-key/--app-key) are reported as api_key_set/app_key_set booleans, never their actual value.

Security Considerations

Trace and span data returned by this server — attribute values, error messages, operation names — comes from whatever application your observability backend is instrumenting, not from tracehub-mcp itself. That makes it fundamentally the same category of untrusted external content as a webpage or a file, even though it's a trusted server (this one) handing it back to your MCP client.

• The HTTP transport runs every tool under your backend credential, so authenticate its callers. --transport http binds loopback (127.0.0.1) by default. Serving on a network interface (--host 0.0.0.0, the Docker image's CMD) requires MCP_HTTP_BEARER_TOKEN (clients send Authorization: Bearer <token>); the server refuses to start without it unless you pass --allow-unauthenticated-http to accept that every peer able to reach the port gets your backend's full read scope. Alternatively keep the loopback bind and front it with an authenticating reverse proxy on a private network. /health and /ready never require the token.
• Backend credentials never ride inside BACKEND_URL. A user:password@host URL is refused at startup with a message naming the supported alternatives (BACKEND_API_KEY, or BACKEND_TEMPO_INSTANCE_ID + BACKEND_API_KEY for Basic auth), because the URL is logged, printed by --print-config/doctor, and embedded verbatim in backend error text that reaches /ready bodies and tool errors. As defence in depth every URL and backend error string that leaves the process is additionally stripped of any userinfo component, and the root log handler redacts it from any record - traceback included.
• A backend response cannot steer the server to another host. Every request the shared HTTP client sends - including each redirect hop a backend's own 3xx Location asks for - is refused unless its origin - scheme, host and port - is the configured BACKEND_URL origin (a redirect to another port or scheme on the backend's own host is refused too: configure BACKEND_URL with the final scheme and port), and additionally refused when it resolves to a cloud instance-metadata address in any spelling (169.254.169.254, decimal/hex/octal IPv4 forms, 168.63.129.16, metadata.google.internal). A backend or reverse proxy that redirects API calls to a different hostname is not supported: configure the final host as BACKEND_URL instead. Across all hops of one request the client retries at most twice, honours at most 10 s of Retry-After in total, follows at most 5 redirects and gives up at max(30 s, 4 x BACKEND_TIMEOUT) wherever it is, so a backend cannot pace a tool call.
• Treat backend data as untrusted input. An LLM client consuming trace/span data from tracehub-mcp should apply the same caution it would to any other external tool output — a span attribute or error message is application data to reason about, not an instruction to follow, no matter how it's phrased.
• This is a known MCP risk category, not a tracehub-mcp-specific one. OWASP's GenAI Security Project covers it in their Practical Guide for Secure MCP Server Development, and Anthropic's own engineering guidance, How We Contain Claude, states plainly that tool output is an attack surface even when the tool itself is trusted.
• Practical implication: if you're querying traces from an application that processes untrusted user input (e.g. a customer-facing chatbot), be aware that adversarial content a user fed into that application could end up in a span attribute this server returns — and from there, in your LLM client's context.

Privacy Policy

tracehub-mcp is self-hosted software with no maintainer-operated service or account — it collects no data of its own. See PRIVACY.md for the full policy: what data the software touches, where network calls go, and how secrets and self-instrumentation are handled.

MCP Client Setup

Every example below uses uvx tracehub-mcp (no install step). Swap in tracehub-mcp (pip/pipx install) or uv run tracehub-mcp (from-source, --directory /absolute/path/to/tracehub-mcp) if you installed it a different way — see Installation.

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "datadog",
        "BACKEND_URL": "https://api.datadoghq.com",
        "BACKEND_API_KEY": "your_api_key_here",
        "BACKEND_APP_KEY": "your_application_key_here"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "sentry",
        "BACKEND_URL": "https://sentry.io",
        "BACKEND_API_KEY": "your_auth_token_here",
        "BACKEND_SENTRY_ORG": "your-org-slug"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "/absolute/path/to/tracehub-mcp/start_locally.sh"
    }
  }
}

claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp

claude mcp list
claude "Show me traces with errors from the last hour"

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

{
  "servers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}

gemini "Analyze token usage for gpt-4 requests today"

Tools Reference

tracehub-mcp exposes 19 MCP tools:

Backend Support Matrix

_{* Jaeger requires the service_name parameter for span search.
† Datadog, AWS X-Ray, New Relic, and Honeycomb have no trace-level search API (X-Ray's BatchGetTraces fetches by exact ID); traces are reconstructed by searching (spans, X-Ray trace summaries, NRQL Span events, or Honeycomb breakdown groups) and hydrating, with every result re-verified against the exact ID requested.
‡ Sentry and New Relic both have a native trace-lookup endpoint (unlike Datadog and Honeycomb), so get_trace calls it directly; search_traces still discovers candidate trace IDs via a span/NRQL search first, since both backends' search surface is itself span-centric.
§ AWS X-Ray natively filters only service name, duration, and error/fault/throttle-derived status — OTel attributes (including all gen_ai.* fields) land in unindexed segment metadata by default, not indexed annotations, so they're always applied client-side rather than pushed down to FilterExpression.
¶ New Relic: search_spans reads gen_ai.* attributes directly off NRQL's well-documented flat row shape; search_traces/get_trace additionally rely on an unverified assumption about the shape of the attributes field inside NerdGraph's distributedTracing.trace response, mitigated by overlaying the original search row's attributes back onto the hydrated result. Honeycomb: gen_ai.* attributes are extracted via the same breakdown-column technique used for every other field (module docstring point 2), an equally unverified extraction method.
♦ Honeycomb's entire "Search"/"Get trace by ID" row of checkmarks above depends on the Query Data API, which the research behind this backend found is documented as Enterprise-plan exclusive - a Free/Pro account may not be able to exercise any of this at all. See the Honeycomb setup section above.}

Key Tool Details

search_traces

{
    "service_name": "my-app",
    "start_time": "2024-01-01T00:00:00Z",
    "end_time": "2024-01-01T23:59:59Z",
    "gen_ai_system": "openai",
    "gen_ai_request_model": "gpt-4",
    "min_duration_ms": 1000,
    "has_error": false,
    "limit": 50,
}

Parameters: service_name, operation_name, start_time/end_time (ISO 8601), min_duration_ms/max_duration_ms, gen_ai_system, gen_ai_request_model, gen_ai_response_model, has_error, tags, filters (see Generic Filter System), limit (1-1000, default 100). Returns trace summaries with token counts.

get_trace

{"trace_id": "abc123def456"}

Returns the full trace tree: all spans with attributes, parsed OpenTelemetry gen_ai.* data for LLM spans, per-span token usage, error information, and each span's raw events (e.g. gen_ai.evaluation.result, or any other instrumentation-emitted event — not filtered to a fixed set of names).

triage_trace

{"trace_id": "abc123def456", "detail_level": "summary"}

Instead of returning raw trace data for an agent to re-derive a diagnosis from every time, this synthesizes one directly: a critical path (the "Last Finishing Child" chain actually responsible for the trace's total latency), the top spans ranked by self-time (latency contribution net of children), and — when the trace contains an error — the deepest error span in the trace's error chain as the likely root cause (confidence: "high", or "medium" when multiple equally-deep error chains make blame ambiguous). Falls back to the highest self-time span as a pure-latency diagnosis (confidence: "low") when no error is present. Deterministic — no LLM call — and works against any configured backend, since it operates on get_trace's already-fetched span data. detail_level: "full" additionally attaches the diagnosed root cause's raw error detail (message/type/stacktrace) when the verdict is error-driven. Results are bounded whatever the trace's depth: in summary mode error_chain and critical_path each list at most 25 spans - the first 5 and last 20 around one explicit [K intermediate span(s) elided] entry, with the real depth stated in the reasoning - and detail_level: "full" raises that to 1000; a trace with no root span (a parent cycle) is reported as such with confidence: "low" rather than as a depth.

correlate_trace

{"trace_id": "abc123def456"}

Tries to find the corresponding trace in a second, independently-configured backend (e.g. a Datadog trace and its downstream Sentry error, joined). Tries a direct trace_id match in the secondary backend first (confidence: "high"); if that fails, falls back to a time-window + service-name-overlap heuristic search (confidence: "low") - queried once per service name in the primary trace, so this works against backends like Jaeger that require service_name on search_traces; at most the 10 most-represented services (by span count) are searched, so one trace cannot fan out into hundreds of secondary-backend queries. Each match also reports root_cause_consistent: null when neither trace has an error to compare, otherwise true/false for whether both sides agree on the error and which service it traces back to (reusing the same error-chain logic triage_trace uses). This is a best-effort correlation, not a guaranteed join - the result always includes the fixed limitations (clock skew, sampling mismatches, partial trace visibility, and the fact that trace-id continuity across a vendor boundary isn't guaranteed even under normal W3C Trace Context propagation) plus any per-call truncation notes: the heuristic search window is capped at 24 h after the primary trace's start, however wide an extent the primary trace's spans claim, so one trace cannot make the server scan years of the secondary backend, and the result says when services beyond the 10-service cap were skipped. Requires a secondary backend configured via SECONDARY_BACKEND_TYPE/SECONDARY_BACKEND_URL (see configuration table); raises a clear error otherwise.

get_llm_usage

{
    "start_time": "2024-01-01T00:00:00Z",
    "end_time": "2024-01-01T23:59:59Z",
    "service_name": "my-app",
    "gen_ai_system": "openai",
    "limit": 1000,
}

Returns aggregated prompt/completion/total tokens, broken down by model and by service, plus request counts.

list_services — no parameters. Returns the list of instrumented service names.

find_errors

{"start_time": "2024-01-15T14:00:00Z", "service_name": "my-app", "limit": 50}

Returns error messages, error types, truncated stack traces, and LLM-specific error info.

list_llm_models / get_llm_model_stats / get_llm_expensive_traces / get_llm_slow_traces / list_llm_tools_tool / search_spans_tool / list_sessions / get_session_stats / compare_time_windows / investigate_cost_spike / investigate_error_spike / get_prompt_version_stats are documented in detail, with worked examples, in CLAUDE.md — this README covers the shape every tool shares; CLAUDE.md is the fuller reference for exact parameters and response fields on the LLM-analysis tools.

Generic Filter System

search_traces and search_spans_tool both accept a filters list in addition to (or instead of) their simple named parameters, for advanced queries. Each filter is:

{
  "field": "gen_ai.usage.total_tokens",
  "operator": "gt",
  "value": 5000,
  "value_type": "number"
}

• field — dotted attribute name, e.g. gen_ai.usage.prompt_tokens, traceloop.span.kind, service.name (1–256 characters)
• operator — see table below
• value — single value (most operators) or values — list (for in, not_in, between; at most 100 values)
• value_type — "string", "number", or "boolean"

Limits and validation. One call accepts at most 50 filters and 50 tags (two independent caps, not one shared budget), each in/not_in filter at most 100 values, and a field name is 1–256 characters, so a single query can only ever produce a bounded number of backend clauses. Operands are normalised to their declared value_type before anything is sent: a numeric string such as "5000" under "number" is accepted and coerced, while a non-numeric one ("abc"), a non-finite value or an integer too large for a double is rejected; "boolean" accepts a JSON boolean or the strings "true"/"false"; "string" accepts a number or boolean and stringifies it. A filter object with an unknown key is rejected rather than silently ignored. Exceeding a limit or failing validation returns a tool error before any backend request is made.

Multiple filters combine with AND logic. Legacy simple parameters (service_name, gen_ai_request_model, etc.) still work and are converted to filters internally — mix and match freely.

The server uses a hybrid filtering strategy: filters are pushed to the backend's native query language when supported (TraceQL for Tempo, span-search syntax for Datadog, Discover syntax for Sentry), and applied client-side afterward for anything the backend can't express natively.

Example — expensive OpenAI traces:

{
  "filters": [
    { "field": "gen_ai.system", "operator": "equals", "value": "openai", "value_type": "string" },
    { "field": "gen_ai.usage.total_tokens", "operator": "gt", "value": 5000, "value_type": "number" }
  ]
}

For the full semantic-convention attribute list (gen_ai.* vs legacy llm.*, token-naming variants across providers, finish-reason values, and the token-calculation fallback chain), see CLAUDE.md.

Example Queries

Find Expensive OpenAI Operations

Ask: "Show me OpenAI traces from the last hour that took longer than 5 seconds"

Tool call: search_traces

{
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "min_duration_ms": 5000,
  "start_time": "2024-01-15T10:00:00Z",
  "limit": 20
}

Response:

{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "status": "OK",
      "duration_ms": 8250,
      "span_count": 3,
      "llm_span_count": 1,
      "total_tokens": 4523,
      "has_errors": false
    }
  ],
  "count": 1
}

Analyze Token Usage by Model

Ask: "How many tokens did we use for each model today?"

Tool call: get_llm_usage

{
  "start_time": "2024-01-15T00:00:00Z",
  "end_time": "2024-01-15T23:59:59Z",
  "service_name": "my-app"
}

Response:

{
  "period": { "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-15T23:59:59Z" },
  "filters": { "service_name": "my-app" },
  "summary": {
    "total_requests": 487,
    "total_prompt_tokens": 82140,
    "total_completion_tokens": 43290,
    "total_tokens": 125430
  },
  "by_model": {
    "gpt-4": { "requests": 156, "prompt_tokens": 58300, "completion_tokens": 26900, "total_tokens": 85200 },
    "gpt-3.5-turbo": { "requests": 331, "prompt_tokens": 23840, "completion_tokens": 16390, "total_tokens": 40230 }
  },
  "by_service": {
    "my-app": { "requests": 487, "prompt_tokens": 82140, "completion_tokens": 43290, "total_tokens": 125430 }
  }
}

Find Traces with Errors

Ask: "Show me all errors from the last hour"

Tool call: find_errors

{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 10
}

Response:

{
  "count": 1,
  "error_traces": [
    {
      "trace_id": "def456...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "start_time": "2024-01-15T14:23:15Z",
      "duration_ms": 1200,
      "status": "ERROR",
      "span_count": 2,
      "llm_span_count": 1,
      "total_tokens": 310,
      "has_errors": true,
      "error_spans": [
        {
          "span_id": "span789...",
          "operation_name": "chat.completions",
          "service_name": "my-app",
          "status": "ERROR",
          "error_message": "RateLimitError: Too many requests",
          "error_type": "openai.error.RateLimitError",
          "is_llm_error": true,
          "llm_provider": "openai",
          "llm_model": "gpt-4"
        }
      ]
    }
  ]
}

Compare Model Performance

Ask: "What's the performance difference between GPT-4 and Claude?"

Tool call 1: get_llm_model_stats for gpt-4

{ "model_name": "gpt-4", "start_time": "2024-01-15T00:00:00Z" }

Tool call 2: get_llm_model_stats for claude-3-opus

{ "model_name": "claude-3-opus-20240229", "start_time": "2024-01-15T00:00:00Z" }

Investigate High Token Usage

Ask: "Which requests used the most tokens today?"

Tool call: get_llm_expensive_traces

{ "limit": 10, "start_time": "2024-01-15T00:00:00Z", "min_tokens": 5000 }

Common Workflows

Cost Optimization

1. get_llm_expensive_traces — find the highest-token requests
2. get_llm_usage — see which models are costing the most
3. get_trace on a specific trace_id — inspect the exact prompt/response

Performance Debugging

1. get_llm_slow_traces — identify latency outliers
2. find_errors — check for failure patterns
3. get_llm_model_stats — check finish-reason distribution for truncation

Model Adoption Tracking

1. list_llm_models — see every model actually being called
2. get_llm_model_stats per model — compare performance
3. Scan list_llm_models results for unexpected models/services (shadow AI)

Development

git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync --group dev   # pulls in pytest, mypy, ruff, etc. for local iteration

# Tests (458 passed, 2 skipped at time of writing)
uv run pytest

# With coverage
uv run pytest --cov=opentelemetry_mcp --cov-report=html

# Format, lint, type-check
uv run ruff format .
uv run ruff check .
uv run mypy src/

CI (.github/workflows/ci.yml) runs Ruff and mypy (strict) on every push, plus the full pytest suite.

mcpb pack mcpb/ builds the Claude Desktop bundle. mcpb/.mcpbignore is a deny-list (mcpb never consults .gitignore): every file under mcpb/ that it or mcpb's built-in patterns do not name is archived, tracked or not - pack from a clean tree (git status --short mcpb/ prints nothing) and check the printed file count before publishing a bundle.

Troubleshooting

Backend connection issues:

curl http://localhost:16686/api/services   # Jaeger
curl http://localhost:3200/api/search/tags  # Tempo

Authentication errors: confirm your key is set —

export BACKEND_API_KEY=your_key_here
# or: tracehub-mcp --api-key your_key_here

No traces found:

• Check the time range (use recent timestamps)
• Verify service names with list_services
• Try searching without filters first

Token usage shows zero:

• Confirm your traces have OpenTelemetry gen_ai.* (or legacy llm.*) instrumentation
• Inspect raw span attributes with get_trace

Datadog/Sentry-specific issues: see the troubleshooting notes under each backend in Configuration.

Roadmap

Both ideas that were previously listed here as deferred are now shipped:

• Agent-native triage — tools that flag a likely root cause rather than just returning raw trace data: see triage_trace in the Tools Reference above.
• Cross-backend correlation — querying a second, independently-configured backend and correlating results against it: see correlate_trace above. This ships as a best-effort, heuristic correlation (direct trace_id match, or a time-window/service-overlap fallback) rather than a guaranteed schema-level join, per the research behind it — trace-id continuity across a vendor boundary is not guaranteed even under normal W3C Trace Context propagation.

New Relic and Honeycomb have since shipped as the 7th and 8th backends, both built entirely from research/documentation without a live account to verify against — see their own setup sections for exactly what's unverified before relying on either in production; Honeycomb in particular carries a real go/no-go risk (its programmatic Query Data API is documented as Enterprise-plan exclusive). There is no further backend currently under research. (Grafana Cloud was never on this list — it already ships today via Tempo's Basic Auth path; see Grafana Tempo above.)

Carried over from upstream's older roadmap, re-prioritized behind the above rather than dropped: a dedicated model-vs-model comparison tool (today's get_llm_model_stats and compare_time_windows cover per-model stats and time-window diffing separately, but not a single tool that diffs two specific models directly), broader prompt-pattern analysis across templates rather than just version-over-version for one prompt (get_prompt_version_stats already covers the latter), MCP resources for common queries, and SigNoz/ClickHouse backend support. Cost calculation with built-in pricing tables and a query-result caching layer are shipped, not pending: every usage-reporting tool (e.g. investigate_cost_spike in the Tools Reference) returns a cost_usd/cost_usd_is_partial pair backed by a vendored litellm pricing table (src/opentelemetry_mcp/pricing/), and query results are cached with in-flight request coalescing via the QUERY_CACHE_TTL_SECONDS env var (see the configuration table above).

Everything else documented elsewhere in this README is shipped.

Contributing

Contributions are welcome. Before opening a PR, make sure:

1. All tests pass: uv run pytest
2. Code is formatted: uv run ruff format .
3. No linting errors: uv run ruff check .
4. Type checking passes: uv run mypy src/

License

Apache License 2.0 — see LICENSE. This project is a fork of traceloop/opentelemetry-mcp-server; full attribution and the fork relationship are documented in NOTICE.

Support

• Issues & feature requests: github.com/mcpsmiths/tracehub-mcp/issues
• Changelog: CHANGELOG.md
• Community: Traceloop Community Slack (upstream's community channel; general OpenTelemetry/GenAI-tracing questions welcome)
• Related: Model Context Protocol · OpenTelemetry GenAI semantic conventions · traceloop/opentelemetry-mcp-server (upstream)