Agent Observability

OpenSSF Scorecard

Record your agent's LLM calls once, replay them offline in under 1 ms, zero API calls, zero cost.

Terminal recording of agent-trace recording a live HTTP call, then replaying the same run offline with zero network requests

Your LangGraph agent fails after step 8. LangSmith shows you what broke. To reproduce it: 8 more LLM calls. 30 more seconds. $0.15 more in API cost. If the failure was caused by a transient model output, you can't reproduce it at all.

Agent Observability fixes this. Record once. Replay offline in 0.93 ms. Zero API calls. Zero cost.

Recording overhead:   0.011%   (0.090 ms added per LLM call)
Replay latency:       0.93 ms  mean (vs ~8,500 ms live on GPT-4o × 10 steps)
Replay fidelity:      100%     (response bytes byte-for-byte identical)
CI cost per replay:   $0

Install

pip install agent-observability-trace-cli
# or
uv add agent-observability-trace-cli

LangGraph support:

pip install agent-observability-trace-cli[langgraph]

OpenAI Agents SDK support:

pip install agent-observability-trace-cli[openai-agents]

Terminal recording of installing agent-observability-trace-cli into a fresh virtual environment, then running agent-trace version and recording a first HTTP call with agent-trace list showing the resulting run

30-second CLI quickstart

# Record a live run (your script just needs `import agent_trace` somewhere)
agent-trace run --name my_agent -- python my_agent.py

# List recorded runs
agent-trace list

# Replay offline: zero network, zero cost
agent-trace replay run_<id>

# Show the trace for a run
agent-trace show run_<id>

list, inspect, diff, replay, and run all support --json for machine-parseable output, so an orchestrating agent or CI job can call any of them the same way a person would and parse the result. (run --json prints its own status to stderr and the child process's output to stdout, ending with one final JSON summary line, since the child's own output can't be made structured.) show has no --json mode of its own. It accepts --errors-only to filter its output to failed spans instead. See the full CLI reference below for every subcommand's flags.

Terminal recording of agent-trace subcommands run with --json, producing structured output an agent or CI job can parse directly

Want programmatic control instead of the CLI? Use the Python API:

from agent_trace import tracer
import httpx

@tracer.instrument(record=True)
def fetch_data(query: str) -> dict:
    with tracer.span("http-call") as span:
        resp = httpx.get("https://httpbin.org/get", params={"q": query})
        span.set_attribute("http.status_code", resp.status_code)
        return resp.json()

result = fetch_data("hello")
# Trace and fixture saved to ~/.agent-trace/runs/run_<id>/

Replay offline, no API calls, no tokens:

from agent_trace import replay

with replay("run_<id>") as ctx:
    result = fetch_data("hello")  # served from fixture, zero network
    print(result)                 # identical to the original run

Terminal recording of replaying a previously recorded run with zero network calls, then running agent-trace show to print the replayed span tree

MCP Server

agent-observability ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can list, inspect, and replay recorded runs directly, without a human invoking the CLI by hand.

Install the extra:

pip install "agent-observability-trace-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json):

{
  "mcpServers": {
    "agent-observability": {
      "command": "uvx",
      "args": ["--from", "agent-observability-trace-cli", "agent-trace-mcp"]
    }
  }
}

The server exposes one tool, run, that shells out to the agent-trace CLI with the given subcommand and arguments plus --json, and returns the parsed JSON result:

run(["list"])
run(["replay", "run_abc123def456"])

Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local subprocess. Source: src/agent_trace/mcp_server.py.

Supported frameworks

LangGraph · OpenAI Agents SDK · CrewAI · AutoGen · LlamaIndex · Haystack · Agno · PydanticAI · Google GenAI Plus: any httpx.Client, httpx.AsyncClient, or requests.Session — no framework required.

CLI reference

agent-trace has 7 subcommands. Every subcommand accepts -h/--help for the same detail shown here.

agent-trace version

Print the installed version and exit. No arguments.

agent-trace list

List all recorded runs in the trace directory (~/.agent-trace/runs by default, or $AGENT_TRACE_TRACE_DIR).

agent-trace show <run_id>

Pretty-print the stored trace.json for a run.

show has no --json mode. It prints the trace (colorized via rich when installed, plain json.dumps otherwise), not a structured summary object.

agent-trace replay <run_id>

Enter replay mode for a run and print the resulting span tree, plus streaming timing, HTTP error exchanges, and the same cross-span diagnostics show prints (error classification, duplicate node spans, retry storms, misattributed spans, checkpoint durability, zero-task updates).

agent-trace inspect <run_id>

Auto-flag known malformed request/response shapes and cross-span anomalies for a run.

Terminal recording of agent-trace inspect auto-flagging malformed request/response shapes and cross-span anomalies for a recorded run

agent-trace diff <run_id_a> <run_id_b>

Diff two recorded runs' exchanges, matched by URL, highlighting field-level differences between request/response bodies, plus a restart-vs-resume check for a shared LangGraph thread_id (see issue #161).

agent-trace run -- <command> [args...]

Exec a child process with recording pre-enabled process-wide (AGENT_TRACE_AUTO_RECORD=1), so the first import agent_trace inside that process, even one owned by a third-party CLI like langgraph dev, starts recording with zero code changes required in your own agent code. Exits with the child process's own exit code.

The problem

A LangGraph run fails after step 8. Your trace in LangSmith or Langfuse shows what broke. But to reproduce it you have to re-run the entire agent: 8 more LLM calls, 30 more seconds, another $0.15 in API cost. If the failure was caused by a specific tool response or a transient model output, you can't reproduce it at all. You're debugging against a moving target.

Agent Observability solves this at the HTTP transport layer. It records every request and response verbatim to a local SQLite file. Replay serves those exact bytes back in sequence, in under 1 ms per exchange: same code path, same span tree, same failure. No API calls.

Use in CI: replay at zero cost

Record once. Commit the fixture. Replay in every CI run at zero API cost:

# tests/test_agent.py
import pytest
from pathlib import Path
from agent_trace import replay

FIXTURE_PATH = Path("fixtures/my_agent_run.db")

@pytest.mark.skipif(
    not FIXTURE_PATH.exists(),
    reason="Run: python scripts/record_fixture.py to generate the fixture"
)
def test_agent_answer():
    with replay(FIXTURE_PATH) as ctx:
        from my_module import my_agent
        result = my_agent("what is 2+2?")
    assert "4" in result

Set AGENT_TRACE_NETWORK_GUARD=1 in CI. Any HTTP call not in the fixture raises NetworkGuardError immediately, catching regressions before they hit production.

AGENT_TRACE_NETWORK_GUARD=1 uv run pytest tests/

What does this save you?

10-step agent × $0.15 per run × 10 debug sessions per week = $15/week in API costs
With Agent Observability CI replay: $0/week

At scale (10 engineers, each debugging 3 failures/week):
Before: ~$45/week, ~5 hours/week waiting for live re-runs
After: $0/week, 0.93 ms per replay

Why not just use LangSmith, Langfuse, or Helicone?

Short answer: they show you what happened. They can't reproduce it offline. LangSmith's VCR cassettes are Python + LangChain only, don't capture full wire bytes, and require a LangSmith account. Agent Observability works on any Python HTTP client, needs no account, and replays in 0.93 ms with 100% fidelity.

Most observability tools for LLM agents are observe-only: they show you a trace of what happened, but reproducing a failure still requires re-running the full agent against live APIs.

¹ LangSmith has LANGSMITH_TEST_CACHE / VCR cassettes (langsmith[vcr]) for Python + LangChain only. It captures HTTP to api.openai.com but not arbitrary HTTP clients, does not record full wire-level bytes, and requires a LangSmith account.

Choose LangSmith if your team is on LangChain and needs dataset management, prompt versioning, and human feedback loops.

Choose Langfuse if you want a fully open-source, self-hostable observability stack with strong Postgres-backed storage.

Choose OpenLLMetry if your team already runs on OpenTelemetry and wants standard gen_ai.* spans without adding a new observability system.

Agent Observability is not a replacement for dashboards and eval pipelines. It solves the specific upstream problem: reproducing a specific failed run without any LLM API cost, for any agent built on any Python HTTP client.

Try it with Docker

Agent Observability emits OTLP spans. Run a local observability stack to browse trace trees:

git clone https://github.com/RudrenduPaul/agent-observability
cd agent-observability
docker compose up -d

Starts three services (all optional):

• Jaeger (http://localhost:16686): OTLP span ingestion and trace UI
• Grafana (http://localhost:3000): dashboards and alerts
• Tempo (port 3200): long-term trace storage backend

Then point your exporter at the collector:

from agent_trace.exporters.otlp import OTLPExporter

# 4317 = OTLP gRPC ingestion endpoint
exporter = OTLPExporter(endpoint="http://localhost:4317")
exporter.export(trace)

Real failures record/replay catches

• Transient model output at step 6 causes a downstream tool to fail. Unreproducible with a re-run, trivial to replay.
• Rate-limit response at step 3 triggers a silent fallback path, visible only in the recorded fixture bytes, not in a live re-run.
• Tool schema serialization error before HTTP dispatch, caught by LangGraphTracer.on_llm_error even though it never reaches the interceptor (see Known Limitations).
• Non-deterministic tool ordering in a parallel branch: replay pins the exact sequence that produced the failure, so you're debugging the actual run instead of a fresh one.
• gRPC unary-stream response from Gemini that only fails on a specific chunk boundary, recorded once and replayed byte-for-byte instead of re-triggering a live streaming call each time.

Known limitations

Agent Observability's capture model is HTTP-interceptor-based (plus instrumented framework callbacks for the integrations under src/agent_trace/integrations/) and process-local. That model has real edges, stated explicitly here so they're clear before you hit one, not after:

• Process-local only. Recording/replay happens inside the Python process you import agent_trace into (httpx.Client(transport= RecordingTransport(...)), session.mount(..., RecordingAdapter(...)), or ReplayEngine.replay()'s monkeypatches, see src/agent_trace/interceptor/). It cannot observe or replay calls made by a third-party hosted service you don't run or deploy yourself (e.g. a vendor's own hosted chat assistant). It only sees your own process's outbound calls.

• gRPC coverage is partial. src/agent_trace/interceptor/grpc_hook.py patches grpc.secure_channel/grpc.insecure_channel (and the grpc.aio equivalents) to capture Gemini/Vertex AI traffic that bypasses httpx entirely. Unary-unary calls (e.g. GenerateContent) and sync unary-stream calls (e.g. StreamGenerateContent) are fully recorded and replayed. Client-streaming and bidirectional-streaming gRPC calls, and any grpc.aio streaming call, are not captured. Those go straight to the live network unintercepted, both during recording and (if attempted) replay.

• Capture starts once a request object exists. RecordingTransport. handle_request/AsyncRecordingTransport.handle_async_request (httpx_hook.py) and RecordingAdapter.send (requests_patch.py) only run once a fully-constructed httpx.Request/PreparedRequest reaches them. Any exception raised before that, while an SDK is serializing a tool schema, building headers, or otherwise assembling the call, or even earlier during plain Python object construction (e.g. TypeError from abc.ABCMeta when instantiating an abstract class incorrectly), happens entirely upstream of the interceptor's capture surface and produces zero fixture rows. A wired-in framework integration's own error callback (e.g. LangGraphTracer.on_llm_error) does still capture such pre-HTTP exceptions when they propagate through that framework's own try/except, so "invisible to the interceptor" is not the same as "invisible everywhere". It depends on whether a framework integration is wired in for the exception to pass through.

• No visibility into a framework's own print/display code. Exceptions raised inside local logging/printing/display machinery, e.g. rich Console output or IPython/Jupyter display hooks, triggered by a framework's own verbose=True logging, have zero HTTP traffic and zero framework callback surface. No existing or planned capture mechanism (HTTP interceptor, MCP stdio-transport hook, or any framework integration) observes this category of failure.

Security

• Supply chain: Releases are built and published via GitHub Actions (release.yml). SLSA Level 2 provenance via Sigstore OIDC signing is verified working (.sigstore.json bundles genuinely produced for every dist artifact); SBOM (CycloneDX JSON + XML) is generated and attached to the GitHub Release alongside the signed artifacts.
• Vulnerability scanning: dependabot.yml opens weekly pip and monthly GitHub Actions version-bump PRs. Dependabot security-advisory alerts, secret scanning, and secret scanning push protection are all enabled on this repo.
• Fixture safety: Fixture files at ~/.agent-trace/runs/ contain full HTTP request and response bodies, including API keys and prompt contents. Add .agent-trace/ and *.db to your .gitignore.
• Disclosure: SECURITY.md — report vulnerabilities to agent.obs.oss.security@gmail.com with a 48-hour response SLA.

Known upstream advisories

• chromadb (optional [crewai] extra only): GHSA for a pre-authentication code injection vulnerability affecting chromadb 1.0.0 through the current latest release (1.5.9). The upstream fix (chroma-core/chroma PR #7237) merged 2026-07-07 but has not shipped in any PyPI release since — there is currently no patched version to pin to. chromadb is pulled in only by the optional crewai integration extra (pip install agent-observability-trace-cli[crewai]), not installed by default, and this project never runs a Chroma server with an exposed HTTP API, so the actual exploit path (an attacker-reachable /api/v2/.../collections endpoint) does not apply to normal usage of this package. If you install the [crewai] extra and run your own Chroma server elsewhere, track the upstream advisory and update chromadb as soon as a fixed release ships.

  Future roadmap: once chromadb ships a release containing the fix, pin chromadb to that version immediately. If no fix has shipped by the next scheduled security review and the [crewai] extra sees negligible real-world usage, removing the extra entirely is the fallback under consideration to close this out for good.

FAQ

What is Agent Observability, and how is it different from a typical LLM tracing tool?

It's a Python library and CLI (agent-trace) that records every HTTP request and response your agent makes, verbatim, to a local SQLite fixture, then replays those exact bytes later with no network call. Most tracing tools, including LangSmith, Langfuse, Helicone, and OpenLLMetry, show you what happened during a run. Agent Observability additionally lets you reproduce that exact run offline, deterministically, without touching the live API. See "Why not just use LangSmith, Langfuse, or Helicone?" above for the full capability breakdown against those four tools.

How does deterministic record/replay actually work?

Recording patches httpx.Client, httpx.AsyncClient, and requests.Session at the transport layer (src/agent_trace/interceptor/) to capture every outbound request and response as raw bytes into fixture.db. Replay installs a FixtureClock (src/agent_trace/core/clock.py) and serves those same bytes back in the original sequence, so the code path, span tree, and timestamps all match the original recording. The benchmark numbers quoted above (0.011% recording overhead, 0.93ms mean replay latency, 100% fidelity) come from benchmarks/test_overhead.py, benchmarks/test_replay_vs_live.py, and benchmarks/test_fidelity.py in this repo, runnable yourself with uv run pytest benchmarks/.

How do I install it, and what platforms does it support?

pip install agent-observability-trace-cli, or uv add agent-observability-trace-cli. It requires Python 3.10 or newer and depends only on httpx and rich, no compiled extensions, so it installs anywhere those wheels do. CI (.github/workflows/ci.yml) passes on Ubuntu, macOS, and Windows, across Python 3.10 through 3.13. An npm wrapper, agent-observability-trace-cli (source under npm/ in this repo), is also published for teams that reach for npx/npm, but it still shells out to the Python CLI under the hood, so the Python package must be installed too.

How does this compare to LangSmith specifically?

LangSmith's LANGSMITH_TEST_CACHE (VCR-style cassettes, via langsmith[vcr]) is the closest built-in equivalent. It's Python plus LangChain only, captures HTTP calls to api.openai.com rather than any HTTP client, doesn't record full wire-level bytes, and requires a LangSmith account. Agent Observability works with any Python HTTP client, plus dedicated interceptors for gRPC, aiohttp, botocore, and WebSocket traffic, records full request and response bytes locally, and needs no account or hosted service. Pick LangSmith if you're already on LangChain and want dataset management, prompt versioning, and human feedback loops alongside tracing. Pick Agent Observability if the goal is reproducing one specific failed run at zero API cost, regardless of which SDK made the call.

What happens if replay can't find a matching fixture entry?

With AGENT_TRACE_NETWORK_GUARD=1 set, any request missing from the fixture raises NetworkGuardError immediately instead of silently falling through to a live call. The most common cause is an HTTP client constructed before the recording or replay context was entered, since the patch only applies to clients created inside the start_trace/replay block. See "Known limitations" above for the full list of edges, including partial gRPC streaming coverage and pre-HTTP exceptions that never reach the interceptor.

Does it capture agents built on non-Python frameworks?

No. Capture is a Python HTTP-transport interceptor plus instrumented callbacks for the integrations under src/agent_trace/integrations/ (LangGraph, CrewAI, AutoGen, LlamaIndex, Haystack, Agno, PydanticAI, Google GenAI, and others). It only sees traffic from your own Python process. Agents built in other languages, or calls made by a third-party hosted service you don't run yourself, are outside its capture surface.

Are fixture files safe to commit to version control?

Not by default. fixture.db contains full HTTP request and response bodies, which means API keys and prompt contents whenever they appear in headers or payloads. Add .agent-trace/ and *.db to .gitignore, and never commit a fixture recorded against a production API key. Strip or redact secrets first if you want to keep a fixture as a committed CI test asset.

Is this free to use commercially?

Yes. The project is Apache 2.0 licensed (see LICENSE), which permits commercial use, modification, and redistribution, including inside closed-source products, subject to the license's own attribution and notice terms. There is no separate paid tier or commercial license.

Contributing

• Read CONTRIBUTING.md before opening a PR
• Good first issues are labeled in GitHub Issues
• Replay engine (src/agent_trace/_replay/) requires 80% test coverage (correctness-critical)
• Interceptor (src/agent_trace/interceptor/) requires 80% test coverage
• GitHub Discussions for design questions and ideas

Apache 2.0. Contributions welcome.

Built by Rudrendu Paul and Sourav Nandy