Trace AI agent execution: every tool call, every error, every dollar. Open source, local-first.
io.github.KryptosAI/agent-observability — MCP Server
The MCP server io.github.KryptosAI/agent-observability provides trace-based observability for AI agent execution. It records every tool call, every error, and every dollar. The project is open source and designed to run locally, with no cloud required.
🛠️ Key Features
Traces AI agent execution
Captures tool calls
Captures errors
Tracks costs (“every dollar”)
Open source; local-first; no cloud required
🚀 Use Cases
Debugging agent failures via recorded errors
Auditing agent behavior through tool-call tracing
Monitoring spend associated with agent runs
⚡ Developer Benefits
Visibility into what agents did (“see what your agents did”)
Visibility into why they failed (“why they failed”)
Visibility into what it cost (“what it cost”)
⚠️ Limitations
The provided excerpt confirms local-first operation but does not specify supported environments, scaling behavior, or persistence/export options.
After every tool call, you MUST call the agent-obs MCP server's log_tool_call tool.
Include toolName, status (success/error), outputSummary, and durationMs.
Start each session with start_session and end with end_session.
Verify — Run agent-obs dashboard, open http://localhost:9400, and look for your session after the agent completes a task.
Errors prevented task completion, or agent abandoned the session
Decision points
When the agent has multiple possible actions, it can record why it chose one over another. Example:
json
{"context":"need to read a file — it could be at src/config.ts or lib/config.ts","options":["read_file src/config.ts","glob **/config.ts","grep 'CONFIG' *.ts"],"chosen":"glob **/config.ts","reasoning":"File might not exist at the expected path; glob guarantees finding it regardless of location"}
Decision points create an audit trail of agent reasoning, making it possible to understand not just what happened, but why.
Full audit trail
Every action is timestamped and linked to a session. The audit trail answers:
Who (which agent/user) accessed what data?
When did each tool call happen?
What data was read or modified?
Was the action authorized?
Grade System
Grades are assigned manually by the agent via obs_record_grade or automatically by the dashboard based on session statistics:
A = Clean — No issues detected. Every tool call succeeded, token usage was efficient relative to task complexity, and the session completed its stated goal.
B = Minor issues — Some inefficiencies (e.g., reading the same file twice, calling a tool that returned empty results and then trying it again). No failures.
C = Inefficient — Needs attention. Redundant tool calls, excessive token consumption, or recoverable errors that were eventually resolved.
D = Risky — Significant problems. Multiple failed tool calls, unusually high cost for the task, or the agent chose demonstrably wrong tools.
F = Failed — Errors occurred that prevented task completion, or the agent abandoned the session.
Grades accumulate over time. The /api/stats endpoint shows your average grade and grade distribution.
Why This Exists
AI agents are a black box. You tell Claude Code or Cursor to fix a bug, refactor a module, or add a feature — and minutes later you have a diff. But what actually happened in between? How many tool calls did it make? How many tokens did it burn? Which files did it read? Did it try three approaches before landing on one? You have no idea.
This isn't just academic curiosity. Without observability:
You can't estimate costs. A session that feels quick might have burned $5 in API calls. A session that felt slow might have cost $0.50. You're operating blind.
You can't debug failures. The agent produced broken code — was it because it read the wrong file? Used a deprecated API? Misunderstood the task? Without traces, you're guessing.
You can't improve your prompts. Is the agent reading too many files? Calling too many tools? Wasting tokens on irrelevant context? You need data to tune your system instructions.
You can't audit access. If an agent reads a file containing secrets or API keys, you should know about it. If it makes an unexpected network call, you need to see it.
Agent Observability gives you a flight recorder for every agent session. It answers: what happened, why it happened, and what it cost.
Provenance
This project combines ideas from three prior projects:
AgentShelf (Shopify AI readiness scanner)
AgentShelf scans a Shopify store's theme, apps, admin settings, and custom code to produce an AI readiness score. As it scans, it builds a full audit trail — every file read, every API call made, every finding logged with a timestamp and data source. Pattern adopted here: structured audit trail with per-action timestamps, data source attribution, and session-level summarization.
Veros (FHIR clinical ground truth)
Veros generates synthetic longitudinal patient records (FHIR R4) and uses them to validate clinical decision support agents. Its core principle: "no trace, no answer." Every snippet of clinical reasoning produced by an AI must be traceable back to the specific data element (lab result, medication, condition) that supports it. Pattern adopted here: decision point tracking — every agent choice must cite its basis, making reasoning auditable and contestable.
MCP Observatory
MCP Observatory secures MCP servers — testing them for vulnerabilities, schema drift, and attack surfaces before agents depend on them. It's used by 850+ developers weekly and powers CI pipelines for MCP server security. Use Observatory to secure your MCP servers. Use agent-obs to trace the agents that depend on them. Observatory validates; agent-obs observes.
Pattern adopted here: tool health scoring, session grading (A-F), and automatic degradation flags.
Open Source vs Cloud
This open source package (agent-observability on npm) is the core engine. It's MIT licensed and will always remain free and local-first:
CLI tools (proxy, server, dashboard)
Local SQLite storage
REST API
MCP server tools
Full audit trail, cost estimation, and session grading
A cloud version (app.agentobservability.dev) is in development with additional features:
Team dashboards with shared session history
Alerting (Slack/email when an agent's grade drops below C)
Historical analytics (cost trends, efficiency trends over months)
Role-based access control for audit trails
SOC 2 compliant infrastructure
The open source package will always be able to run independently. The cloud version is additive, not a replacement.