hotato
Find what broke in your agent calls. Pin it and CI stays red until you fix it.
pip install hotato
hotato check --demo
hotato check ./call.wav
hotato vapi health
Zero config. Vapi, Retell, Bland, Synthflow, Millis, or local audio.
Works with any recording: mono, dual-channel, or a transcript. Timing math, not a judge. Offline. Free. MIT.
hotato.dev
What it finds
- Say-do gaps: the caller interrupts to cancel (a barge-in), the agent says "canceled", the booking tool fires anyway. hotato takes the turn timing from the audio and the tool call from your OTel trace.
- Latency spikes: the pause before a reply going from 800 ms to over 2 s.
- Dead air: that pause reaching 5 s, or the line going quiet.
- Talk-over: the agent starts a fresh utterance over the caller.
Quickstart
Vapi
pip install hotato
export VAPI_API_KEY=...
hotato vapi health --last 7d --output report.html
Open report.html: every critical incident, timestamped, and your Voice Stability Score.
Retell
export RETELL_API_KEY=...
hotato retell health --call-id CALL_ID
--call-id is required and repeatable: you name the Retell calls to pull.
hotato bland health, hotato synthflow health, and hotato millis health
follow the Vapi shape. Those stacks mix both voices onto one channel, so
they measure silence timing, dead air and latency gaps, each finding with
its measured confidence; barge-in and talk-over need two channels.
Local audio
hotato autopsy ./call.wav
Writes a self-contained HTML report to hotato-output/, plus the JSON
pin reads. Open the HTML in your browser.
From finding a bug to gating on it
autopsy turns one recording into timestamped incidents. scan reads a
folder and tracks the trend. When you are ready, move a finding into CI:
hotato pin turns one incident into a portable failure check, and hotato prove re-runs every stored check and fails the build rather than pass on
evidence it cannot re-read. Every verdict carries its own evidence across
five dimensions: outcome, policy, conversation, speech, reliability.
A pinned check re-measures its own stored recording, so it holds the build
red on that bug and catches an edited bundle, a loosened policy, or an
engine upgrade that scores the same bytes differently. Clearing it takes a
fresh recording of the same moment against the current agent: hotato drive <bundle> places that call on Vapi or Twilio, and
docs/RECAPTURE.md is the walkthrough for every other
stack.
Pin a bug →
For continuous use: run hotato vapi health on a schedule, and open
hotato console --production-db evidence.db to watch calls land live.
Wire it into CI
The exit code is the verdict: 0 pass, 1 fail, 2 refuse (could not tell).
on: [pull_request]
jobs:
hotato:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: attenlabs/hotato@v1.20.0
with:
contracts: contracts/
hotato-version: 1.20.0
contracts/ holds what pin wrote. Full workflow: docs/CI.md.
Point your agent at it
Point Claude Code, Cursor, or any coding agent at this repo: it reads
AGENTS.md and runs the loop end to end offline, no key. Over local
stdio the MCP server adds the scorer plus read/verify/propose tools:
uvx --from "hotato[mcp]" hotato-mcp (docs/MCP.md). Deploying is yours.
Nothing leaves your machine
hotato runs offline, on the machine that invokes it. The core is stdlib-only
Python: no account, no key, no network call of its own. Your traces, prompts,
and audio stay on your disk. Scoring with a language model you host yourself
is a separate opt-in add-on, outside that core.
Go deeper
The whole loop, command by command: docs/LIFECYCLE.md.
First touch to a CI gate: docs/GETTING-STARTED.md.
Feed it what you already have: docs/CONNECT.md ·
docs/TRACE.md · docs/SIMULATE.md.
What every verdict stands on: docs/EVIDENCE-CONTRACT.md.
Next to the hosted alternatives: docs/COMPARE.md.
The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder,
the fleet control plane -- lives under hotato lab (hotato lab --help).
The public commands are durable. hotato lab moves faster, and every
command name that worked before 1.17 still runs unchanged.
Specifications
| Property | Value |
|---|
| Footprint | ~10 MiB installed, 0 runtime dependencies (stdlib-only) |
| Reproducibility | byte-for-byte: the same recording, the same report |
| Exit codes | 0 pass · 1 fail · 2 refuse |
| Release integrity | OIDC Trusted Publishing + build-provenance attested |
| Runtime | offline, off the production data path |
Verify the measurement yourself
PYTHONPATH=src python3 -m hotato.benchmark \
--scenarios corpus/real/scenarios --audio corpus/real/audio
On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is 20 ms. Provenance: corpus/real/README.md · method: METHODOLOGY.md.
Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked NOT SCORABLE and refused (hotato trust --stereo call.wav). The full four-tier evidence policy (what each verdict stands on, per input) is docs/EVIDENCE-CONTRACT.md.
Contribute
Issues and PRs welcome: CONTRIBUTING.md · SECURITY.md · CHANGELOG · docs/
License
MIT (LICENSE)
Know when to pass it on.
mcp-name: io.github.attenlabs/hotato