Volante

A transparent, user-owned model router that checks its own answers (alpha). Volante routes a goal across the models you configured — enforcing hard capabilities first, ranking the rest on explainable metadata and evaluation evidence — and can then verify the result by RUNNING it: assertions derived from your goal, executed in a sandbox, reported check by check. Inventory, policy, credentials and decision traces stay on your machine. By default it answers in ONE call — across 72 paired runs, planning and synthesising scored -0.047 against that [95% CI -0.135, +0.041] at 8.8x the cost, so the interval bounds its best case near +0.04. --orchestrate opts back into the task DAG for the work that genuinely needs a tool loop, which one call cannot do.

What the measurements say, including the parts that do not flatter it. This project runs a 3-arm eval against itself and publishes the losses. Orchestration does not reliably beat a single strong model: an earlier release of this README claimed +0.289 at p<0.005, it failed to reproduce, and on the goal built to give decomposition headroom a stronger model in the same family scores 0.958 alone. Four predictive gates, a repair pass and a model-escalation policy were all built, measured, and discarded. What survived every test is narrower and real — the router, the reliability work (unparsable output 2 runs in 8 -> 0 in 8), and the verifier. Full numbers, method and corrections: eval/artifacts/README.md.

Selection quality is an evidence-based prediction, not a claim of a universal winner. Volante does not yet ship published representative cross-provider benchmarks or automatic score calibration. Built without an orchestration framework (no LangChain / CrewAI / LiteLLM).

A volante steers the game: the deep-lying midfielder who reads the whole pitch and sends the ball where it does the most good — then takes it back. One mind, many players.

Highlights

• Supervisor + explainable routing. An LLM plans a validated, acyclic task DAG; the router evaluates every configured model, rejects hard capability mismatches, scores the eligible set, and records the complete decision trace. Optional evaluation-derived quality profiles replace coarse metadata with evidence. quality, local, cheap, and cash_protect_quota are distinct routing objectives.
• User-owned, cross-provider control plane. AnthropicProvider and a generic OpenAICompatProvider speak to Anthropic, Google AI Studio (Gemini), Groq, OpenRouter, DeepSeek, Moonshot (Kimi), local Ollama, and any other OpenAI-compatible endpoint. The inventory and credentials stay in the user's environment.
• Scoped provider failover. Model/deployment, provider authentication/endpoint, exhausted-retry, and timeout failures are recorded and can move work to the next ranked eligible candidate, including planner/synthesizer fallbacks. Malformed or semantically invalid requests still fail fast instead of spraying the request across providers.
• Hybrid one-shot / agentic. Tasks run as a single call or as a model↔tool loop (run_python in a Docker-isolated sandbox when a daemon is available, withheld when none is — plus host-mediated fetch_url / read_file).
• Shared context. An append-only blackboard carries provenance; each task gets a scoped, budget-capped projection of only the dependency artifacts it needs.
• Streaming everywhere. Live token streaming through the supervisor, workers, and synthesizer, with per-task labels for parallel workers and cooperative early-stop.
• Optional Web UI. A small FastAPI + SSE app streams a run live in the browser (plan → per-task worker output → synthesis → result); runs with real providers or a no-key demo.
• Answers you can check. volante --verify derives assert statements from your goal, runs them against the result in the sandbox, and prints which passed and which failed — with the check text, not a badge, because a failing check is often the CHECK being wrong and only the line shows that. It reports three states, including "not enough evidence" when your goal states no expected result. Measured: 0 false positives and 0 false negatives across 8 goals against a coarse defect, 5 of 6 caught against edit-shaped ones. A clean report is evidence, not a guarantee.
• Cost & honesty. A CostMeter tallies per-model usage and cost, and propagates an estimated flag when a provider returns no usage.
• Forgery-resistant evaluation that reports its own power. A 3-arm eval (baseline vs. orchestration vs. single-agent) with a scorer that runs untrusted solution code under process + filesystem separation so a model cannot fake a passing score. Arms are compared PAIRWISE and every report prints a confidence interval and the smallest effect the run could have detected — added after three published numbers had to be corrected downward, all of them point estimates from designs too small to support them.
• Tested. 1200+ tests, zero-network by default (FakeProvider + local subprocesses), ruff-clean, mypy-clean.

Architecture

flowchart TD
    goal(["goal"]) --> S["Supervisor<br/>plan → validated task DAG<br/>(acyclic · typed · one_shot | agentic)"]
    S --> R["Router<br/>best predicted fit per task<br/>(hard constraints → explainable score)"]
    R --> P
    subgraph wave["wave execution · asyncio fan-out · fail-fast"]
        direction TB
        P["Projector<br/>scoped, budget-capped request<br/>(system + task + deps)"]
        P --> W["Worker<br/>one-shot"]
        P --> AW["AgenticWorker<br/>model ↔ tool loop<br/>(run_python · fetch_url · read_file)"]
    end
    W --> BB[("Blackboard<br/>append-only · provenance · latest-wins")]
    AW --> BB
    BB --> SY["Synthesizer<br/>combine artifacts → final answer"]
    SY --> result(["result<br/>+ CostMeter totals · usage · duration"])

    classDef io stroke:#8b5cf6,stroke-width:2px;
    classDef store stroke:#f59e0b,stroke-width:2px;
    class goal,result io;
    class BB store;

                    ┌──────────────┐
   goal ──────────► │  Supervisor  │  plan → validated task DAG (acyclic, typed, one_shot|agentic)
                    └──────┬───────┘
                           ▼
                    ┌──────────────┐   per task: hard-capability filter, then rank the
                    │    Router    │   predicted fit from configured evidence
                    └──────┬───────┘
                           ▼
        ┌───────────── wave execution (asyncio, fan-out cap, fail-fast) ─────────────┐
        │   ┌───────────┐   scoped, budget-capped request (system + task + deps)      │
        │   │ Projector │──────────────────────────────────────────────────────────► │
        │   └───────────┘                                                             │
        │        ▼                              ▼                                      │
        │   ┌─────────┐  one-shot          ┌───────────────┐  model↔tool loop         │
        │   │ Worker  │                    │ AgenticWorker │  (run_python sandbox,     │
        │   └────┬────┘                    └───────┬───────┘   fetch_url, read_file)   │
        │        └──────────────┬──────────────────┘                                   │
        └───────────────────────┼───────────────────────────────────────────────────┘
                                 ▼
                    ┌──────────────────────┐   append-only, provenance, latest-wins
                    │  Blackboard          │◄──────────────────────────────────────
                    └──────────┬───────────┘
                               ▼
                    ┌──────────────┐
                    │ Synthesizer  │  combine artifacts → final answer
                    └──────┬───────┘
                           ▼
                        result  (+ CostMeter totals, usage, duration)

Quickstart

Requires Python 3.11.10+ (the floor is a security boundary — see Providers).

Install it:

pip install volante          # the CLI and the library
pip install "volante[mcp]"   # ...plus the MCP server, for IDE use
volante --version
volante --list-models        # what your environment currently offers, offline

volante needs at least one configured provider before it can run a goal — see Providers for the two-line setup. Until then --list-models and --help work and everything else will tell you what is missing.

Or work on it: the clone gives you the tests, the Web UI, and a zero-key demo that the wheel deliberately does not ship.

git clone https://github.com/ribato22/volante
cd volante
uv sync --dev            # install deps + dev tools
uv run pytest            # 1200+ tests, no network
uv run ruff check .      # lint

# See it orchestrate end-to-end with ZERO API keys (FakeProvider demo):
uv run python examples/fake_provider.py

Then configure at least one real provider (see Providers) and run a demo:

cp .env.example .env     # fill in one provider, then `set -a; . .env; set +a`

uv run python demo.py               # show detected providers
uv run python demo.py orchestrate   # full supervisor → workers → synth, streamed live
uv run python demo.py agentic       # one cross-provider agentic coding task (run_python loop)
uv run python demo.py eval          # 3-arm eval suite

Example output

demo.py orchestrate streams every phase live, then prints the result (illustrative):

Orchestrate demo — planner/synth model=openai/gpt-4o-mini

(planning + workers + synthesis stream live)
[haiku] Threads run as one— / tasks bloom in parallel time, / the join gathers all.

STATUS: success

FINAL:
Threads run as one—
tasks bloom in parallel time,
the join gathers all.

cost: $0.001834

demo.py eval prints the 3-arm table (format_report). Here is what it actually printed on the last full run:

GOAL            WINNER          BASE   ORCH   AGEN
--------------------------------------------------
slugify         baseline        1.00   1.00   0.67
roman           baseline        1.00   1.00   0.70
calc            orchestration   0.96   1.00   0.27
csv_stats       baseline        1.00   1.00   0.57
json_flatten    baseline        1.00   1.00   0.70
textkit         baseline        1.00   1.00   0.47
ledger          baseline        1.00   1.00   0.80
toolbelt        baseline        1.00   1.00   0.70
debug_gauntlet  baseline        0.98   0.98   0.63
--------------------------------------------------
wins: baseline=8  orchestration=1  agentic=0  ties=0
totals: baseline $0.018108  orchestration $0.139354  agentic $0.099910
VERDICT: BASELINE

openai/gpt-4o-mini, 9 goals x 3 arms x k=3 = 81 real runs against a Docker sandbox, read from results-0.4.2-corrected.json — the self-describing artifact each run writes (models, k, per-goal scores, costs).

"VERDICT: BASELINE" is a cost tie-break, not a quality result. Orchestration matches baseline on eight goals and beats it on one; baseline takes those eight because it reached the same score for 7.7x less money. On quality the suite is a tie at the ceiling: 0.993 baseline vs 0.997 orchestration.

Earlier releases published orchestration at 0.919 here, losing four goals outright. That was substantially a grading bug of ours, fixed in cd94390: the scorer closed a fenced code block at the first ``` even when that fell inside a string literal, so a model that embedded the requested README inside its own module scored 0.000 with working code in it. It punished single-self-contained-block answers — the shape a synthesis pass emits — and therefore hit the orchestration arm far harder than the baseline arm. Every orchestration number published before that commit is a lower bound.

Reproduce it — this needs your keys and spends real money:

uv run python -m eval.run --k 5 --json results.json

Read that table with its limits, because they are large. Both arms now sit at the ceiling on this suite — baseline 0.993, orchestration 0.997 — so it can no longer discriminate between them at all. A tie at 1.00 is not evidence of equivalence; it is evidence the goals are too easy to measure anything. What these nine goals establish is "orchestration does not lose on tasks one model already solves in a single turn", which is worth knowing and is not the claim the idea rests on.

The claim needs a goal with headroom. resolve (in eval/tasks_depth.py, run with --suite depth) is one: implement a single function governed by seven overlapping precedence rules, graded on 48 cases balanced twelve-per-answer so a constant answer scores exactly 0.25. Same model, temperature 0, n=8:

Read the last column, not the mean. An earlier release of this README reported +0.289 at Welch t=3.54, p<0.005, from one batch of 8. It did not reproduce. Pooled over four batches the gap is +0.159, and the most recent batch REVERSED it: orchestration 0.476 against baseline 0.497. Between-batch drift on this goal is larger than the effect being measured, so this project does not currently have evidence that orchestration beats a single call, even on the goal built to give it headroom.

Worse for the claim, the headroom turns out to be model weakness. Running the same goal on a stronger model in the same family, same batch, n=6 each:

gpt-4o solves it alone, deterministically, and orchestration makes it slightly worse for 7.6x the money. So what decomposition was closing on resolve was a gap that a better model does not have. That is a real result about this engine and it is published here because it is true, not because it helps.

What survives: the reliability work is unaffected and stands on its own — unparsable output went from 2 runs in 8 to 0 in 8, and --verify never approved a wrong answer in 22 measured runs. Those are properties of the engine, not of a comparison between arms.

Everything above is one model on one class of small coding tasks, and orchestration costs 7.7x baseline. --prefer cheap now buys that back where it is not earning anything: it asks the planner to split only when splitting helps. Measured, n=5 per arm —

On a goal one model already aces it halves the bill for identical output. On the goal with headroom it costs 0.20 of score and saves nothing — which is the trade the flag is for, and why it is opt-in rather than the default.

It is opt-in for a measured reason: choosing automatically does not work. Three signals were tried and all three failed. The planner's own difficulty labels give resolve the same profile (medium:2, easy:1) as csv_stats, where baseline already scores 1.00. Told it may return a single task for a simple goal, it never abstains. Shown its own answer and asked to check it against the goal, it replies OK on work scoring 0.417. There is no signal at this tier, so the choice belongs to whoever knows what the task is worth. Whether it pays off for a stronger model, a larger task, or work that genuinely exceeds one context window is unmeasured, and this project does not claim it. The agentic arm still failed 2 of its 27 runs.

The benchmark's most useful output so far was not the score. It found three real engine bugs — a deterministic livelock in the agentic loop, an eval arm that failed a model for answering correctly without calling a tool, and a stall guard that gave up after one warning when a second one recovers the run — and fixing those took agentic terminal failures from 79% to 7%. It also refuted two of the author's own hypotheses (that a wider goal would break the ceiling effect; that raising the iteration cap would help). That is what the suite is for.

Usage

Volante ships three surfaces: a one-command CLI (the primary entrypoint), an optional Web UI, and an importable library. All three need at least one configured provider — see Providers — or (Web UI only) fall back to a no-key demo.

CLI (primary)

uv run volante "write a haiku about concurrency, then explain the metaphor"

volante streams the plan, each parallel worker's output (labelled per task), and the synthesis live, then prints a summary. Flags (volante --help):

Exit codes: 0 success, 1 run failure, 2 config error (e.g. no provider configured), 130 Ctrl-C (prints whatever partial output had streamed so far — never a raw traceback).

The summary reports billed_usd (real cash spent) vs. credit_usd (subscription/plan API-equivalent value, not cash), physical subscription_calls, and a per-task route trace. JSON mode also includes every eligible/rejected candidate, score component, selected model, and model actually executed after any fallback.

Web UI

uv sync --extra ui
uv run python -m webui          # then open http://127.0.0.1:8000

A small FastAPI + Server-Sent-Events app streams a run live in the browser — the plan, each parallel worker's output (labelled per task), the synthesis, and the final result with cost and model routes. It runs with your configured providers, or a built-in FakeProvider demo if none are set (no API key needed). This is a source-checkout feature — webui/ is not shipped in the built wheel/PyPI package.

VOLANTE_UI_HOST / VOLANTE_UI_PORT override the bind address. Volante refuses a non-loopback host unless VOLANTE_UI_AUTH_TOKEN is set, and refuses to carry that token over plain HTTP: set VOLANTE_UI_TLS_CERT + VOLANTE_UI_TLS_KEY to serve TLS directly, or VOLANTE_UI_TRUST_PROXY=1 if a TLS-terminating reverse proxy sits in front. For remote access also set VOLANTE_UI_ALLOWED_HOSTS to the comma-separated hostnames clients will use. VOLANTE_UI_MAX_GOAL_CHARS (default 20000) and VOLANTE_UI_MAX_CONCURRENT_RUNS (default 2) bound input and concurrency. The page inserts all model output via textContent only (never raw HTML), so streamed text cannot inject markup. A /usage page (linked from the header, gated by the same VOLANTE_UI_AUTH_TOKEN) shows the usage ledger below.

Monitoring usage

Every run from the CLI, the Web UI, and the MCP server best-effort appends one JSON line to a usage ledger — ~/.volante/usage.jsonl by default (VOLANTE_USAGE_LOG to relocate, empty to disable) — recording status, cash vs. plan credit, physical subscription calls, duration, the models used, and a truncated goal. This is how you monitor Volante when an IDE agent delegates goals to it over MCP:

uv run volante --usage            # recent runs + totals, newest first
uv run volante --usage --json     # the same as one JSON object

The Web UI /usage dashboard reads the same ledger (summary tiles + a recent-runs table). Set VOLANTE_LOG=debug (or info/warning/error) to raise engine diagnostics on stderr — for an MCP server that surfaces in the client's server-output pane (e.g. VS Code's Output → the Volante MCP server); it is silent by default and never writes to stdout.

Library

import asyncio
import volante


async def main() -> None:
    registry, providers, model_id = volante.build_providers_from_env()
    factory = await volante.make_verified_runtime_factory(
        registry, providers, model_id
    )
    runtime = factory()
    result = await runtime.aexecute("your goal")
    print(result.status, result.billed_usd, result.credit_usd, result.cost_estimated)


asyncio.run(main())

A Runtime is single-use: it runs one goal, and its accounting, route trace, and planner are all per-run. Call factory() again for each goal (a second or overlapping aexecute on the same instance is refused with a RuntimeError rather than silently mixing two runs' numbers). Read result.cost_estimated alongside the amounts — it is True when a provider reported no token counts for at least one call, so part of the figures is inferred from configured rates rather than reported by the provider.

The top-level volante package re-exports the common library API (Runtime, Registry, Router, RoutingDecision, ModelQualityProfile, inventory helpers, build_providers_from_env, make_verified_runtime_factory, RunResult, ModelInfo, Task, LLMProvider, ProviderError — see volante.__all__) so you don't need to reach into submodules.

See examples/ for runnable scripts — including examples/fake_provider.py, which needs no API key at all. For a guided tour with hardcoded goals, see the demo script: uv run python demo.py orchestrate|agentic|eval (walked through in Quickstart).

Using your Claude / ChatGPT subscription (no API key)

Volante can drive the official headless CLIs you're already logged into instead of (or alongside) a card-billed API key:

export CLAUDE_CODE_ENABLED=1               # needs `claude` installed and logged in
# CLAUDE_CODE_SYSTEM_PROMPT_MODE=replace is the default — makes `claude -p` behave as a
# raw completion; `append` breaks strict-JSON planning, so leave it unset unless you know why.
export CODEX_ENABLED=1 CODEX_TIER=3         # needs `codex login`
uv run volante "your goal"

⚠️ Subscription runs are cash-free but consume your interactive Claude Code / Codex quota — the same pool your interactive coding sessions draw from. A heavy orchestration run can trip a rate-limit pause. The default quality objective favors the highest predicted fit per task, which can lean on subscription models. Pass --prefer cash_protect_quota to mitigate this — it sends bulk/easy work to cheaper local/free-tier models and reserves subscription models for hard tasks only. A card-billed, free-tier, or local model as planner is recommended: subscription CLIs ignore temperature, so Volante retries planning with self-correction and can gate claude -p as planner behind a live parse-plan check (it only plans if it demonstrably emits valid plan JSON).

This drives the official headless CLIs (claude -p, codex exec) that you are already logged into — never the claude.ai / ChatGPT web apps. Scraping those web apps is not implemented (it would violate their Terms of Service).

In your IDE (VSCode) & MCP

Volante is a CLI first, so it already works in any editor's integrated terminal (uv run volante "…"). For VSCode there are two extra conveniences:

1. One-keystroke tasks. The repo ships .vscode/tasks.json. Open Terminal → Run Task… (or press ⌘/Ctrl+Shift+B) and pick:

2. MCP server — let the AI inside your editor call Volante. Volante ships an MCP server (volante_mcp/) exposing one tool, volante_run(goal, prefer?), that plans → routes → runs → synthesizes and returns the final answer plus an honest cash/plan-credit footer. Any MCP-capable assistant (Claude Code, Cursor, VS Code Copilot agent mode, Windsurf) can then delegate whole goals to Volante.

Install it clone-free (recommended), or from a source checkout:

# clone-free — uv fetches the published package + the `mcp` extra on demand:
uvx --from "volante[mcp]==0.9.0" volante-mcp

# or install it and run the console script:
pip install "volante[mcp]==0.9.0"   # then:
volante-mcp

# or from a source checkout:
uv sync --extra mcp && uv run --extra mcp python -m volante_mcp

Register it with your client. Claude Code — one command:

claude mcp add volante -- uvx --from "volante[mcp]==0.9.0" volante-mcp

Cursor / VS Code / Windsurf — add to the client's MCP config (e.g. .cursor/mcp.json, or VS Code's .vscode/mcp.json under a "servers" key):

{
  "mcpServers": {
    "volante": {
      "command": "uvx",
      "args": ["--from", "volante[mcp]==0.9.0", "volante-mcp"]
    }
  }
}

The server reads providers from the environment exactly like the CLI (including CLAUDE_CODE_ENABLED / CODEX_ENABLED), so configure at least one provider first — it does not fall back to a demo. A full branded VSCode extension is intentionally not shipped; the CLI, tasks, and the MCP server cover the same ground.

In the official MCP registry. Volante is published to the official MCP registry as io.github.ribato22/volante (a validated server.json manifest plus a publish-mcp.yml GitHub Actions workflow that re-publishes it via OIDC on each release). Directories such as mcp.so, PulseMCP, and Glama index from it.

As a Claude Code plugin. This repo also doubles as a plugin marketplace — one command wires the MCP server and a /volante:run slash command into Claude Code:

/plugin marketplace add ribato22/volante
/plugin install volante@volante

Requires uv on your PATH (the plugin launches the server with uvx). See plugins/volante/.

Other MCP clients (Codex, Cursor, Windsurf, Gemini CLI, Cline). These don't have a plugin marketplace — they consume MCP servers via config. Point them at the same launch command:

• OpenAI Codex CLI — codex mcp add volante -- uvx --from "volante[mcp]==0.9.0" volante-mcp (writes an [mcp_servers.volante] block to ~/.codex/config.toml; add providers with repeated --env KEY=VALUE).

• Gemini CLI — gemini mcp add volante uvx -- --from "volante[mcp]==0.9.0" volante-mcp (the -- is required because Volante's first arg is --from).

• Cursor / Windsurf / Cline / Roo — add the standard mcpServers entry to the client's MCP config (~/.cursor/mcp.json, ~/.codeium/windsurf/mcp_config.json, or the Cline settings):

  json

  { "mcpServers": { "volante": {
      "command": "uvx",
      "args": ["--from", "volante[mcp]==0.9.0", "volante-mcp"],
      "env": { "CLAUDE_CODE_ENABLED": "1", "ANTHROPIC_API_KEY": "sk-ant-..." }
  } } }
  

Set your providers in each client's env block (CLAUDE_CODE_ENABLED, CODEX_ENABLED, ANTHROPIC_API_KEY, OPENAI_COMPAT_*).

Claude Desktop & Smithery — MCPB bundle. A one-file MCPB bundle is attached to each release as volante-<version>.mcpb. Open it in Claude Desktop for a one-click install (it shows a provider-config UI). Or publish it to Smithery as a local server — because Volante is stdio (not a hosted HTTPS endpoint), that's the CLI bundle path, not the "publish a URL" web form:

# download volante-<version>.mcpb from the release, then (needs a Smithery API key):
npx -y @smithery/cli mcp publish ./volante-<version>.mcpb -n <your-namespace>/volante

The bundle wraps uvx --from "volante[mcp]==0.9.0" volante-mcp, so it runs locally and your subscription CLIs + API keys work as usual (needs uv on PATH).

Providers

Set environment variables for any subset. Every configured model becomes part of the user's model inventory considered by the router; the Anthropic > OpenAI-compat > Kimi > Ollama priority only chooses the default planner/synthesizer. See .env.example for the full list.

Model inventory and quality evidence

Volante never guesses which remote models an account owns. Provider catalog and entitlement APIs are inconsistent, can require extra permissions, and do not prove that a model is currently usable. Instead, the routing inventory is explicit and auditable:

# Comma-separated lists; the singular forms remain backward compatible.
export ANTHROPIC_MODELS=claude-opus-4-8,claude-sonnet-4-5
export ANTHROPIC_NAMES=anthropic/opus,anthropic/sonnet
export MOONSHOT_MODELS=kimi-k3,kimi-k2.6
export OLLAMA_MODELS=qwen2.5-coder:14b,llama3.2
export CLAUDE_CODE_MODELS=opus,sonnet
export CODEX_MODELS=gpt-5-codex,gpt-5-codex-mini

uv run volante --list-models
uv run volante --list-models --json

*_NAMES is optional and, when supplied, must contain exactly one canonical id per wire model. All configured models are registered; no default seed is silently added. The inventory command is offline: “configured/detected” does not claim live entitlement, quota, or service availability. At execution time, an explicit model/deployment-not-found or model-access denial marks only that candidate unavailable. A generic authentication or endpoint failure marks the provider unavailable. Volante records either fallback event and can try the next ranked model; malformed or semantically invalid requests still fail fast instead of blindly calling every provider.

Without measured evidence, quality is deliberately a prediction based on declared strengths, tier, context/output headroom, and task difficulty. Volante does not pad that prediction: a scoring component that says the same thing about every eligible model (task_fit when no model declares a specialized strength, reliability when no profile is configured) is given zero weight and its share is redistributed to the components that actually carry information. The route trace names the components it dropped, so a tier-driven ranking reads as exactly that instead of hiding behind a 45-point “task fit” constant.

Two levers turn those components back on.

1. Declare what a model is actually good at. Beyond the required strength for a task type (coding for code, reasoning for the rest), these optional tags raise task_fit and let peers of the same tier be ranked apart:

Set them per provider slot with the *_STRENGTHS env vars (for example ANTHROPIC_STRENGTHS=coding,reasoning,software_engineering,debugging) or per model in the overrides file. These are your declarations, not vendor claims Volante bakes in.

2. Calibrate from measurements you own. Convert scores you actually observed into a strict profiles file — no provider calls, so it costs nothing:

# measurements.json — one entry per observed run, each naming the goal it came from;
# a null score means "produced nothing usable":
#   {"anthropic/opus": {"code": [{"goal": "slugify", "score": 1.0},
#                                {"goal": "roman",   "score": 0.8}],
#                       "write": [{"goal": "incident_note", "score": 0.9}]},
#    "ollama/qwen":    {"code": [{"goal": "slugify", "score": 0.4},
#                                {"goal": "roman",   "score": null}]}}
uv run volante --calibrate measurements.json --calibrate-out quality-profiles.json
export VOLANTE_QUALITY_PROFILES_FILE=$PWD/quality-profiles.json

Name the goal on every entry. confidence counts distinct goals, not runs, because the profile's claim is about a task TYPE and the goal is the unit that claim generalises over — measured, five runs of one goal returned the identical score five times, so runs two through five carried no information yet moved confidence from 0.25 to 0.625. A bare number is still accepted for hand-written files, but a task type recorded that way has unverifiable breadth: it keeps the old run-count confidence and --calibrate warns that it may be overstated.

--calibrate averages per task type. It emits overall_score only when every task type was measured, because the router applies that field to every task type — including ones your measurements never touched — so deriving it from a partial sample would let your coding evidence stand in for research ability nobody looked at. With partial coverage the field is omitted, the router falls back to the coarse declared tier, and its trace says so instead of citing a profile that never looked. When coverage IS complete it macro-averages the per-type means, so an unbalanced sample (30 code runs, one write run) describes the model rather than your sampling. confidence follows the weakest-sampled task type and is capped below 1.0 — the router applies one confidence to every task type, so unrelated runs must not make a single observation read as certain. It is goals/(goals+3), so broadening coverage raises it and re-running the same goal does not. reliability_score is emitted only when you recorded null runs: deriving it from low scores would make the reliability component a copy of the quality component. Model ids must exist in your configured inventory (the loader is strict), and the output replaces rather than merges, so calibrate every model you care about in one file.

Where those measurements come from. eval/calibrate_models.py runs the eval suite's baseline arm — one model, one call, no orchestration — across several models and writes the measurements file for you. It spends real money (models x goals x k calls):

uv run python -m eval.calibrate_models --models gpt-4.1-nano,gpt-4o-mini,gpt-4.1 --k 3
uv run volante --calibrate measurements.json --calibrate-out quality-profiles.json

measurements.json and quality-profiles.json in this repo are a real run of exactly that (2026-07-29, 81 calls). Read them as a worked example, not as defaults: they describe three OpenAI models you probably do not have, and the router only loads a profile you point it at. They predate the goal field, so --calibrate warns that their breadth is unverifiable and keeps their original run-count confidence — the per-run goals were never recorded, and inventing an assignment now would fabricate the provenance these files exist to preserve. eval/artifacts/measurements-k5.json is a fully labelled run if you want to see the current format on real data.

What the measurements establish. Three OpenAI models produced a clean monotonic gradient on code — gpt-4.1-nano 0.951, gpt-4o-mini 0.990, gpt-4.1 1.000 — which agreed with the tier order the router already used, so calibrating within one family confirmed the heuristic rather than overturning it. Measuring a second family is what changed the picture:

glm-4.5-flash is last at code and second at analysis; gpt-4o-mini is nearly best at code and worst at analysis — 0.29 the wrong way on one task type, 0.22 the right way on the other, against a 0.016 within-family wobble that was noise.

How much weight that carries, stated plainly. The code figure is 9 goals x k=3 per model. The analyze figure is one goal — eval/tasks_text.py currently has a single item — at k=3 to 5, and every score sits in the lower half of that rubric's range. A swap this size on a single item is suggestive, not established; treat it as a reason to measure your own inventory, not as a result to cite. Whether models from different labs have genuinely complementary strengths is the assumption per-task routing rests on, and this is the first evidence here pointing at it — one goal's worth.

With that evidence loaded and the strongest model excluded, the router picks gpt-4o-mini for code and glm-4.5-flash for analyze — different models for different work, from measurement rather than from a tier constant. When the strongest model IS available it still wins both, because it is genuinely best at both; the swap shows up in everything below it, which is what a cost or quota objective actually chooses among.

Two limits worth knowing before you trust a profile of your own:

• The suite saturates. gpt-4.1 scored 27/27 perfect, so nothing above it can be measured and the gradient is compressed into the top 5%. At k=1 the weakest model scored a perfect 1.000 by luck; only k=3 separated them. Calibrate at k>=3, and expect a task set that everything passes to tell you nothing.
• One task type carries all four. Every goal in the coding suite is a coding task, so a code-only run measures only code. That used to leak: overall_score was derived from whatever was measured and then applied to every task type, so coding evidence silently became a research claim. It no longer does — a general claim now requires complete coverage — and the effect is visible in the scores: with a code-only profile loaded, research ranks exactly as it does with no profile at all, while code moves. eval/tasks_text.py adds goals for the other types.

The profile file format is the same one you can write by hand:

{
  "anthropic/opus": {
    "task_scores": {"code": 0.96, "research": 0.94, "write": 0.91, "analyze": 0.95},
    "overall_score": 0.95,
    "reliability_score": 0.98,
    "confidence": 0.9,
    "source": "team-eval-2026-07"
  },
  "ollama/qwen2.5-coder:14b": {
    "task_scores": {"code": 0.78},
    "overall_score": 0.70,
    "reliability_score": 0.86,
    "is_local": true,
    "confidence": 0.8,
    "source": "team-eval-2026-07"
  }
}

Save that strict JSON outside source control and set VOLANTE_QUALITY_PROFILES_FILE=/absolute/path/to/quality-profiles.json. Scores are normalized 0..1; supported task keys are code, research, write, and analyze. Unknown model ids, unknown fields, duplicate JSON keys, invalid/non-finite scores, and credential-like extra fields fail closed. A decision trace records the evidence source and caveat rather than claiming an empirically universal winner.

The repository includes a 3-arm evaluation harness, but Volante does not yet publish representative cross-provider empirical benchmark results or automatically calibrate quality profiles. Treat the configured scores as user-owned evidence, validate them against your own task distribution, and recalibrate them as models or endpoints change.

Plural family settings share their provider-level defaults. When two models in the same family have different hard capabilities, limits, prices, or tiers, declare those per canonical model id in a second strict JSON file:

{
  "anthropic/haiku": {
    "strengths": ["reasoning"],
    "context_window": 200000,
    "max_output_tokens": 64000,
    "supports_tools": true,
    "cost_per_1k_in": 0.001,
    "cost_per_1k_out": 0.005,
    "tier": 2
  },
  "anthropic/opus": {
    "strengths": ["coding", "reasoning", "long_context"],
    "context_window": 1000000,
    "max_output_tokens": 128000,
    "supports_tools": true,
    "cost_per_1k_in": 0.005,
    "cost_per_1k_out": 0.025,
    "tier": 4
  }
}

Set VOLANTE_MODEL_OVERRIDES_FILE=/absolute/path/to/model-overrides.json. Overrides may contain only the seven fields shown above; model ids must already exist in the configured inventory. Quality profiles carry soft, evaluation-derived evidence, while model overrides carry hard/economic metadata used for eligibility, projection, routing, and accounting. Neither file accepts secrets.

Agentic tool availability is also explicit. run_python is offered only when an isolating sandbox is available (see Security); set VOLANTE_FETCH_ALLOWLIST=example.com,docs.python.org to enable fetch_url, and set VOLANTE_READ_ROOT=/absolute/path/to/trusted/files to enable read_file. The supervisor declares required_tools per agentic task, and Volante rejects a plan or execution when those tools are not enabled or were never actually invoked. The hostname allowlist is not a complete SSRF defense against DNS rebinding/private resolution, and VOLANTE_READ_ROOT should point to a trusted tree without adversarial concurrent symlink changes.

Subscription CLI agents are opt-in and consume your interactive quota. Scraping claude.ai / ChatGPT is not built (ToS, fragile, ban risk). Instead Volante can drive the official headless CLIs you're already logged into — Claude Code (claude -p) and Codex (codex exec) — with no API key. This is off by default (CLAUDE_CODE_ENABLED=1 / CODEX_ENABLED=1 plus the CLI installed) and is never used by the eval. Honest caveat: claude -p and codex exec today draw from the same interactive subscription pool as the chat apps (not a separate/metered bucket), so a full orchestration run — and especially the 3-arm eval — can burn your Claude Code / Codex allowance and trip a mid-run hard-pause. Volante reports credit_usd (subscription value consumed) separately from billed_usd (cash), routes only hard/high-tier tasks to subscription (bulk work goes to local/free-tier), and caps physical subscription calls per run (VOLANTE_MAX_SUBSCRIPTION_CALLS, default 16, including the subscription-planner compatibility preflight, planning, retries, worker/agent turns, and synthesis). Under the default quality objective, any eligible subscription model may be selected; the hard-task reservation behavior applies only to --prefer cash_protect_quota.

Billing surface moves — re-verify before trusting it. Whether claude -p bills against the subscription pool vs. a metered API-rate credit bucket has flipped several times in months (announced 2026-06-15, then paused; still paused as of 2026-07-22). When Anthropic next announces a billing change, repeat the live gate in docs/claude-code-live-gate.md and re-check the Help Center banner, then update the "verified" date recorded there.

Free, high-intelligence option — Google AI Studio (Gemini Flash), via the generic slot:

export OPENAI_COMPAT_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
export OPENAI_COMPAT_KEY=<ai-studio-key>       # aistudio.google.com/apikey
export OPENAI_COMPAT_MODEL=gemini-flash-latest # pick a current model from the endpoint's /models
export OPENAI_COMPAT_NAME=google/gemini-flash
uv run python demo.py orchestrate

The generic slot defaults to context 128k, output 8k, tool support off, and cost 0. Set OPENAI_COMPAT_TOOLS=true only after confirming function-calling support, and set the provider's actual costs/tier/strengths; Volante registers that ModelInfo so routing and accounting use the declared metadata rather than a hidden seed.

Several models/providers at once — add OPENAI_COMPAT_2_*, OPENAI_COMPAT_3_*, … (each with its own model_id / pricing / context), or use the plural model lists above for Anthropic, Moonshot, Ollama, Claude Code, and Codex. Slots may point to different providers or to several models from the same compatible endpoint. For example, configure Gemini plus Groq so the supervisor plans on Gemini while a Groq model runs parallel workers. See .env.example.

Evaluation

demo.py eval runs a 3-arm comparison over 5 composite coding goals: baseline (one strong model, one shot), orchestration (the full engine), and agentic-single (one model + a run_python loop, no decomposition). Each goal is scored by a hidden reference test.

The scorer runs the model's generated solution.py in a subprocess under process + filesystem separation: a trusted runner drives the untrusted solution in a separate process that never sees the expected outputs (nonce-authenticated RPC), so a solution must actually compute correct answers — it cannot fake a passing score.

Read the verdict together with the warnings the harness emits:

• WARNING: some costs are estimated … — a provider returned no usage; cost comparison is soft.
• WARNING: agentic arm failed N run(s) … — a 0.0 may be infra/provider failure, not capability.
• WARNING: goal(s) […] produced NO trusted result … — the reference runner itself is broken; those scores are harness artifacts, not real zeros.

Security & limitations (honest)

Volante is alpha software; its isolation guarantees are deliberately scoped and documented.

• Code execution is secure by default, and fails closed. With no VOLANTE_SANDBOX set, Volante probes for a running Docker daemon: if one answers, run_python executes in a container (--network none, read-only root, dropped capabilities, cgroup limits). If none answers, run_python is not offered at all — the planner is told it cannot execute code — rather than quietly running model-written code with your files and network. The subprocess Sandbox protects against accidents, not adversaries (host network and disk stay reachable), so it is now an explicit opt-in: VOLANTE_SANDBOX=subprocess, which prints a warning on every start. An unknown VOLANTE_SANDBOX value is rejected instead of silently downgrading.
• External tools are host-mediated. fetch_url (hostname allowlist, no redirects, bounded body) and read_file (resolved-path root check, bounded read) run in the trusted orchestrator so sandboxed code stays network-isolated. The allowlist does not defeat DNS rebinding/private address resolution, and the root check is not race-proof against a hostile symlink swap; use trusted domains and trusted local trees. Prompt-injection containment holds only under Docker.
• Eval scoring is forgery-resistant, best-effort POSIX. Process + filesystem separation stops a solution from faking a score; a solution calling setsid() can still escape the killpg group (the wall-clock timeout still bounds the run). It is process isolation, not a security sandbox for arbitrary hostile code.
• Never put secrets in model context. Treat allowlists and the read-file root as explicit exposure controls, not as a general-purpose adversarial security sandbox.

Project layout

src/volante/     # engine (importable package: `volante`)
  providers/          # Anthropic + OpenAI-compatible adapters, FakeProvider
  tools/              # Sandbox, DockerSandbox, run_python, fetch_url, read_file
eval/                 # goals, 3-arm harness, forgery-resistant scorer, runner
examples/             # small runnable library-API scripts (incl. a no-key FakeProvider demo)
webui/                # optional FastAPI + SSE web UI (uv run python -m webui)
tests/                # 1200+ tests (unit + opt-in integration)
docs/                 # internal design/build records — see docs/README.md; not user docs
demo.py               # end-to-end demo (orchestrate | agentic | eval)

Development

• Test-driven, zero-network by default. uv run pytest uses FakeProvider and local subprocesses; integration tests that touch the network/Docker are marked integration and skipped by default (uv run pytest -m integration to opt in).
• Lint: uv run ruff check . (line length 100; E,F,I,UP,B).
• Contributions welcome — see CONTRIBUTING.md and our Code of Conduct. Security reports: SECURITY.md. Release notes: CHANGELOG.md.

Project status & non-goals

Volante is an alpha, transparent, user-owned model router and orchestration control plane. It is usable today through its CLI, library, Web UI, and MCP server for inventories and providers that the user explicitly configures. It is not a managed model gateway, an automatic entitlement-discovery service, or a guarantee that its predicted fit is empirically optimal. Interfaces may still evolve.

For production use, validate Volante's routing against a representative workload, supply and maintain your own quality evidence, choose the documented isolation mode for the threat model, and retain an application-level recovery path. The framework-free implementation is intentional: supervisor, router, projector, and blackboard behavior stays inspectable instead of being hidden behind LangChain, LiteLLM, or CrewAI abstractions.

Roadmap

• Publish representative cross-provider 3-arm benchmark results and interpret whether orchestration beats a single model.
• Calibrate task-specific quality profiles automatically from representative user evaluations.
• Async-generator/backpressure streaming API.
• Ollama tool-calling / streaming integration coverage.

License

MIT © 2026 ribato.