Inference AIops (preview)
Disclaimer: Community-maintained open-source project. Not affiliated with, endorsed by, or sponsored by the vLLM or Ray projects or any inference-serving vendor. Product and trademark names belong to their owners. MIT licensed.
Governed AI-ops for GPU inference clusters — vLLM (OpenAI API + Prometheus
/metrics) and Ray Serve / Ray Jobs (Ray dashboard) — with a built-in
governance harness: unified audit log, policy engine, token/runaway budget
guard, undo-token recording, and graduated-autonomy risk tiers. It parses
vLLM's Prometheus /metrics directly (no Prometheus server required) and probes
the Ray dashboard independently. A bearer token is optional (many stacks run
open). Preview — mock-validated only, not yet verified against a live cluster.
What it does
The flagship value is root-cause analysis, wrapped in guarded reads and writes:
diagnose_latency_spike(flagship RCA) — when TTFT/TPOT/e2e latency climbs, it correlates queue depth (running vs waiting), KV-cache pressure / preemptions, and prefix-cache locality into a ranked cause plus the specific knob to turn (add replicas, raisemax-num-seqs, fix routing, enlarge KV cache). Every flag is a number, not a black-box verdict.diagnose_low_utilization— the inverse: idle GPUs, over-provisioned replicas, or routing that strands a cache-warm replica → what to scale down.- Prometheus-native — reads vLLM's
/metricsendpoint directly; no Prometheus/Grafana deployment needed. - Governance-grade — the first governance-grade entrant in this niche: audit + budget + risk-tier approval + undo-token + prompt-injection sanitize, with dry-run + double-confirm on the fragile prod ops (scale-down, scale-to-zero, drain, redeploy, hot-swap) the community reports as dangerous.
- Laptop self-test — ~80% of the tool self-tests free: vLLM on a single GPU
or CPU-mock + Ray in one local container (
ray start --head).
Capability matrix (30 MCP tools)
| Group | Tools | Count | R/W (risk) |
|---|---|---|---|
| Metrics & RCA | request_metrics, queue_depth, kv_cache_stats, diagnose_latency_spike, diagnose_low_utilization | 5 | read |
| Ray Serve (read) | serve_deployment_list, deployment_status, replica_list, autoscale_config_get | 4 | read |
| Ray Serve (write) | scale_replicas_up, scale_replicas_down, scale_to_zero, autoscale_config_update, drain_replica | 5 | write (med / high) |
| Models / vLLM | model_list, model_info, lora_load, lora_unload, model_hot_swap | 5 | read + write (med / high) |
| Ray cluster / jobs / GPU | ray_cluster_resources, ray_dashboard_status, ray_job_list, gpu_utilization, ray_job_cancel, replica_restart | 6 | read + write (med / high) |
| Deploy lifecycle | model_deploy, model_undeploy, deployment_redeploy, routing_policy_update | 4 | write (med / high) |
| Cost | cost_per_token | 1 | read |
16 read, 14 write. High-risk writes (scale_replicas_down,
scale_to_zero, drain_replica, lora_unload, model_hot_swap,
replica_restart, model_undeploy, deployment_redeploy) all support
dry_run + double-confirm; reversible writes record an undo descriptor.
Install
uv tool install inference-aiops # or: pipx install inference-aiops
Quick start
inference-aiops init # wizard: host + ray_port + vllm_port + scheme
inference-aiops doctor # probes BOTH the Ray dashboard and vLLM independently
inference-aiops overview # deployments + total replicas + queue backpressure
inference-aiops metrics diagnose # why is inference slow? ranked RCA + the knob to turn
inference-aiops serve list # Ray Serve deployments + replica counts
Run as an MCP server (stdio) for the full 30-tool surface:
export INFERENCE_AIOPS_MASTER_PASSWORD=... # only if a bearer token is stored
inference-aiops mcp
The CLI is a convenience subset (init, overview, serve …, metrics …,
secret …, doctor, mcp); the full 30 tools are exposed via the MCP server.
Governance
Every MCP tool passes through the bundled @governed_tool harness:
- Audit — every call (params, result, status, duration, risk tier,
approver, rationale) logged to
~/.inference-aiops/audit.db(relocatable viaINFERENCE_AIOPS_HOME). - Budget / runaway guard — token and call budgets trip a circuit breaker on tight poll/retry loops.
- Risk tiers — graduated autonomy; high-risk ops can require a named
approver (
INFERENCE_AUDIT_APPROVED_BY/INFERENCE_AUDIT_RATIONALE). - Undo recording — reversible writes (scale, autoscale-config, routing, hot-swap, LoRA load) record an inverse descriptor.
Supported scope + limitations
Preview / mock-only. All behaviour is validated against mocked vLLM
/metrics, vLLM OpenAI API, and Ray dashboard responses. ~80% of the tool
self-tests on a laptop — vLLM on a single GPU or CPU-mock plus a local
one-node Ray head. Not yet verified against a live production cluster.
Unverified against real hardware / topology:
- multi-GPU tensor-parallel / pipeline-parallel deployments,
- real GPU thermal / throttle telemetry (utilisation is best-effort from
the Ray dashboard's
/api/nodes), - multi-node drain and node-reboot orchestration.
The fastest live check is inference-aiops doctor.
Missing a capability?
This is the GPU-inference member of the AIops-tools family (governed AI-ops with audit + budget + undo + risk tiers). If a vLLM or Ray capability you need is missing, or your stack speaks a dialect these tools don't yet handle — open an issue or a PR. Contributions welcome.