Ceph AIops (preview)
Disclaimer: Community-maintained open-source project. Not affiliated with, endorsed by, or sponsored by the Ceph project or any storage vendor. Product and trademark names belong to their owners. MIT licensed.
Governed AI-ops for Ceph — talks to a vanilla ceph-mgr Dashboard REST API
(HTTPS :8443, username + password exchanged for a short-lived JWT at
POST /api/auth) with a built-in governance harness: unified audit log,
policy engine, token/runaway budget guard, undo-token recording, and
graduated-autonomy risk tiers. Works against stock ceph-mgr — cephadm,
Proxmox-hosted Ceph, or MicroCeph — with no croit and no Kubernetes
dependency. Self-contained: no external skill-family dependency.
Preview — mock-validated only, not yet verified against a live cluster.
What it does
The flagship analysis, plus the guarded reads and writes around it:
cluster_health— HEALTH_WARN/ERR root-cause analysis. Instead of echoing raw check codes (PG_DEGRADED,OSD_NEARFULL,SLOW_OPS,MON_DOWN,LARGE_OMAP_OBJECTS, …), it turns each active check into plain language: what it means, the likely cause, and the suggested next action. This is the differentiator vs the hobby Ceph MCPs that just proxyceph -s.- Governed destructive ops. The operations operators actually fear —
osd_purge,pool_delete,set_pool_size,rbd_image_delete— carry dry-run + double-confirm and a high risk tier; reversible tuning (osd_reweight,throttle_recovery,cluster_flag_set, pool quota/pg_num/ autoscale) records an undo descriptor capturing the prior state.
What works
- CLI (
ceph-aiops ...):init,overview,health detail/health status,osd tree/df/reweight/out/purge,secret set/list/rm/migrate/rotate-password,doctor,mcp.osd outandosd purgerequire--dry-run+ double confirm. - MCP server (
ceph-aiops mcporceph-aiops-mcp): the full 35 tools (17 read, 18 write), every one wrapped with the bundled@governed_toolharness. The CLI is a convenience subset; the MCP surface is the whole tool. - Encrypted credentials: the Dashboard password lives in an encrypted store
~/.ceph-aiops/secrets.enc(Fernet + scrypt) — never plaintext on disk. Unlock with a master password fromCEPH_AIOPS_MASTER_PASSWORD(MCP/CI) or an interactive prompt (CLI). - Reversibility: reversible writes capture the prior state and record an
inverse undo descriptor (e.g.
osd_reweight→ prior weight,set_pool_quota→ prior quota,throttle_recovery→ prior backfill/recovery settings). - Safety: destructive ops (
osd_purge,osd_mark_out,pool_delete,set_pool_size,rbd_image_delete,rbd_snapshot_delete) arehighrisk withdry_runand CLI double confirmation.
Capability matrix (35 MCP tools)
| Group | Tools | Count | R/W |
|---|---|---|---|
| Health | cluster_health (flagship RCA), cluster_status | 2 | read |
| OSD | osd_tree, osd_df, osd_perf | 3 | read |
cluster_flag_set (low, undo), osd_reweight (med, undo), osd_mark_in (med, undo) | 3 | write | |
osd_mark_out (high, dry-run), osd_purge (high, dry-run) | 2 | write | |
| PG | pg_summary, pg_dump_stuck, scrub_status | 3 | read |
trigger_scrub (low), trigger_deep_scrub (low) | 2 | write | |
| Pool | pool_ls, pool_df | 2 | read |
set_pool_quota (med, undo), set_pool_pg_num (med, undo), set_pool_autoscale (med, undo), pool_create (med) | 4 | write | |
set_pool_size (high, dry-run), pool_delete (high, dry-run) | 2 | write | |
| RBD | rbd_ls | 1 | read |
rbd_image_create (med), rbd_snapshot_create (low) | 2 | write | |
rbd_image_delete (high, dry-run), rbd_snapshot_delete (high, dry-run) | 2 | write | |
| CephFS / RGW | cephfs_status, rgw_status | 2 | read |
| Cluster-ops | mon_status, mgr_status, slow_ops, capacity_forecast | 4 | read |
throttle_recovery (med, undo) | 1 | write |
Totals: 35 tools — 17 read, 18 write.
Quick start
uv tool install ceph-aiops # or: pipx install ceph-aiops
ceph-aiops init # wizard: add a mgr target + store the Dashboard password (encrypted)
ceph-aiops doctor # JWT login + mgr-dashboard reachability
ceph-aiops overview # HEALTH status + active checks + OSD up/in
ceph-aiops health detail # decode the active HEALTH_WARN/ERR checks (RCA)
ceph-aiops osd df # per-OSD utilization, most-full first, near/backfill-full flags
Run as an MCP server (stdio):
export CEPH_AIOPS_MASTER_PASSWORD=... # unlock secrets non-interactively
ceph-aiops-mcp
Governance
Every MCP tool passes through the bundled @governed_tool harness:
- Audit — every call (params, result, status, duration, risk tier,
approver, rationale) is logged to
~/.ceph-aiops/audit.db(relocatable viaCEPH_AIOPS_HOME). - Budget / runaway guard — token and call budgets trip a circuit breaker.
- Risk tiers — graduated autonomy; high-risk ops (purge/delete/replica
change) can require a named approver
(
CEPH_AUDIT_APPROVED_BY/CEPH_AUDIT_RATIONALE). - Undo recording — reversible writes record an inverse descriptor.
Supported scope & limitations
- Deployments: vanilla ceph-mgr with the dashboard module enabled — cephadm, Proxmox-hosted Ceph, or MicroCeph. No croit, no Kubernetes dependency.
- Ceph has no ETag / pagination on the Dashboard API, so this tool exposes none — nothing is missing, the upstream API simply doesn't offer them.
- Preview / mock-only. Behaviour is validated against mocked Dashboard
responses. The cheapest live check is a single-node MicroCeph
(
snap install microceph→ bootstrap → loop-file OSDs) runningceph-aiops doctor; a 3-node Vagrant cluster exercises real rebalance behaviour. Multi-node rebalance and the write ops are unverified against a real cluster.
Missing a capability?
RGW multisite, per-daemon config sprawl, NFS-Ganesha exports, orchestrator (cephadm) host management — not here yet. Open an issue or send a PR — feedback and contributions are welcome.