readmeExcerpt: HTML-driven header and badge-rich introduction describing a curated list of AI agent harnesses and harness techniques for reliable agentic systems.
🚀 Use Cases
Provides recommendations and picks for agent harnesses.
Acts as a reference for search and decision guidance within MCP-related workflows.
Serves as a catalog entry for model-context-protocol ecosystem resources.
⚡ Developer Benefits
Clear, structured metadata aligned with MCP and model-context-protocol topics.
Readme excerpt offers quick context for integration and navigation.
Grounded in a curated set of harness-related tooling and frameworks.
⚠️ Limitations
Source data is limited to excerpted readme content and topic tags; depth may vary.
No runtime API details or configuration examples provided in this excerpt.
🤖 Agents can query this list — an MCP server (recommend, pick_harness, …), llms.txt & JSON, so your agent recommends harnesses too.
What is an agent harness?
A model answers; an agent acts. An agent harness is the runtime that turns one into the other: the model thinks, the harness decides what that thinking is allowed to touch.
Simon Willison's definition of the agent itself is the cleanest: "an LLM agent runs tools in a loop to achieve a goal." The harness is everything around that loop: which tools exist, what needs approval, what the model sees each turn, what survives a crash. Andrej Karpathy named the architecture back in 2023: the model is "the kernel process of a new Operating System", and the harness is the rest of that OS, its scheduler, permissions, and memory. The SWE-agent paper proved the stakes by coining the agent-computer interface: how tools and feedback are presented changes what a model can do, independent of the model. The field's advice has since converged on investing here rather than in framework plumbing, from Anthropic's build-simple guidance to Jerry Liu's argument that the framework era is over and the layers that matter now are skills, tools, and context quality. Those are the layers this list catalogs.
Why harnesses matter
Better models make harnesses more important: more capabilities mean more failure modes, and production needs retry logic, fallbacks, and validation. Harness quality, not just model quality, determines whether agents actually ship. This list ranks projects by relevance to harness concerns (environment, orchestration, lifecycle, guardrails) and by stars/activity.
The benchmark data now backs this up. On SWE-bench Pro, "swapping the agent harness changed pass@1 more than many model upgrades do" (AINews, Aug 8 2026, citing analysis by @joelniklaus). Same model, different harness: 23% to 52% pass@1 on GLM-5.2, and 15% to 36% on Gemma 4 26B. Harness rankings barely transfer across models (rank correlation -0.05), so a small model in the right harness can approach a much larger model in the wrong one.
That is the problem the MCP server in this repo solves. Point your agent at it and it can call recommend or pick_harness to choose a harness matched to your model and task, instead of inheriting whichever harness someone else benchmarked.
The landscape at a glance
Every project in the list, plotted by adoption surface area (the simplicity ↔ capability axis) against GitHub stars. Colors are categories; the largest projects in each tier are labeled.
The same projects placed by how much unsupervised rope they're designed to give (autonomy) and what happens when a run dies (recovery). In the tables below, ★ marks headless-ready projects and ✱ marks durable ones. Both charts regenerate from the list data on every refresh.
How to Pick a Harness
Start with the guide, then the head-to-head decision pages — grounded in the same data as the tables below:
How to pick a harness: six questions that turn this list into a decision, plus the chart to internalize first (the harness moves scores more than the model)
How to test-drive a harness: the two-week trial protocol, with a fair setup, tasks from your own repos, seven measurements, and the walk-away test
OpenClaw vs Hermes — the always-on personal-agent debate: presence vs discipline, plus what the field reports actually say
Reader's index: pick by what you want to do, not by category. Tag chips (e.g. mcp · memory) next to each row let you cross-filter by capability — see TAGS.md for the full cross-reference.
This list is also published in machine-readable form, so coding agents and research agents can recommend harnesses — not just humans browsing GitHub:
harnesses.json — every project with category, complexity tier, capability tags, stars, license signal, and a concrete example link, plus the full use-case index.
llms.txt — the entire list in one agent-readable file. Point any agent at the raw URL.
MCP server — recommend (one opinionated pick + alternatives + what to avoid, e.g. repos flagged for star manipulation), compare/compare_for (2–4 harnesses side by side — by id or by task — who leads on which axis incl. researched sandboxing/memory/hooks/prompt-optimization ratings, graveyard warnings, the matching decision guide), pick_harness (ranked, with complexity/autonomy/recovery filters), pick_infrastructure (picks at any level of the infra stack plus a live GitHub/Hacker News discovery pass, so answers aren't limited to this list), search_harnesses, get_harness, list_categories, plus list_comparisons/get_comparison for the decision guides. Published to PyPI and the official MCP registry as io.github.RyanAlberts/agent-harnesses. One-line install (needs uv):
sh
claude mcp add agent-harnesses -- uvx agent-harnesses-mcp
Or hire a skeleton
Don't just read the list — agents/ ships three agent skeletons: open-source agents that run on the AI subscription you already pay for. Clone the file, customize the instructions, done. All three work against the current week's data and deliver to Slack or Notion when either is connected:
harness-scout — describe what you're building; it picks your harness, with evidence and a graveyard check.
stack-auditor — flags the harnesses in your codebase that died, and can trace your agent session logs to show how the harness steers your technical decisions.
harness-radar — weekly movement briefing: climbers, arrivals, deaths, graduations.
⭐ Stars — GitHub star count, captured 2026-09-06; tables sort by stars descending.
⚖️ Simplicity ↔ capability — adoption surface, 4 tiers: super simple (a format, one concept) → mostly simple (thin layer) → slightly complex (real SDK) → complex (product suite).
★ Headless-ready — designed for unattended runs, batches, and fleets (the top of the autonomy scale: step-gated → checkpoint-gated → bounded → headless).
✱ Durable — persisted execution state survives restarts mid-task (the top of the recovery scale: none → retry → resumable → durable).
✅ Open source — ✅ standard OSS license · ⚠️ source-available/restricted · ❓ no or unclear license.
🏷️ Tags — capability chips auto-derived from descriptions; full cross-reference in TAGS.md.
🎯 Examples — one concrete "show me it in action" link per project, not a docs root.
Every project's full autonomy and recovery tier is plotted in the grid above and carried in harnesses.json and llms.txt; scores are editorial, from public docs — maintainer corrections via issue/PR are merged fast.
Formats, runtimes, and patterns that reveal context, tools, or instructions in layers—index first, details on demand—to control tokens and improve agent focus (the "map, not encyclopedia" principle).
Compresses tool outputs, logs, files, and RAG chunks with content-aware compressors before they reach the model—claimed 20% fewer tokens for coding agents and 60–95% fewer for JSON, same answers. Ships as a library, HTTP proxy, or MCP server, so it drops in front of whatever harness you already run. mcp · rag
Curated .cursorrules and skills that leverage Cursor's index-then-load model; the canonical collection for rules-as-progressive-disclosure in the IDE. ide
Open format for repo-scoped agent briefings; nested AGENTS.md files scope instructions per directory, so agents get a map of what exists and load only what's relevant. Read by 20+ tools including Codex, Cursor, and Copilot. ide · typescript
Context-window optimization layer that sandboxes tool output before it reaches the model (claimed 98% reduction) and persists session memory across 17 agent platforms via MCP and hooks—progressive disclosure applied to tool results, not just instructions. mcp · memory · sandbox
Build LangGraph agents with large tool sets; retrieval and on-demand tool loading so agents scale beyond context without stuffing every schema upfront. tool-discovery · python
Active tool discovery for autonomous agents: model requests tools by requirement; hierarchical semantic routing over 308 servers / 2,797 tools with ~98% token reduction (APIBank). tool-discovery
ICLR 2025: unified tool retrieval and calling via generation; 47k+ tools without context stuffing—retrieval and invocation in one generative step. tool-discovery · python
Semantic tool retrieval for LLMs; serves only the tools the user query demands (MCP-compatible), unlimited tool sets with zero context penalty. mcp · tool-discovery
Turnkey coding agents you install and run: IDE extensions, terminal CLIs, Dockerized workspaces. Each entry notes which part is the harness (the agent loop, tool wiring, approval model) versus the UI shell (VS Code extension, TUI, browser client).
Open-source terminal coding agent (formerly sst/opencode; transferred to anomalyco). The harness is a multi-provider tool-call loop (Claude, OpenAI, Gemini, local) with strong plugin and MCP support; the TUI is the shell. 100% OSS, very actively shipped. mcp · provider-agnostic · cli · tui · typescript
OpenAI's terminal coding agent. The harness is the sandboxed tool-call loop with multi-provider support; the CLI is the shell. Reference implementation for "official CLI that ships code." sandbox · provider-agnostic · cli
Google's first-party terminal agent for Gemini. The harness is the plugin/MCP tool-call loop; the terminal is the shell—Google's parallel to Claude Code / Codex, not just an API. mcp · cli · typescript
The upstream AI agent toolkit behind this list's oh-my-pi fork: a unified multi-provider LLM API, agent loop, and TUI shell providing the harness that oh-my-pi's Rust rewrite builds on. provider-agnostic · tui · rust
Dockerized software-engineering agent. The harness is the bash/editor/browser toolset with micro-agents and event-stream session bridging; Docker is the sandbox. Main OSS choice for teams self-hosting autonomous repo work. memory · browser · sandbox · python
Lightweight terminal coding agent oriented to open models (DeepSeek, Kimi, Qwen). The harness is a code-execution loop — the model writes code, the harness executes it with confirmation gates; the CLI is the shell. The original "let the LLM run code on my machine" project, reborn for open weights. cli · python
VS Code extension whose harness is a plan-then-act loop with per-step human approval and cost transparency; the VS Code integration is the UI shell. Open-source counterweight to Cursor. ide · typescript
Block-originated Rust agent, now stewarded by the Linux Foundation's Agentic AI Foundation (aaif-goose/goose). The harness is the MCP/ACP extension model with recipes and provider choice; there's no fixed UI slot—you bolt it into whatever shell you use. mcp · rust
DeepSeek-native terminal coding agent. The harness is engineered around prefix-cache stability for long-running sessions; the TUI is the shell. memory · cli · tui · typescript
Terminal coding agent (fork of Pi) that wires the IDE into the harness: hash-anchored edits, a 32-tool loop tuned per-model, LSP rename/references/diagnostics on every write, a real DAP debugger (lldb/dlv/debugpy), long-lived Python + Bun execution kernels that call back into the agent's tools, browser control, and 40+ providers (Claude/OpenAI/Gemini/local). ~55k-line Rust core. browser · provider-agnostic · cli · ide · rust
Kanban-style fleet manager for running Claude Code, Codex, or any coding agent across many tasks at once. The harness contribution is the task-queue/review layer on top of whichever agent executes; not an agent loop itself.
Charm's terminal coding agent (Charm's fork of the original OpenCode). The harness is the tool-calling loop with session persistence; the Bubble Tea TUI is the shell. memory · cli · tui
Alibaba's official terminal coding agent, forked from Gemini CLI's agent loop and retuned for Qwen models. The harness is the same sandboxed tool-call loop as its upstream; the terminal is the shell. sandbox · cli · typescript
VS Code extension and CLI in the Cline/Roo-Code lineage — a natural pick now that Roo-Code is archived upstream. The harness is an approval-gated autonomous-mode loop with a provider/tool marketplace; the IDE is the shell. mcp · cli · ide · typescript
OpenAI's harness for fanning a task out into many isolated, autonomous coding-agent implementation runs and surfacing the ones that pass, so a team manages outcomes instead of supervising each session. sandbox
VS Code/Cursor extension in the Cline lineage. The harness is the approval-gated agent with custom modes and a strong MCP story; the IDE is the UI. Popular community fork when you want that workflow without the upstream extension. mcp · workflow · ide · typescript
Open-source desktop harness positioned as a local, free alternative to Claude Cowork and Codex: multi-agent workspace orchestration in a self-hosted app rather than a hosted product. multi-agent · local
Local-first desktop workspace harness for Claude Code and other agents: multi-agent sessions, Git worktrees, code diffs, a skill marketplace, and chat-app access (WeChat, Telegram, WhatsApp). memory · multi-agent · typescript
Open-source AI IDE for Claude Code, Codex, OpenCode, and more. The harness contribution is the workspace/session orchestration layer: run multiple coding agents in parallel, locally or in the cloud, with isolated workspaces, reusable workflows, and shared team context. multi-agent · sandbox · ide · typescript
Runs multiple coding agents in parallel, each in its own sandboxed VM, locally or in the cloud, from one command. The harness contribution is the VM-per-agent isolation and fleet fan-out layer; whichever agent runs inside owns the loop. sandbox · typescript
Skill packs, slash-command libraries, meta-prompting frameworks, and official SDKs that give you the harness (the agent loop, planning, memory, hooks) without bundling a specific IDE or CLI shell.
Performance-oriented harness pack for Claude Code and 13 other harnesses (Codex, Cursor, OpenCode, Gemini CLI, more): skills, instincts, memory, security, research-first workflows. Treats harness engineering itself as the performance lever. memory · cli · ide
Anthropic's official Agent Skills repository: SKILL.md-based folders (instructions, scripts, resources) Claude dynamically loads on Claude Code, Claude.ai, and the API. The reference for progressive-disclosure skill packs in 2026.
Garry Tan's Claude Code skill stack: 23 slash-command modes (CEO/eng/design review, QA, ship, browse, retro, …) that structure one assistant as a virtual engineering team. Daily driver while running YC. typescript
Addy Osmani's production-grade skill pack: 24 engineering skills and 4 specialist agent personas that encode senior-dev workflows (spec through deploy) across 70+ coding agents including Claude Code, Cursor, and Copilot. The harness contribution is the skill/workflow layer, not a new agent loop. workflow · ide
Large community-curated index of Claude Code skills, slash commands, status lines, and plugins—resources for extending the harness, not a harness itself, but the most-followed catalog of the genre.
Cross-harness marketplace of drop-in subagents and skills for Claude Code, Codex CLI, Cursor, OpenCode, and Copilot; specialized, production-ready agent definitions you install rather than hand-write. multi-agent · cli · ide
Skill for persistent, file-based planning across long-running coding-agent sessions: crash-proof markdown plans, session recovery after /clear/compaction, and a deterministic completion gate—Manus-style planning as a drop-in harness layer via the Agent Skills standard. memory
LM-driven harness built for SWE-bench: edit state, command execution, and issue-focused loop—the reference agent stack next to the benchmark itself. memory · evals · python
Goal-backward planning and wave-based execution over fresh context windows; avoids context rot by design. Python/JS meta-prompting for Claude Code, OpenCode, Gemini CLI. cli · python
Google's official CLI and skill pack that layers agent-creation, evaluation, and deployment skills on top of whatever coding assistant you already run, rather than shipping its own agent loop—the harness as a config/skills add-on, not a new runtime. evals · cli
iFlytek's self-hosted registry for publishing, versioning, and governing agent skill packages—the harness config layer treated as an enterprise artifact store rather than a CLI or IDE shell. local · cli · ide
Reference implementation from the Meta-Harness paper: an academic testbed for harness-engineering research, not a product—useful as a citation-grade baseline rather than something you'd run in production.
Repo-scoped research harness: builds function-call and module-dependency graphs to explore only what's needed; large relative gains on MLE-bench and GitTaskBench with lower token use. workflow · python
Config layer that chains LLM councils for planning, Ralph loops for iterative refinement, and OpenCode worktrees for shipping. The harness contribution is the council → loop → worktree pipeline; OpenCode underneath executes. typescript
Claude Code config for AI product managers: CLAUDE.md plus skills for competitive analysis, PRD-from-signal, metric frameworks, stakeholder briefs, and agent eval design. "GStack for PMs." evals
Always-on, self-hosted agents you run as a daemon and talk to from chat apps: gateway runtimes, second brains, and self-improving assistants. The agent as a product you operate, not a library you build with.
Self-hosted, always-on personal agent (formerly Clawdbot/Moltbot): a gateway + event-loop runtime that treats messages, heartbeats, crons, and webhooks as one input queue, persists state to local files, and lives in your chat apps (WhatsApp, Telegram, Slack, Discord). 13,700+ community skills; the fastest-growing repo in GitHub history. typescript · multi-agent
Nous Research's self-improving agent: a learning loop turns experience into reusable skills, builds a persistent user model across sessions, and checkpoints state to disk with rollback; lean enough for a $5 VPS, driven from chat, and model-agnostic (Nous Portal, OpenRouter, OpenAI, or any endpoint). memory · python · provider-agnostic
Ultra-lightweight, self-hosted personal agent framework: the harness is a Python daemon wiring tools, memory, and MCP into chat/webhook front ends (Telegram, Discord, web); minimal footprint alternative to heavier personal-runtime stacks. mcp · memory · local · python
Self-hostable "AI second brain": answers over your docs and the web, custom agents, scheduled automations, and multi-client reach (web, Obsidian, Emacs, WhatsApp). A personal-agent harness with retrieval at the core. python
Open "agentic operating system" (elizaOS): persistent multi-agent runtime with character files, a plugin ecosystem, and social/platform integrations — the harness behind a large share of autonomous social agents. memory · multi-agent · typescript
Organic, prompt-defined personal agent framework: hierarchical sub-agents, persistent memory, browser and code tools, and self-modifying behavior; runs in Docker with a web UI. memory · multi-agent · browser · sandbox · python
❓
slightly complex (prompt-defined, Docker + web UI)
Open agent harness with a built-in personal agent ("Ohmo") that runs across Feishu, Slack, Telegram, and Discord; core tool-use, skills, memory, multi-agent coordination with auto-compaction for multi-day sessions. memory · multi-agent
Multi-platform personal agent living in Telegram, Discord, Teams, and the terminal. The harness is a pluggable-backend loop (Claude, Kilo, OpenCode, Codex, OpenAI Agents) with full MCP tool access and persistent background agents (Goals, Heartbeat, Dream); the chat apps are shells. mcp · memory · cli · typescript
Fair-code workflow engine with 400+ nodes and native AI nodes; the self-hosted Zapier that actually does agents and LangChain. workflow · local · typescript
The original autonomous loop: goal in, agent iterates with tools and memory; Forge is the dev framework, Benchmark the eval harness. memory · evals · python
One-stop LLM app platform: visual workflows, RAG pipeline, 50+ tools, model management; "ship from prototype to prod" in a single UI. low-code · rag · python
Python web-agent harness: natural-language goals become browser actions, driven directly over the Chrome DevTools Protocol (it dropped Playwright in August 2025). The biggest community in the browser-agent category. browser · python
Python agents with memory, knowledge bases, tools, and structured outputs; continues the PhiData-era product line under the Agno name—production apps, evals, and pipelines. memory · evals · python
Open-source orchestration framework for context-engineered LLM apps: modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation—closer to LangChain's territory than a coding-agent harness. memory · rag · python
Browserbase's SDK for browser agents: natural-language actions (act, extract, observe) and deterministic Playwright code mix in one script, so agent flexibility and repeatable automation live in the same harness. browser · typescript
Google's official Agent Development Kit: code-first Python toolkit for building, evaluating, and deploying agents. Optimized for Gemini but model-agnostic; deploys to Cloud Run / Vertex AI; ships a dev UI with eval and a code-execution sandbox. evals · sandbox · python
✅
complex (official Google SDK, eval, deploy — product suite)
RAG-first: hybrid search, knowledge graphs, multimodal; the framework for "production RAG" when you care more about retrieval than chat UI. vision · rag · workflow · python
AWS-originated orchestrator (now under 2FastLabs): intent classification, streaming, SupervisorAgent; "agent-as-tools" so one agent delegates to a squad. multi-agent
Task-solving and simulation envs for multi-LLM agents; deploy many agents in custom environments without building infra from scratch. multi-agent · python
Tencent Cloud's agent framework: a minimal tool-calling harness designed to perform well with open-source models, positioned as a lighter alternative to heavier orchestration frameworks.
Lightweight multi-agent orchestrator with an AGI-angle; minimal surface, docs-first, for teams that want orchestration without the kitchen sink. multi-agent · python
The "AI software company" multi-agent framework: role-played PM, architect, and engineer agents turn a one-line requirement into specs, designs, and code along an SOP assembly line. The landmark of the genre; development pace has slowed in 2026. multi-agent · python
Conversable agents and group chats; code execution and human-in-the-loop; Microsoft origin, AG2 ecosystem. ⚠️ In maintenance mode since late 2025: no new features, and Microsoft directs new users to Agent Framework. multi-agent · python
Open, invite-free general agent from the MetaGPT team: planning plus tool use over a multi-agent loop, aimed at reproducing Manus-style autonomous task completion on your own keys. multi-agent · python
Microsoft's convergence of AutoGen and Semantic Kernel: build, orchestrate, and deploy agents and multi-agent workflows in Python and .NET, with graph-based workflows and checkpointing — the designated successor harness for both lines. multi-agent · workflow · python
Self-hosted multi-agent harness aimed at production workloads: human-in-the-loop checkpoints and a self-improving agent loop, distinct from single-session coding-agent shells. multi-agent · python
Open-source meta-harness: orchestrates Claude Code, Codex, Cursor, Pi, and custom agents behind one policy/sandboxing layer so teams swap harnesses without rewriting workflows. sandbox · ide · python
AG2 (formerly AutoGen): the community-governed continuation of the original AutoGen project after Microsoft's fork diverged—conversable multi-agent groups, code execution, and human-in-the-loop under an open-source AgentOS banner. Graduated off the radar this cycle. multi-agent · python
Multitask, multiturn RL for LLM agents; Ray-based scaling, rollout/actor workers—for teams that want to train agents, not just run them. training · python
The official reference collection of Model Context Protocol servers (filesystem, git, fetch, memory, time, and more)—the canonical, vetted toolset agents connect to, and the pattern every other MCP server is measured against. mcp · memory · typescript
MCP server that injects up-to-date, version-specific library docs into an agent's context on demand; kills the stale-training-data hallucinations that plague codegen. mcp · training · typescript
Google's official Chrome DevTools MCP server: exposes console, network, and performance-trace inspection as tool calls agents can drive directly, instead of a human clicking through the DevTools panel. mcp · browser · typescript
Playwright's official MCP server: structured browser control (navigate, click, fill, extract) via the accessibility tree rather than screenshots, so web tasks stay fast and deterministic. mcp · vision · browser · typescript
GitHub's official MCP server (Go): repos, issues, PRs, code search, Actions. Replaces the older community cyanheads/github-mcp-server as the canonical way to give agents GitHub access. mcp
Official, community-driven registry for MCP servers—the "app store" MCP clients use to discover servers. Maintained by Anthropic + ecosystem maintainers; v0.1 API frozen, production-grade. mcp
Microsoft's policy-enforcement layer for autonomous agents: zero-trust identity, execution sandboxing, and OWASP Agentic Top-10 coverage sit in front of the agent loop as a harness security layer—the governance counterpart to Infisical's agent-vault. sandbox · python
IBM's official AI gateway/registry/proxy that sits in front of any MCP, A2A, or REST/gRPC API: unified endpoint, centralized discovery, guardrails, and plugin support—the enterprise front door for harness tool calling. Graduated off the radar this cycle. mcp · python
Embedded, tree-sitter/AST-based code-search CLI and MCP server that gives coding agents fast semantic lookups over a repo instead of grepping or re-reading whole files into context. mcp · cli
Infisical's HTTP credential proxy that fronts secrets for Claude Code, OpenClaw, and other agent harnesses so the agent's tool calls never see raw credentials—a harness security layer, not an agent loop itself.
Persistent decision memory for any project: remember, recall, reflect. Single-file Python CLI that stores decisions as JSONL and uses Claude or Gemini to retrieve and synthesize patterns—Karpathy's LLM Wiki concept as a CLI. memory · cli · python
Persistent memory layers that give agents recall across turns and sessions: knowledge graphs, vector stores, and session-capture tools that survive a restart. The state a harness needs but rarely ships with.
Session-memory plugin for Claude Code, Codex, OpenClaw, Gemini, Copilot, and more: captures everything an agent does during a session, AI-compresses it, and injects the relevant context into future sessions. Session-to-session memory as a drop-in. memory
Universal memory layer for AI agents: stores user/org/session memory, retrieves on demand. Apache-2.0; the de-facto memory primitive paired with most harnesses in 2026. memory · python
Zep's open-source memory engine: real-time temporal knowledge graphs that track how facts about users and entities change over time, so agents can answer "what was true when." The layer behind Zep's hosted memory platform. memory · rag · workflow · python
Open-source memory layer for agents: an extract–cognify–load pipeline that turns your data into a queryable knowledge graph plus vector store, so agents recall facts and relationships across sessions instead of re-reading context. memory · rag · workflow · python
Portable persistent-memory layer for coding agents: tracks decisions and task state outside the harness's own context window so it survives session resets and model swaps. memory
Microsoft's training-oriented harness: optimization loops for agent behavior—when you need to improve policies over rollouts, not only score a fixed prompt. evals · training · python
✅
complex (agent training, Microsoft stack — product suite)
Inspect AI core: composable eval tasks, sandboxes, scorers, and multi-model runs; the framework behind inspect_evals, not just the task bundle. evals · sandbox · python
Self-improving QA harness for web and mobile apps: natural-language tests, memory-backed self-healing, dashboard/CLI, MCP and skills support, plus sandboxed hooks for production regression checks. mcp · memory · sandbox · cli · typescript
The terminal-task benchmark coding agents now cite next to SWE-bench: hard, containerized terminal tasks scored end to end. Terminal-Bench 2.0 runs on the harbor evaluation framework; the 1.0 tasks live on in the org's terminal-bench-1 repo. evals · cli · python
Tracing, monitoring, and production evaluation for live agent runs: capture every step, tool call, and token, then score and debug in the loop. Distinct from the fixed-task benchmarks above—this is what you run against your own traffic.
Open-source LLM engineering platform: full-trace observability, online and offline evals, prompt management, and cost metrics for agent runs in production—the monitoring layer most harnesses lack out of the box. evals · typescript
Mature ML platform now covering GenAI: MLflow Tracing captures every agent step, tool call, and token, with built-in LLM evals and prompt versioning—observability for teams already standardized on MLflow. evals · python
Comet's open-source agent observability and evaluation platform: tracing, scoring, and experiment comparison with the whole core feature set free to self-host under Apache-2.0. evals · python
Arize's source-available, local-first tracing and eval layer: run it on your laptop or your own infra, and graduate to the managed Arize AX platform only when you need it. evals · python
ByteDance's long-horizon research harness: a LangGraph-based agent loop that researches, codes, and creates using sandboxes, memory, tools, and subagents behind a message gateway—an alternative to the reference gpt-researcher stack for multi-hour tasks. memory · multi-agent · sandbox · python
Autonomous deep-research agent: web + local sources, citation-grounded reports, multi-agent and deep-research modes. The reference open-source research harness. multi-agent · python
Fully autonomous, self-evolving research harness: multi-agent debate and citation verification carry a project from idea to a written paper without a human in the loop. multi-agent
Deep-research harness tuned for long browsing-and-reasoning chains; benchmarked on BrowseComp, GAIA, and HLE by pairing a dedicated agent loop with its own MiroThinker models rather than bolting search onto a generic chat agent. evals
Elastic dev environments for AI-generated code: workspaces, Git, previews—infra harness between "the model wrote a patch" and "it ran in a real machine." ⚠️ Public repo unmaintained since June 2026: core development moved to a private codebase (final open release v0.190.0, AGPL-3.0). sandbox
LangChain's Python+TypeScript agent harness on top of LangGraph: planning tool, virtual filesystem, shell sandbox, sub-agent spawning—the "Claude Code-style" harness as a reusable library. `multi
