Terminal-Bench and Agentic Coding: Why the Harness Is Part of the Score

What Terminal-Bench adds

SWE-style benchmarks center on patch correctness. Terminal-Bench asks a wider question: can an AI agent use a terminal to complete hard, realistic work? The paper describes tasks such as asynchronous cleanup logic, program rewrites, and workflows that require iterative command execution.

That shift matters for Claude and GPT comparisons. A model can be strong at writing a patch in a constrained harness and weaker when it must inspect the system, run commands, recover from failures, and decide when to stop. Conversely, a strong agent scaffold can compensate for some model weaknesses.

Why public leaderboard rows are product-like

The Terminal-Bench public leaderboard names an agent, model, date, agent organization, model organization, and accuracy. That is the right unit for a buyer comparing shipped agents. It is the wrong unit for a claim that a base model alone is superior.

The July 6, 2026 public 2.0 leaderboard snapshot used for this site's baseline included GPT-5.5 entries above 80% and a Claude Opus 4.7 entry at 80.2%. Those rows are now historical on both counts: the model versions have been superseded and the benchmark has moved on. They remain useful as an illustration of how much the scaffold label matters — a Claude Code row, a Codex CLI row, a custom research harness row, and a neutral scaffold row are different products — and not as a current ranking.

The version problem is now the main problem

Anthropic's Fable 5.1 announcement reports Terminal-Bench 4.0: Fable 5.1 at 55.8%, Opus 5 at 52.3%, Fable 5 at 42.0%, and the trusted-access Mythos 5.1 at 60.9%. It also reports Terminal-Bench-Science 0.1, a separate task family, where Fable 5.1 reaches 52.6% against Opus 5 at 29.0%.

None of those numbers can be placed next to the 80%-plus figures on the public 2.0 leaderboard. Different version, different task set, different difficulty calibration, different harness, different reporting party. The apparent collapse from 80% to 55% is a property of the benchmark, not of the models, and any chart that plots them on one axis is producing a false narrative.

The practical rule is to fix the version before comparing anything. If you need a cross-vendor Terminal-Bench comparison at a specific version, the public leaderboard is the place to get it, because it at least applies one scoring pipeline to every row. Provider announcements will not give you that, and are not trying to.

How to use Terminal-Bench responsibly

First, decide whether you are buying a model or an agent. If you are adopting Claude Code, Codex, or another packaged agent, agent-plus-model scores are relevant. If you are building your own harness, prioritize results that resemble your scaffold or run the benchmark yourself.

Second, compare cost and elapsed time, not just accuracy. The Terminal-Bench paper notes that some tasks can involve hundreds of API calls and very large token usage. A model that wins by spending much more may still be wrong for a production queue. Fable 5.1's 2.5% cache-read rate changes that arithmetic for prefix-heavy agents, which is a cost property rather than a capability one.

Third, fix the effort setting. Fable 5.1 defaults to high effort in Claude Code but medium on Claude.ai and Cowork, so the same model can be benchmarked at one operating point and deployed at another. Record the setting alongside the score or the score is not reproducible.

FAQ

Cite this page