AI Agent Benchmarks: What a Solved Task Actually Costs

Agent leaderboards publish two numbers side by side: how often the agent succeeds, and what the run cost. Almost nobody divides one by the other. Do that and the ordering changes, because an attempt that fails still bills you. What a buyer needs is not the price of a run — it is the price of a run that worked.

We took every result with both a success rate and a cost figure from two public leaderboards — SWE-bench Verified under the mini-SWE-agent scaffold, and terminal-bench@2.1 — and computed cost per solved task for each. Thirty results. The full dataset is available as CSV, and the script that produces every number below is in the methodology section.

The clearest case: two models, one price, 8.4x apart

On SWE-bench Verified, MiniMax M2.5 high and Qwen2.5-Coder 32B Instruct both cost $0.07 per attempt. On the leaderboard they sit in the same price column. MiniMax resolves 75.80% of tasks; Qwen2.5-Coder resolves 9.00%.

Per solved task that is $0.092 against $0.778. To be 90% confident of getting one task solved, MiniMax needs 1.6 attempts and Qwen2.5-Coder needs 24.4. Same headline price, entirely different economics.

The full SWE-bench cohort

Comparing across scaffold versions is unfair, so the table below is restricted to the thirteen models run on mini-SWE-agent v2.0.0 in February 2026. Same harness, same task set, so the only thing varying is the model.

Claude 4.5 Opus high tops the board at 76.80% resolved. MiniMax M2.5 high reaches 75.80% — one percentage point behind — at $0.07 against $0.75. Per solved task that is $0.092 against $0.977, so the leading model costs 10.6x more for one point of resolve rate.

Two cost metrics, two different winners

Cost per solved task answers "what will a batch of work cost?" It does not answer "what will this one ticket cost me?" For that you need the chance of failing repeatedly. Assuming independent retries, the attempts needed for a 90% chance of at least one success is ln(0.10) ÷ ln(1 − resolve rate).

The two measures disagree. GPT 5 mini is cheapest per solved task at $0.089, but reaching 90% confidence costs $0.139 because it needs 2.79 attempts. MiniMax M2.5 high costs slightly more per solved task ($0.092) but only $0.114 to reach the same confidence, needing 1.62 attempts. A higher resolve rate compounds in your favour exactly when you care most: a single task that has to get done.

terminal-bench@2.1: a statistical tie at 3.73x the price

terminal-bench@2.1 publishes total run cost and — unusually, and to its credit — a confidence interval on every score. That second column is what makes the leaderboard's own ranking questionable.

Rank 1 is Claude Code with Fable 5 at 83.8% ± 1.2, costing $552.67. Rank 2 is Codex with GPT-5.5 at 83.1% ± 1.1, costing $2,059.19. Those intervals are [82.6, 85.0] and [82.0, 84.2]. They overlap. The results are statistically indistinguishable, and one costs 3.73x the other.

Cursor CLI running Grok 4.5 is the cost-efficiency winner and it is not close. It places fourth on raw accuracy at 79.3%, and its cost index of 169 is 14.65x better than the most expensive result on the board. Against the top-ranked Claude Code with Fable 5, it gives up 4.5 accuracy points to save roughly a factor of four.

The scaffold matters more than the leaderboard implies

terminal-bench runs the same models under different agent scaffolds, which lets you isolate the harness. GPT-5.5 at xhigh effort scores 83.1% under Codex for $2,059.19, and 78.0% under Terminus 2 for $493.85. The Codex scaffold buys 5.1 points of accuracy for 4.17x the cost.

The comparison runs the other way too. Gemini 3.1 Pro scores 65.8% under Gemini CLI ($236.49) and 65.6% under Terminus 2 ($229.99) — the same result for the same money. So scaffold overhead is not a constant you can budget for. It is specific to the pairing, and it is invisible if you compare models on accuracy alone.

How we calculated this

• Cost per solved task = cost per attempt ÷ resolve rate. This is the expected cost of obtaining one success when each attempt is independent, which is the mean of a geometric distribution.
• Attempts to 90% confidence = ln(0.10) ÷ ln(1 − resolve rate). The number of independent attempts giving at least a 90% chance of one success.
• Cost to 90% confidence = cost per attempt × attempts to 90%.
• Cost index (terminal-bench) = total run cost ÷ accuracy. terminal-bench reports total cost for the whole task set rather than per attempt, so this index is proportional to cost per solved task rather than equal to it. Because the task count is identical across rows, the ranking is unaffected.
• SWE-bench figures come from the mini-SWE-agent v2.0.0 cohort (results dated February 2026) except in the full-board spread, which includes earlier scaffold versions and is labelled as such.
• Both leaderboards were retrieved on 21 August 2026. Every derived number is produced by a script from the transcribed source tables; nothing is hand-adjusted.

Limitations, stated plainly

Retries are not independent, so the confidence figures are floors rather than forecasts. A model that fails a given SWE-bench instance tends to fail it again — the failure is usually a capability gap on that specific problem, not bad luck. Real attempts-to-success will run higher than the numbers here, and the gap widens as resolve rate falls. Treat the 24.4 attempts for Qwen2.5-Coder as optimistic.

Three further caveats. Benchmark tasks are not your tasks — SWE-bench draws on public Python repositories with existing test suites, which is a narrow slice of software work. Published costs reflect the price at the time of the run, and inference pricing moves fast, so the ratios will age better than the absolute figures. And leaderboard results are self-reported by submitting organisations, with the incentives that implies.

What to do with this if you are buying

Ask a vendor what their agent costs and you will usually get a per-token or per-seat figure. Neither tells you what a completed piece of work costs. The question that separates a real quote from a hopeful one is: what is your success rate on our tasks, and what does one successful outcome cost including the failures?

The data also argues against defaulting to the top of the leaderboard. On both benchmarks the frontier models are within a few points of much cheaper alternatives, and those few points cost multiples. If your workflow has a human reviewing output anyway — which most production deployments do — you are paying a large premium for a marginal reduction in the review load. That trade is worth pricing before you commit, and it is the calculation we run with clients before recommending a model for an AI automation build.