A Model Scored 81.4% by Running git log: How to Read an AI Agent's Success Rate

When someone sells you an AI agent, a number usually comes with it. Ninety percent task completion. Seventy percent of tickets resolved without a human. State of the art on some benchmark.

Over the past year the research community has been taking those numbers apart, and the results are worse than "benchmarks are imperfect." In several documented cases the benchmark was not measuring task completion at all.

This matters to a small business more than to a lab. A lab that overestimates an agent publishes a correction. A business that overestimates an agent puts it in front of customers.

Eight benchmarks, near-perfect scores, zero tasks solved

In April 2026, Berkeley's Center for Responsible, Decentralized Intelligence published work in which they scored 73 to 100% across eight industry-standard agent benchmarks without solving a single underlying task.

The exploits were not exotic.

The SWE-bench line is the one to sit with. Ten lines in a single configuration file caused all 500 tests to report as passing. That benchmark is cited constantly in AI coding marketing.

The problem is not only deliberate gaming

Berkeley was attacking the benchmarks on purpose to show they were attackable. The more troubling finding is that models get credit for the same shortcuts without anyone intending it.

A survey published in August 2026, The Horizon Gap, which reviewed 1,547 papers from 2024 to 2026, reports that on SWE-bench 32.67% of successful patches involved solution leakage and a further 31.08% passed only because the test suite was too weak to catch a wrong answer. After filtering for both, one system's reported resolution rate fell from 12.47% to 3.97%.

Roughly a two thirds reduction in apparent capability, from correcting the measurement rather than changing the model.

The most direct example is in the same Berkeley report. IQuest-Coder-V1, a 40-billion-parameter open-source coding model, claimed 81.4% on SWE-bench. Researchers then found that 24.4% of its trajectories simply ran git log and copied the fix out of the commit history. The answer was sitting in the repository the agent had been handed, because the benchmark's shared environment made the shortcut trivial.

To be fair to that team: they re-ran the benchmark in the correct environment and published a corrected score of 76.2%, which is still a strong result for a model of that size. The problem was the evaluation setup, not a deliberate deception. That is rather the point. A five-point error appeared without anyone intending it, and it took outside scrutiny to surface.

The gap shows up whenever the test gets harder

You do not need exploit research to see this. You can watch the number move when someone builds a cleaner benchmark.

Top models score over 70% on SWE-bench Verified. On SWE-bench Pro, constructed to resist leakage and shortcuts, the same class of systems lands around 23%.

Same models. Same category of task. A third of the score.

The finding that should change how you scope work

The Horizon Gap survey makes a point that matters more operationally than any single percentage. Agents do not degrade gracefully as tasks get longer. On Vending-Bench the authors describe "high variance rather than smooth decay": capable models were either fine or completely derailed, with little in between.

The survey also reports that across seven representative long-horizon memory systems, including plain long-context models, RAG and memory-augmented frameworks, average accuracy under multi-target interference was 27.9%.

Note what the authors are careful about. They explicitly decline to claim a law of degradation, calling independent per-step error compounding "a useful null model, not an empirically confirmed law." We are repeating that caution because it is the sort of qualifier that gets dropped when a research finding turns into a marketing claim.

The practical consequence: a bimodal failure mode is much worse for a business than gradual decline. Gradual decline gets caught in testing. An agent that works fifteen times and then does something incoherent on the sixteenth passes your pilot and fails in production.

Questions to ask a vendor

• Which benchmark is that number from, and which version? SWE-bench Verified and SWE-bench Pro give the same systems wildly different scores.
• Is it measured on your data or theirs? A number from a public benchmark says nothing about your documents and your customers.
• What does failure look like? If they cannot describe the failure mode, they have not watched it fail enough.
• What percentage of runs need a human, and how does the agent know to ask? Bimodal failure makes the handoff the most important part of the design.
• Can we run a fixed test set of our own before committing? Any honest supplier will say yes.

What we do instead

We scope agent work to short horizons with a defined handoff. Not because long-horizon agents are impossible, but because the evidence says reliability falls off unpredictably with task length and there is currently no good way to predict where your cliff is.

In practice that means an agent that drafts a reply for a human to send, rather than one that runs a customer conversation unsupervised. It means automation that completes one bounded step reliably rather than a chain of six that works in the demo. We wrote about the same trade-off from the tooling side in n8n or custom code for automation.

There is also a well-evidenced middle path: compile the repetitive steps into deterministic code and reserve the model for the judgment. We cover the numbers behind that in stop prompting for the parts of your workflow that never change.

This costs a client some ambition and buys them a system that behaves the same way in month six as in week one.

If a vendor has quoted you a success rate, ask them which benchmark and which version it came from before you ask anything else. That single question separates the suppliers who measured something from the suppliers who repeated something. If you want a second read on the answer they give, send it to us.