Cost Per Token Is the Wrong Metric: The 26x Problem

Every AI pricing page, comparison table, and vendor pitch ranks models the same way: dollars per million tokens. It is the number in the headline of our own DeepSeek price increase analysis. It is also close to useless on its own, and the research quantifying why has been available for months while the comparison content ignores it.

The problem is simple to state. Per-token price is a rate. Your bill is a rate multiplied by a quantity. Models differ enormously in the quantity, and nobody publishes it.

The size of the gap

OckBench (arXiv 2511.05722) set out to measure reasoning efficiency directly, and its findings are more extreme than most practitioners assume. The paper reports models solving the same problems at similar accuracy while differing by up to 5x in token length, and in the worst cases by more than 25x: one model answering in roughly 1,600 tokens where another spends around 42,000.

The specific comparison that matters for anyone reading a pricing table: DeepSeek-V4-Flash (max) reaches 82.5% overall accuracy against GPT-5.4 (medium) at 82.0%, and consumes 26.3 times more tokens doing it. Half a percentage point of accuracy, twenty-six times the output.

The paper's broader conclusion is the one worth internalising. Top open-source models have largely closed the accuracy gap with commercial models while consuming up to 26x more tokens for comparable results, and smarter, larger models generally produce shorter responses, reaching correct answers with denser reasoning. Claude Opus-4.7 averages 7.5k tokens where other models spend 28.6k at nearly the same accuracy.

Token efficiency, in other words, tracks capability. The intuition that a cheaper model saves money assumes the token count holds constant across models. It does not, and it moves in the direction that works against you.

Why reasoning models make this worse

Reasoning tokens bill as ordinary output tokens on every major provider. There is no separate rate and no surcharge for enabling thinking. The cost effect is indirect: thinking produces more output tokens, and output typically costs three to five times input, so a verbose reasoning trace is charged at the most expensive rate on the invoice.

Longer reasoning also does not reliably buy accuracy. Work on test-time compute scaling (arXiv 2604.10739) finds models exhibit overthinking, where extended reasoning is associated with abandoning previously correct answers. Separate work on measuring reasoning effort (arXiv 2602.13517) concludes that raw token counts are an unreliable proxy for reasoning quality, and that increased generation length does not consistently correlate with accuracy. A 2025 survey (arXiv 2503.16419) catalogues the same failure mode across the field: reasoning models expend unnecessary effort on trivial queries, and excessive chain-of-thought can reduce accuracy.

So the expensive path is often also the wrong one. That is an unusual and useful property, because it means optimising for cost and optimising for quality point the same direction here.

A task-normalised number you can actually use

Artificial Analysis publishes something closer to what you want: the cost to run its entire Intelligence Index evaluation suite against each model, calculated from that model's real token consumption across the evaluations and its own input, cache, reasoning, and output prices. Version 4.1.1 spans nine evaluations including GDPval-AA v2, Terminal-Bench v2.1, SciCode, GPQA Diamond, and Humanity's Last Exam.

It is a fixed basket of work, priced per model. That is a cost per task, and the spread is instructive.

Two things fall out of that table. The cheapest model on the list to run the whole suite is an open-weight 26B model, and the same model family running at a higher reasoning effort more than doubles its token consumption and its cost. Effort setting is a pricing decision, and most teams never touch it.

How to measure this on your own workload

Published benchmarks tell you the gap exists. They cannot tell you its size on your prompts, because token consumption depends heavily on task shape. Here is the measurement that takes an afternoon and is worth more than any comparison article, including this one.

• Build a fixed basket of 50 to 100 real tasks from your production traffic, with known-good outputs. Not benchmark questions. Your extraction schemas, your documents, your edge cases.
• Run the basket against each candidate model and log input tokens, output tokens, reasoning tokens where they are reported separately, wall-clock time, and whether the output was correct.
• Compute cost per correct output, not cost per call. A model that is 30% cheaper per token and fails 15% more often is more expensive, because failures cost a retry plus whatever the bad output breaks downstream.
• Repeat at each reasoning effort setting the model exposes. The Luna figures above show effort roughly doubling token consumption within one model, which is a larger lever than switching provider.
• Re-run it quarterly. Prices moved twice in July 2026 and again on 16 August. Any migration decision older than a quarter is running on stale numbers.

What this changes in practice

The practical conclusion is that provider choice matters less than most cost discussions assume, and three other levers matter more: which tier of model you point at each task, what reasoning effort you allow, and how much output you ask for. Setting a sensible max_tokens, requesting a field or a list instead of prose, and disabling thinking on tasks that do not need it will usually beat any migration on savings, and none of them requires changing vendor.

It also means you should be sceptical of any comparison, ours included, that ranks models on a price table alone. The table is the rate. Your workload supplies the quantity, and the quantity varies by more than an order of magnitude. If you want this measured against your real traffic rather than a benchmark suite, that is the first thing our AI and automation team does on a cost review.