Should You Move a Production Agent to a Cheaper Model?

A model launched yesterday that costs less than the one running your agent. That happens about once a month now, and each time somebody proposes a migration on the strength of a price table. Most of those migrations should not happen, and you can identify the ones that should in an afternoon.

Four measurements, in order. Any one of them can end the evaluation, and running them in this order means most evaluations stop before the expensive stage.

Gate 1: does your agent edit or author?

Benchmark averages hide this and it matters more than any other capability question. Qwen3.8-Flash-Next scores 62.5 on SWE-bench Pro, which hands a model an existing repository and asks for a patch, against DeepSeek-V4-Flash at 56.0. On NL2Repo-Bench, which asks for a repository built from a written specification, it scores 48.1 against DeepSeek at 54.2.

Same two models, opposite results, 12.6 points between them. Sort your agent's work into editing and authoring before reading any leaderboard, because the leaderboard averages the two together.

Gate 2: what share of your tokens does the model write?

Take a week of production traffic and divide output tokens by total tokens billed. Most teams guess this number and guess high. Classification and routing sit near 2%. Support chat sits near 20%. Long-form drafting runs past 60%.

Then check your cache-hit rate, because it can invert the answer. Luna prices cached input at $0.02 per million tokens. At a 90% cache-hit rate it beats Qwen below a 14.3% output share and loses above it. We worked that crossover through in Qwen3.8-Flash vs GPT-5.6 Luna.

Gate 3: does the licence permit what you sell?

This gate costs nothing to check and ends more evaluations than the benchmark work does. Qwen3.8-Flash-Next ships under Qwen Community License 1.0, whose second condition requires any model-as-a-service or AI work assistant business to obtain a separate licence from Qwen. No revenue threshold applies.

If you sell access to the model, or sell a standalone coding or office productivity product, read the licence before the benchmark. Qwen3.8-27B ships under Apache 2.0 and clears this gate outright. The full breakdown is here.

Gate 4: cost per completed task

Price per token describes one attempt. Your bill is attempts multiplied by price, and a model that fails more often costs more while looking cheaper. Take twenty real tickets from your backlog, run both models, and count completions rather than tokens.

Twenty is enough to catch a large difference and too few to settle a small one. If the two models land within a few percentage points, the migration is not worth the engineering time, and the argument for that is in cost per token is the wrong metric.

If all four gates pass

• Move one workload rather than the system. Pick the one with the highest volume and the least business risk.
• Keep the previous provider configured behind a flag for a fortnight. Model launches get withdrawn and rate limits change.
• Re-run your evaluation set rather than trusting the vendor table, since the vendor selling the model ran every published benchmark.
• Log cost per completed task from day one, because the number that justified the migration is the number that has to hold.

We run this on client systems every time a launch makes the case, and most of the time the answer is to stay. When we build AI agents the provider is a configuration value rather than a commitment, which is what makes the question cheap to answer each time it comes up.