The Coding Agent Harness Now Matters More Than the Model It Runs
Quick verdict
The scaffolding around a coding agent, the harness, now drives its score more than the model checkpoint does. A fresh SWE-bench Pro comparison showed the same model swinging from 23% to 52% just by changing harness, and the harness that wins for one model often loses for another. If you pick a coding tool by the model badge on the box, you are optimizing the wrong variable.
What the numbers actually showed
An analysis of SWE-bench Pro runs from Joel Niklaus put concrete figures on something a lot of people suspected. Holding the model fixed and only changing the harness, the wrapper that manages prompts, tools, retries, and context, moved pass@1 more than many model upgrades do.
On GLM-5.2, results ranged from 23% to 52% depending on the harness. On Gemma 4 26B, the spread ran from 15% to 36%. Those are not rounding-error gaps. A bad harness can cut a capable model's usable performance roughly in half, and a good one can nearly double it.
| Model | Worst harness (pass@1) | Best harness (pass@1) |
|---|---|---|
| GLM-5.2 | 23% | 52% |
| Gemma 4 26B | 15% | 36% |
The part that breaks the usual intuition is transfer. You might expect a good harness to be good everywhere, so you could just find the best one and stop. It does not work that way. The rank correlation of harness performance across models came out at -0.05, which is statistically no relationship at all. The harness that tops the chart for one model can sit near the bottom for another.
The practical takeaway from the same run: a 26B model in the right scaffold can get close to a 744B model in the wrong one. That is a roughly 28x gap in parameter count erased by the wrapper. One more number for anyone watching their bill: 97% of the input tokens in these agent runs were repeated conversation prefix, which is why prompt caching is not a nice-to-have but a core cost lever.
Why it matters for what you pay
If harness beats model, then the cheapest path to good output is often not a bigger model. It is a smaller model in a well-tuned harness, with caching turned on. Databricks published its own version of this the same week: it cut internal AI coding spend by up to 90% in some cases while usage kept climbing. The breakdown was telling. Moving defaults to cheaper, more efficient models saved about 50%. Smart routing added about 30%. Giving users visibility and adaptive budgets added about 10%. Pruning context bloat and tuning the harness added the last 10%.
Notice what is missing from that list: buying access to a more expensive flagship. Every lever they pulled was about the system around the model, not the model itself. The same logic applies to individuals. If you are paying per token through an agent and the tool wraps your model badly, you are paying for tokens that a better harness would not have spent.
How to use this when you pick a tool
The headline benchmark number on a model is close to meaningless without knowing the harness it ran in. When you compare coding tools, the thing you are really comparing is the harness plus the model plus the caching policy, as one bundle. Two products can ship the same model and land in completely different places on a real task.
So test the tool, not the checkpoint. Run your own repo through it on a task you understand, watch whether it wastes turns re-reading files it already saw, and check whether it caches the prompt prefix or pays for it again every step. Those behaviors are harness decisions, and they are what you actually live with. Our rundowns of the best AI coding agents and Claude Code vs Cursor vs Codex are written around exactly these differences rather than the model badge.
Video: the model doesn't matter, the harness does
A short explainer on why the wrapper around a coding agent decides more of its behavior than the underlying model.
FAQ
What is a coding agent harness?
It is the software wrapped around the model that decides how prompts are built, which tools the agent can call, how it retries on failure, and how much of the codebase it keeps in context. Two tools running the same model can have very different harnesses.
Does this mean the model choice no longer matters?
No. A stronger model still helps. The point is that the harness now explains a bigger share of the variance than a single model upgrade, so it deserves at least equal weight when you choose a tool or try to cut costs.
Can a cheaper model really match an expensive one?
On these runs a 26B model in a strong harness approached a 744B model in a weak one. That will not hold on every task, but it means a small model plus a good wrapper is a serious option, especially when you factor in routing and cost.
Why does prompt caching keep coming up?
Because 97% of the input tokens in these agent runs were repeated prefix. Without caching you pay for the same conversation history on every step, which is one of the biggest hidden costs in agent workflows.
Sources
- @joelniklaus - SWE-bench Pro harness analysis and the 23-52% / 15-36% swings
- Patrick Wendell (Databricks) - how internal AI coding spend was cut up to 90%
- Ali Ghodsi - Databricks routing and budgeting breakdown
- Artificial Analysis - model intelligence and agentic indices for context
Further reading
Try all the models mentioned in this article
Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Free to start.
Start free on Admix