Cognition's FrontierCode Says the Best AI Coder Scores 13%, Not the 50% You've Been Sold

Cognition's FrontierCode Says the Best AI Coder Scores 13%, Not the 50% You've Been Sold

6 min readJune 10, 2026

Quick verdict

Cognition shipped a coding benchmark called FrontierCode that grades the thing teams actually care about: is the code mergeable, or does it just turn a test green. On the hardest subset the best model, Opus 4.8, scores about 13%. That is a long way from the 50%-plus numbers everyone quotes from SWE-Bench, and it is the most honest reset on "AI can code now" that we have seen in a while. Coding is not solved. It looked solved because the old benchmark asked an easier question.

What actually shipped

FrontierCode is a new evaluation built with open-source maintainers rather than scraped from existing test suites. Each task took 40-plus hours to construct, and the grading goes well past "did the unit tests pass." A submission is judged on regression safety, cleanliness, scope, test correctness, and maintainability. In other words, the same things a human reviewer checks before clicking merge.

That design choice is the whole story. SWE-Bench and its descendants reward a model for producing a diff that makes a failing test pass. FrontierCode rewards a model for producing a change a maintainer would accept into the codebase. Those are not the same task, and the gap between them is where the score collapses.

  • Top model on the leaderboard: Opus 4.8, at roughly 13% on the hardest subset.
  • Comparison point: 50%-plus is the normal range on SWE-Bench-style evals for frontier models.
  • Build method: tasks co-designed with open-source maintainers, 40-plus hours each.
  • Scoring dimensions: regression safety, cleanliness, scope, test correctness, maintainability.

The release did not land without pushback. Theo raised fair questions about variance and reproducibility, the kind of scrutiny any new benchmark should get before its number becomes gospel. Cognition responded, and Scott Wu and swyx both posted breakdowns of what the score is and is not measuring. The short version: treat 13% as a directional signal about how hard real merges are, not as a precise ranking you can shop models by.

Why it matters

For the past year the pitch for AI coding agents has leaned on benchmark numbers that imply the hard part is done. FrontierCode pulls that apart. Passing a test is the easy half. Writing a change that does not break three other things, stays in scope, and reads cleanly enough to maintain is the half that takes a senior engineer, and it is the half the models are still bad at.

If you ship software with these tools, the practical read is the one you probably already feel in your day-to-day. The agent is genuinely useful for bounded, verifiable work and still needs a human on the merge button for anything that touches a real codebase. That matches what we found comparing the leading tools in the best AI coding agents and in the head-to-head on Claude Code vs Cursor vs Codex: the differences that matter are about workflow and review, not raw leaderboard rank.

There is also a buyer angle here. If one benchmark can swing a model from "looks 50% solved" to "13% on the hard stuff," then picking a coding tool on a single headline number is a mistake. The better move is to run more than one model against your own code and keep the one that actually merges. That is easier when you are not locked into a single subscription, which is the case we make in the best app for running multiple AI models.

Video: benchmarking coding agents at the limits

This walks through why measuring coding agents against hard, human-level tasks gives such different numbers than the standard suites.

FAQ

What is FrontierCode?

It is a coding benchmark from Cognition that scores whether a model's output is mergeable, judged on regression safety, cleanliness, scope, test correctness, and maintainability, rather than whether it makes a unit test pass. Tasks were built with open-source maintainers and took 40-plus hours each.

Why does Opus 4.8 only score 13% when SWE-Bench numbers are above 50%?

Because the two benchmarks measure different things. SWE-Bench rewards a diff that turns a test green. FrontierCode rewards a change a maintainer would actually accept. The harder, more realistic question produces a much lower score for every model, with Opus 4.8 leading at about 13% on the hardest subset.

Does this mean AI coding tools are not worth using?

No. It means they are strong at bounded, verifiable tasks and still need human review before merge. The takeaway is to test models against your own code rather than trusting one benchmark headline. For lower-stakes work you can also lean on cheaper or free options, which we cover in the best free AI coding agents.

Sources

Further reading

Try all the models mentioned in this article

Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Free to start.

Start free on Admix

Related articles