Coding Agent Index v2.0 resets the ruler: FrontierCode replaces DeepSWE, Opus 5.5 leads at 61.7

A stricter yardstick, not a worse generation of models

On October 23, 2026, Artificial Analysis released Coding Agent Index v2.0. The most important thing to say first is not who came first, but that the yardstick itself changed: the leg of the composite that most separates the leaders was replaced, and the grading rules were tightened. Scores fell across the board, and that is precisely what makes this update more honest — it does not pretend the models got worse, and it does not pretend old and new scores are directly comparable.

The headline first. Version 2.0 is led by Claude Code running Claude Opus 5.5, at 61.7. But that number only means something together with how it was measured: it was earned under a harder set of rules, closer to "could this change actually be merged into a real repository." Of the 21 configurations tested under both versions, 20 scored lower, and the mean fell from 54.8 to 47.4. This is not a collective slide. It is a reset of the measurement scale. Any comparison that puts a v2.0 score of 61.7 next to a higher v1.5 score as if they were on the same scale is comparing Celsius with Fahrenheit.

What the new index is made of

Version 2.0 covers 290 tasks across three equally weighted benchmarks. Each benchmark counts for one-third of the composite, regardless of how many tasks it has. Each benchmark score is the mean over three attempts per task, then averaged across tasks:

• FrontierCode v1.1: 100 tasks, software engineering, by Cognition. This is the new leg, replacing DeepSWE. It asks an agent to modify a real open-source repository, then grades the change against the standard of whether a maintainer would merge it — not merely whether the tests pass.
• Terminal-Bench 4.0: 66 tasks, agentic terminal use, by the Laude Institute. This leg keeps its name and task count in v2.0. It tests multi-step work in a command-line environment: using tools correctly and completing a full shell-driven workflow.
• SWE-Atlas-QnA-AA: 124 tasks, repository Q&A, by Scale AI, with grading adjusted by Artificial Analysis. It tests reading an unfamiliar repository, tracing code behavior, and producing a correct technical answer — not writing a patch.

100 plus 66 plus 124 equals 290. The structure itself is a judgment: Artificial Analysis treats a "coding agent" as a combination of three skills — changing repositories, driving a terminal, and explaining code — rather than a single skill. An agent that is strong at only one of them should not be able to hide weakness in the other two behind a single headline number.

Why FrontierCode is harder

FrontierCode is hard not because its questions are more obscure, but because it upgrades "it works" to "it is mergeable." Its scoring is binary and severe: an attempt that fails any blocking check scores zero, however good it looks otherwise. Only attempts that pass every blocking check earn between 70% and 100%, depending on non-blocking checks. A change can pass the tests and still score zero if it touches files outside the requested scope, makes an oversized diff, or lacks tests that actually establish the change's correctness.

The standard comes from Cognition's work with open-source maintainers: the 100-task main set is drawn from a pool of 150 tasks and is not publicly released, which reduces training-data contamination. The cost is clear — outsiders cannot audit individual tasks, and independent verification is limited to the methodology level. Artificial Analysis also scans agent trajectories: if an attempt fetched the upstream solution, such as the source pull request, its merge commit, or a mirrored copy, that attempt scores zero. Terminal-Bench has similar integrity rules: editing test files, writing directly to the verifier's reward file, or copying a reference solution counts as reward hacking.

My judgment: this "maintainer's view" is closer to real engineering review than the old "tests pass, full marks" rule, but it adds a new subjective layer. Which checks are blocking, and how the rest are weighted, are decisions made by the benchmark designers. Scores therefore reflect engineering judgment more faithfully, and also depend more on trusting those design decisions. Treating the number as absolute truth and dismissing it as meaningless are both lazy.

What changed in Q&A grading

The other easily missed change is in the Q&A leg. SWE-Atlas-QnA-AA keeps Scale AI's 124 questions and 1,306 rubric criteria, but Artificial Analysis reworded 127 of those criteria for more consistent verdicts, moved judging to GPT-6 Luna at the "xhigh" setting, and casts five votes per criterion. The dataset and judge configuration are published on Hugging Face, which deserves credit: the task set may be small, but at least the grading configuration is not a black box.

This change is not cosmetic. According to cross-reported figures, the revised Q&A grading alone lowered the mean of the 21 dual-version configurations by another 1.7 points, and individual configurations moved by as much as seven points in either direction under the new judge. That shows the judge is itself a variable: changing the examiner changes the ranking. Whenever a Q&A benchmark score moves in future, the first question should be "did the model change, or did the examiner change?"

Two different champions: the leader and the cost frontier

On the cross-reported v2.0 table, Claude Code with Claude Opus 5.5 leads at 61.7, with 56.0% on FrontierCode and about $11.42 per task. Next come Claude Code with Claude Sonnet 5.5 at 59.3 and Muse Code with Muse Wonder Full at 58.9. Notably, the Q&A leader is not the overall leader: Muse Code with Muse Wonder Full reaches 68.8% on SWE-Atlas-QnA-AA, above the Opus 5.5 combination's 65.9%. That is the point of an equal-weight composite — it rewards breadth, not a single-subject star.

But once cost enters the picture, the champion changes. On the same reported table, four configurations sit on the cost-performance frontier: Codex with DeepSeek V4.1 Flash at about $0.39 per task and 47.3 overall; Codex with GPT-6.1 Sol at about $1.50 and 54.1; Codex with GPT-6 Astra at about $7.04 and 54.7; and the Opus 5.5 combination at $11.42 and 61.7. In other words, climbing from the cheapest frontier point to the top score costs nearly 29 times as much for 14.4 extra points. For individual developers and small teams, that trade-off matters more than who is first: the larger your task volume, the more you should check where you sit on the cost curve instead of copying the leader.

An honest distinction is required here: the table above — including 61.7, 56.0%, $11.42, and the mean move from 54.8 to 47.4 — consists of figures from Artificial Analysis's October 23 announcement as relayed by multiple reports. I personally opened Artificial Analysis's public index page and confirmed the v2.0 composition, task counts, and equal weighting, but the text fetch of that page does not itself list this results table. These specific rankings and costs should therefore be cited as "announcement and reporting figures," not as numbers I re-verified row by row from the public leaderboard. Readers citing them should keep that qualification.

Why scores fell overall

Across the 21 configurations tested under both versions, the mean fell from 54.8 to 47.4 — a drop of 7.4 points, with 20 falling and one not falling. The bulk comes from the benchmark swap: the same configurations averaged only 45.1% on FrontierCode v1.1, versus 65.6% on the retired DeepSWE v1.1 — a gap of more than 20 points. The revised Q&A grading accounts for about another 1.7 points. The rest is configuration-specific movement under the new judge.

That breakdown is more informative than the headline. It says v2.0 is not "models got 7.4 points worse" but "the same models score 7.4 points less under a stricter ruler." More importantly, the 45.1% FrontierCode mean reveals the state of the field: even today's strongest coding-agent combinations average under half on the standard of "would a real maintainer merge this." The leader's 56.0% is a real lead, but it is far from "safe to auto-merge." Any claim that coding agents can already merge production code unsupervised does not survive contact with this table.

A methodology-page correction: it has been updated

One secondary analysis noted that the methodology page linked from the index still described v1.5, leaving the two pages inconsistent. When I personally opened the methodology page to check, the situation had changed: the page is now titled Coding Agent Index v2.0 Methodology, and its version history lists "Version 2.0: October 2026 — current," stating that FrontierCode v1.1 replaced DeepSWE v1.1 and that SWE-Atlas-QnA was renamed SWE-Atlas-QnA-AA with adjusted grading; v1.5 is now listed as a historical version, "September 2026 — October 2026." In other words, the claim that the methodology page still describes v1.5 may have been true at the time of that analysis's crawl, but it was already out of date when I checked.

This detail is worth spelling out because it exposes a common flaw in benchmark reporting: the leaderboard page, the methodology page, and the announcement post are three separately updated documents, and crawling them at different times yields different conclusions. Any citation of a benchmark page should state which page was opened and which version history was visible on that day, not just "the official site says." For this article, the check date is October 25, 2026, and on that day both the index page and the methodology page displayed v2.0.

Boundary with LSCBench: two different questions

A boundary must be drawn explicitly: the Coding Agent Index v2.0 discussed here is not the same thing as the LSCBench article previously published on this site, nor a restatement of the same evaluation. LSCBench is a secure-code benchmark focused on whether code is safe: it tested six frontier coding agents and found that even the best dual-pass rate was only 25% — it asks whether generated code introduces vulnerabilities and passes security checks. Coding Agent Index v2.0 asks a different question: how does a model-plus-harness combination perform across three kinds of work — changing real repositories, driving a terminal, and answering repository questions — with its FrontierCode leg asking whether a maintainer would merge the change. One is a specialist exam in security; the other is a general index of working capability. Their tasks, scoring, and conclusions are not interchangeable, and neither can be cited for the other. This article does not repeat LSCBench's conclusions and offers no new reading of its 25% figure.

The distinction matters for selection too: a high Index v2.0 score does not imply passing a security benchmark, and a cautious agent on security checks may not top the general index. Production decisions need both tables side by side — general capability from Index v2.0, the safety floor from LSCBench — plus testing on your own repositories. Three tables, none optional; deciding from a single table is deciding from a single subject's report card.

Three practical suggestions for vibe coders

First, stop citing v1.5 scores to endorse a v2.0 configuration. The version changed, the tasks changed, and the judge changed; old scores are valid only for the old version. Whenever you see a claim that something "rose" or "fell" versus last month, check that both sides carry the same version number.

Second, look at the component scores before the composite. If your main work is repository changes and pull requests, the FrontierCode component matters more than the composite. If you mainly explain code and onboard people, SWE-Atlas-QnA-AA is more indicative. If you automate heavily through the terminal, Terminal-Bench is your main arena. The composite is for people who do a bit of everything, not for people with a clear main job.

Third, price cost and waiting time into selection. A leading configuration at $11.42 per task becomes a real bill at dozens or hundreds of calls a day, while a $0.39 configuration that scores 14 points lower may cover 90% of everyday tasks. The pragmatic split is tiered: run volume on the cheap configuration, and reserve the top configuration for reviewing critical pull requests — not everything on the leader.

Two kinds of noise to watch

One is vendors re-interpreting a third-party index. Around the v2.0 release, Cognition claimed its Devin Fusion combination scored 68 on the Artificial Analysis index. That is a vendor claim: Artificial Analysis's own announcement figures do not confirm it, and other rows on the vendor's own chart score even higher. Any "a vendor says it leads on a third-party leaderboard" claim should be checked back against the leaderboard publisher's own page before deciding how much to believe. The 61.7 figure in this article at least has an Artificial Analysis announcement plus three independent reports cross-confirming it; the 68 figure has no equivalent confirmation and should not be cited alongside it.

The other is equating a benchmark score with production reliability. FrontierCode's tasks are private, its grading involves model judgment, its harness is the controlled Stirrup setup, and each task allows 250 turns — none of which matches your own repository, permissions, or CI pipeline. Benchmark scores narrow the candidate list; they do not replace running a small batch of real tasks on your own codebase. The leaderboard is a starting point, not a verdict. What 61.7 really means is "worth trying first," not "ready to use blindly."

Overall, Index v2.0 is a revision in the more honest direction: it puts "could this be merged" inside the composite score, making the numbers uglier and more credible. It does not prove any model has won. It proves the field's measuring tools are learning to be stricter with themselves. For developers, the best response is not a new slogan chasing the leader, but learning to read component scores, price costs, and run small tests of your own — the ruler changed, so the way we measure should change too.