Even the best model only gets 12% right: GDPx moves agents from exam papers back to the workplace

Why the number 12% deserves to be taken seriously

On November 2, 2026, SemiAnalysis, a chip and infrastructure research firm, released GDPx, a new benchmark for "general agentic capabilities." The strongest model on its debut leaderboard — Anthropic's Claude Opus 5.5 — scored 12.0%. Second was Google's Gemini 4 Carbon at 10.9%; third was OpenAI's GPT-6 Astra at 9.6%. Thirteen models had published scores, and the lowest, Z.ai's open-weight GLM 5.3, managed only 2.7%.

Twelve percent sounds like a failing grade, and that is exactly the point SemiAnalysis wanted to make. For years, model benchmarks have mostly graded answers: hand the model a prompt and a document, ask for a deliverable, and score it against a rubric. GDPx changes the exam: it drops the agent into a simulated workplace, gives it up to eight hours, and asks it to actually do the work — sifting through a private-equity deal spread across roughly 4,000 anonymized files, operating live services carrying real vulnerabilities, running verification inside chip-design tools. Work differs from exams in one fundamental way: getting 90% of the way there can still amount to zero, because the deliverable is wrong, a rule was broken, or the key file was never found. Under binary grading, nearly done does not count as done.

There is a second reading of the number to get straight first: it does not mean "AI can only do 12% of work." GDPx tasks are deliberately chosen hard, high-value professional tasks across 11 domains; the short, well-bounded jobs that take half an hour to explain are not on this paper at all. The correct reading of 12% is this: faced with a whole, undecomposed, multi-hour professional job, today's best agent has roughly a one-in-eight chance of delivering an acceptable result on the first attempt. Extrapolating that into "AI is useless" or "AI can replace everyone tomorrow" are both misreadings.

How the paper was set: 206 tasks, 11 domains, 14 partners

GDPx version 1.0 contains 206 tasks across 11 domains: chip design, cybersecurity, biology, finance, legal, quantitative trading, recursive self-improvement, hardware design, materials science, alignment, and software engineering. The tasks were built by 14 specialized data partners, with SemiAnalysis doing final quality checks. The domain selection criterion was not "cover the whole economy" but the firm's judgment about where frontier intelligence will earn the highest returns soonest; it explicitly left out industries such as government, real estate, and insurance. That choice is itself a position: GDPx is not trying to be a paper everyone likes, it is trying to be a paper about work that pays.

Three scoring rules matter. First, the headline score is pass@1 averaged over three attempts — the probability of getting it right the first time, not the best of three. Second, all 11 domains are weighted equally, as is each partner within a domain, regardless of task counts. Third, each task uses its intended grading method, binary or continuous; finance and legal scored essentially zero under strict binary grading, so the leaderboard shows partial scores there — but the underlying conclusion stands: no model produced a complete, approvable finance transaction or legal deliverable.

One methodological detail is essential to reading the ranking: models ran in their native harnesses where available. Opus 5.5 ran in Claude Code, GPT-6 Astra in Codex, Gemini 4 Carbon in Antigravity, and models without a native harness fell back to OpenCode. GDPx therefore compares model-plus-harness systems, not bare models. A model on a different harness could score differently; reading the leaderboard as a pure ranking of model intelligence misreads the methodology.

Reading the leaderboard closely: the winner is expensive, the runner-up is cheap, the third wins elsewhere

Start with the headline table. Claude Opus 5.5 leads at 12.0%, at an average of $51.41 per task and about 548,000 output tokens; Gemini 4 Carbon is at 10.9% and $16.29 per task; GPT-6 Astra at 9.6% and $53.99 per task. Fourth is Meta's Muse Wonder Full at 9.0% with no public price; below come Claude Fable 5.1 (8.5%), GPT-6.1 Sol (8.4%), Claude Sonnet 5.5 (8.2%), and Gemini 4 Argon (7.3%).

Put scores next to costs and the picture changes immediately. Doing simple arithmetic on the leaderboard's own figures: Opus 5.5 costs about $428 per successful outcome ($51.41 divided by 12%), Carbon about $149 ($16.29 divided by 10.9%). That arithmetic is this article's calculation from the published table, not SemiAnalysis's own claim, but it surfaces the question procurement should actually ask: you are not buying "one attempt," you are buying "one successful outcome." Carbon reaches about 91% of the leader's score at about 32% of the per-task cost, and about a third of the cost per success — with the caveat that it is not yet generally available, so that price is not yet a price everyone can pay.

By domain, the leaderboard fragments. SemiAnalysis's domain breakdown shows Opus 5.5 winning quantitative trading, hardware, and chip design; Carbon winning biology and narrowly software engineering; Astra winning recursive self-improvement and materials science. No model is champion everywhere. The selection lesson is direct: the headline winner is not necessarily the best choice for your domain; checking the domain slice for your own work beats copying the overall ranking.

There is also a counterintuitive footnote: SemiAnalysis says safety classifiers depress some models' scores. In cybersecurity, biology, and alignment, Opus 5.5 and GPT-6 Astra lost points because their classifiers refused some tasks; excluding those three domains, SemiAnalysis's alternative view puts Astra first and Carbon down to fifth. That alternative view describes "a different eight-domain paper," not "true capability with safeguards off," and it should not be treated as exoneration. But it is a reminder that a leaderboard score mixes capability, safety policy, and task selection — read them apart to make sense of it.

How GDPx differs from GDPval: from handing in homework to taking a desk

GDPx has a direct predecessor: GDPval, released by OpenAI on September 25, 2025. GDPval handed the model its prompt and documents and scored it non-interactively, outside any real environment. SemiAnalysis's criticism is that GDPval measured "knowing," not "doing." GDPx's fix is to put the agent inside a working environment with tools, files, and feedback loops, and to give it real time — a default limit of eight hours, shorter for some tasks.

That change matters most for vibe developers. Day-to-day agent work is already shifting from "answer a question" to "take a task": read the repo, run the tests, change the code, open the pull request. GDPx measures exactly that second shape, only with deal documents, chip tools, and lab instruments instead of repositories. Its low scores show precisely where that shape is weakest: context management, coherent tool use across a long session, and self-correction after a mid-task error — if any link breaks, the whole task scores zero.

GDPx's boundaries deserve equal attention. It does not test short tasks, conversational experience, creative work outside code, or real-company collaboration — no colleague interrupting, no requirements changing midway, no "stop that for now" from a manager. Treating a GDPx score as "an agent's competence rate in a real job" is the same kind of over-extrapolation in the other direction. It is a mirror held up to long-task delivery, not to all of work.

How much to trust it: a self-examination checklist for a day-one benchmark

Any day-one benchmark deserves the same set of questions, and GDPx is no exception. First, are the tasks publicly auditable? Most GDPx tasks were privately built by partners and cannot be re-checked item by item from outside; SemiAnalysis has published some task trajectories, which is the best external check available so far, but it is not independent replication. Second, is the scoring independent? Every number comes from SemiAnalysis's own runs, with no third-party rerun. Third, are sample sizes adequate? Some conclusions rest on a small number of runs, and vivid individual examples should be treated as illustrations, not statistics.

Fourth, is task quality itself sound? SemiAnalysis required every partner to pass its own automated quality checks, and claims the same process flags nearly half of Terminal-Bench 4.0's tasks as broken. That claim is itself in need of external verification — a quality standard vouching for itself is circular reasoning. Fifth, are the cost figures comparable? Token pricing, harness overhead, and retry strategies differ across models, so "cost per task" is a leaderboard convention, not the figure on your bill.

None of this is meant to dismiss GDPx. It is to place it correctly: one of the closest public benchmarks yet to real long-task work, but still a vendor-run benchmark. The right way to use it is as a starting point for selection and a reference for budgeting, not as a final verdict. SemiAnalysis plans to refresh the leaderboard every three to six months, and the numbers will move; today's conclusions should carry an expiry date.

Three things vibe developers can do with it

First, select models by task slice, not by overall ranking. Split your agent work into three buckets: short and well-defined (fix one error, write one test), medium-length (implement a small feature, refactor a module), and long and open-ended (build a service from scratch, migrate a system). GDPx is informative only for the third bucket; for the first two, your own evaluation set should do the talking.

Second, budget by cost per success, not cost per attempt. Divide the leaderboard's per-task cost by the success rate to get the expected cost of one successful outcome, then multiply by your task volume and rework rate. A model that is cheap per attempt but rarely succeeds can end up more expensive; a pricey model that gets it done first time can end up cheaper. Both columns belong in the budget sheet.

Third, keep a human gate on zero-tolerance domains such as finance and legal. GDPx's zero in those domains is not a gentle signal that "models are not good enough yet" but a clear one that "the current form of agent is not fit to deliver independently." Agent flows touching money, contracts, or compliance should position the agent as "drafter and first screener" and keep approval firmly with a human; any design for "fully automatic posting" or "fully automatic contracts" needs to be re-argued against this score.

Return finally to the number 12% itself. Its greatest value is not to dent confidence but to calibrate expectations: long-task delivery is still hard — hard enough that the best system gets it right first time only about one time in eight. For individual developers, that means the agent remains a powerful deputy, not an employee you can send home; for teams, it means evaluation, gates, and rework budgets are not optional extras but fixed costs of working with agents. Admitting that is precisely how you use them well.

Looking at software engineering alone, would the ranking change?

The overall leaderboard hides a fact worth stating: the headline score is an equal-weighted average across 11 domains, while any real team cares about only one or two of them. SemiAnalysis's domain data shows Carbon only narrowly ahead in software engineering, Opus 5.5 ahead in chip design, hardware design, and quantitative trading, and Astra ahead in recursive self-improvement and materials science. For a team whose work is software engineering, the gap between first and second place may be far smaller than the 1.1-point headline gap suggests, and on some task types it may reverse entirely.

More fundamentally, equal weighting is itself a value judgment. It assumes a chip-design task and a legal task matter equally — but for a software company the legal tasks should weigh zero, and for a law firm the chip tasks should weigh zero. SemiAnalysis has said explicitly that it does not weight by GDP share or occupational taxonomy, because it believes that would distort the "where frontier intelligence monetizes fastest" lens. That position is debatable, but readers should at least recognize that 12.0% is not a neutral "capability value" — it is a composite under a specific weighting, and under a different weighting the champion could change.

The most useful way to use GDPx for selection is as a "domain map" rather than an overall ranking. First pin down your own task distribution, then look up the scores and costs for the matching domains; if the public material has no breakdown for the domain you care about, the benchmark has limited value for your decision, and you should return to your own evaluation set. Ordering by the overall ranking is like ranking specialists by their all-subject total score — the ranking itself is not wrong, but it is being used in the wrong place.

Run your own micro-benchmark: 50 tasks are enough to surface problems

Rather than arguing over whether GDPx's scores are accurate, use its methodology to run a micro-benchmark for your own team. It is not complicated: pick 50 tasks from real work that have a clear delivery standard, can be finished within two hours, and have an existing way to check them — for example, fixing 10 real bugs, implementing 5 small features, migrating 3 legacy modules, writing 10 test suites, and doing 5 code reviews. Write the pass criteria in advance for each task: all tests pass, review comments actionable, behavior unchanged after migration — the more specific the criteria, the harder the scoring is to game.

On execution, hold to at least three disciplines. First, run each task once per model or harness and record both success rate and spend, rather than showcasing a single success. Second, keep graders blind to which model produced which output, so brand effects cannot sway judgment. Third, classify failures by cause: a key file never found, a broken tool-call chain, a mid-task change of direction, or a deliverable that was simply not good enough. GDPx's biggest lesson hides in exactly this kind of classification: long-task failures concentrate in a few causes, and fixing the two or three most common ones lifts success rates far more than switching to a pricier model.

Cost should likewise be recorded as cost per success. Per-attempt spend, success rate, and rework count, multiplied together, give a task's true cost. A model that is cheap per attempt but needs three runs for one success costs three times its sticker price. Posting that table inside the team guides procurement better than any public leaderboard: it measures your tasks, your repositories, and your standards — not someone else's simulated office.

Citing with version awareness: leaderboards refresh, conclusions need expiry dates

One citation habit closes the loop. SemiAnalysis plans to refresh GDPx every three to six months, and model versions, harness versions, and task sets will all change; today's 12.0% could become 14% in the next version, or 10% if tasks get harder. Any article, selection document, or procurement report citing GDPx should therefore state three things: the leaderboard version, the date consulted, and the qualifier "under the version 1.0 methodology." A citation that says only "some model scored 12% on GDPx" with no version or date becomes an unverifiable orphan number within three months. Half of a benchmark's value is the number itself; the other half is the number's verifiability, and writing the version into the citation is the minimum responsibility to readers. The same applies to your own micro-benchmark: task list, pass criteria, model versions, and run dates — miss any one of them and the next person rerunning it will not reproduce your results.

One final caution: do not use a GDPx score to label a model once and for all. Today's leader may lead only because its harness is more mature or its task mix is more favorable, not because it is smarter in every respect; today's last-place model may perform best in precisely the domain your work lives in and the leaderboard omits. Benchmarks narrow the field; they do not end the discussion. What ends it is always your own tasks and your own evaluation.