Back to Explore
NewsVibeFix editorialUpdated Nov 20, 2026

One Bug Is Easy, Fifty Breaks Everything: BuggyMLE Pulls the Cover Off Coding Agents

On 18 November 2026, Google Research and co-authors posted BuggyMLE to arXiv: bugs injected into 222 working ML repositories, and 14 models slide from 95% on the easiest tier to 9% on the hardest. A 9-billion-parameter model trained easy-to-hard beats Claude Opus 5.5 on the 10-bug tier.

A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right an ML dashboard with loss curves, a confusion matrix and a pipeline diagram

In one sentence: the benchmarks are nearly saturated, so someone built a harder one

On 18 November 2026, thirteen authors from Google Research, Google DeepMind, the University of Illinois and Georgia Tech posted a paper to arXiv titled "BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection". The problem it tackles is straightforward: the benchmarks used to test coding agents are running out of room. Some only ask a model to edit a single file, while real machine learning engineering means changing a whole repository. Some are already saturated, with frontier models outscoring human reference solutions. And some rely on hand-written tasks, so question-writing cannot keep up with model progress. This paper does the opposite: it starts with 222 working machine learning repositories, has an agent inject controlled bugs into them — "poisoning" the code — and then asks other models to clean the poison out. Add more bugs, and the task gets harder, with no ceiling in sight.

The results hit hard. Fourteen models took the test. On the easiest tier, the best model fixed 95% of tasks; on the hardest, only 9%. More interesting is what happened in between: a model with just 9 billion parameters, trained on an easy-to-hard curriculum, beat Claude Opus 5.5 on the 10-bug tier — and what it learned transferred elsewhere, lifting its SWE-bench Lite score from 24% to 33%. This article covers five things: how BuggyMLE builds its tasks, why a difficulty formula called CCI can predict which tasks are hard, what the fourteen-model scoreboard actually looks like, how the small model caught up with the flagships, and what any of this means for people building apps.

How the tasks are built: poisoning working repositories

BuggyMLE works in three steps. First, find clean foundations: 222 real machine learning repositories that run and produce results, spanning tabular, text, time-series, image, signal, geospatial, clinical and genomics data. Second, inject bugs: an agent edits the code following a set of rules, producing more than 6,600 bug-injected repository variants containing nearly 260,000 lines of injected code. The injected bugs are not random damage — each one is recorded and verifiable, because reverting the injected changes exactly restores the repository, a property the paper calls oracle verification. Third, evaluate: another model receives the buggy repository and must repair it so it runs, the metric recovers, and every injected bug passes individual verification.

The test set is called BuggyMLE-Bench-Lite: 22 tasks across 4 difficulty tiers, 88 tasks in total. Tiers are defined by the number of injected bugs: 1, 5, 10 and 50. The 50-bug tier is brutal: repositories expand to roughly twice their original size, with 23 to 60 injected bugs per task. The grading bar is stricter than "does it run": full repair requires three conditions at once — the target metric recovers to at least 99%, the repository's entry point still executes the mutated modules so an agent cannot route around the broken parts, and an LLM judge inspects each bug site against its ground-truth diff and mechanism, confirming a genuine fix rather than a workaround that masks the symptom. Restoring the metric while leaving bugs inside does not count.

How the injected bugs are distributed across a repository is also worth noting. The paper's analysis of bug impact shows mild and strong impacts dominating the 1-, 5- and 10-bug tiers, catastrophic failures rare, and execution crashes occurring only at the 50-bug tier. Bugs land across every stage of the ML pipeline — data, preprocessing, model, training and output. So even the easy tier is not typo-fixing: the bugs genuinely degrade the metric. Between tiers, what changes is not just the count but how badly the repository is damaged.

The per-bug verification design is the paper's most important choice. Most benchmarks only check the final metric, so a model can restore the number by taking shortcuts. BuggyMLE requires every bug site to pass, separating "looks fixed" from "actually fixed". The scoreboard below keeps showing the same pattern: models with high metric recovery and low full repair — and the gap is exactly this step.

CCI: ten structural features that predict how hard a task is

Before building tasks, the authors answered a more basic question: what makes a coding task hard? Their answer is the Code Complexity Index (CCI), a formula built from ten structural features in three groups: repository scale (lines of code, file count, maximum directory depth), dependency depth (dependency count, longest dependency chain), and patch size (files changed, hunks, lines added, lines removed, total lines changed). Four parameters combine the ten features into one score.

What makes CCI remarkable is that it uses no model information at all, yet predicts how models will do. Applied to eight public coding benchmarks, CCI tracks their average solve rates with an R² of 0.951 — nearly a straight line. The strongest predictors are patch size and overall complexity. In other words, how hard a task is depends less on what knowledge it tests than on how many places it touches and how deeply those places are connected. That conclusion has a direct implication for everyday work, covered later.

The fit was checked with leave-one-benchmark-out validation: drop one benchmark, fit on the other seven, predict the dropped one — and the formula still holds. So it is not curve-fitted to one particular test set; it captures something real about difficulty itself. The authors still hedge: CCI has only four parameters and ten structural features, and whether it keeps working as models and task types change will need more independently generated synthetic test sets to confirm. The limitations section returns to this point.

The scoreboard: a slide from 95% to 9%

Fourteen models took part, in three groups. The frontier group: Claude Opus 5.5, Claude Sonnet 5.5, Claude Sonnet 5, Claude Sonnet 4.6 and Gemini 3.8 Flash. The budget group: each provider's cheapest tier or comparably sized open-weight models — Claude Haiku 5.5, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, Qwen3.8-27B without thinking, and untuned Qwen3.5-9B. The third group: the authors' own 9B model trained on the BuggyMLE curriculum, plus three ablations. Every model ran in the same sandbox with the same task prompt.

On the easiest 1-bug tier, Opus 5.5 fixed 95.5%, Sonnet 5.5 fixed 90.9%, Haiku 5.5 fixed 77.3%, Sonnet 5 fixed 63.6%, Qwen3.8-27B fixed 40.9%, Gemini 3.8 Flash and Sonnet 4.6 both fixed 27.3%, untuned Qwen3.5-9B fixed 13.6%, Haiku 4.5 fixed 9.1% and Gemini 3.5 Flash-Lite fixed 4.5%. With one bug, flagships and budget models are already far apart, but the best models mostly cope.

Then the slide begins. At 5 bugs, Opus 5.5 falls to 31.8% and Sonnet 5.5 to 27.3%; every other model is below 20%. At 10 bugs comes the most interesting upset in the whole paper: Sonnet 5.5 leads at 31.8%, the authors' trained 9B model is second at 18.2%, Opus 5.5 manages only 13.6% — tied with Gemini 3.8 Flash — and every other model scores 0%. At 50 bugs, end-to-end task resolution nearly vanishes: only Opus 5.5 at 9.1% and Gemini 3.8 Flash at 4.5% complete anything in full; the other twelve models score 0%. Thirteen of the fourteen models decline monotonically with difficulty, and the top full-repair rate falls from 95% to 9%.

Fix rates alone tell half the story; the paper also reports recovery, the fraction of the metric degradation recovered. Opus 5.5 recovers 92.2% of the metric on the 10-bug tier but fully repairs only 13.6% of tasks. That is the phenomenon per-bug verification is designed to catch: the model restored the target number while leaving injected bugs unresolved. Haiku 5.5 recovers 71.1% on the same tier and fully repairs 0%. For people building apps, that gap matters more than the headline score: when an agent says "fixed", the question is whether it fixed the metric or the bugs.

The gap between the budget and frontier groups deserves its own look. Sonnet 4.6 manages 27.3% on the 1-bug tier and drops to zero at 5 bugs; Gemini 3.5 Flash-Lite starts at 4.5% and scores zero on the other three tiers; Haiku 4.5 only reaches 9.1% on the first tier. These numbers say budget models are not "a bit worse" at multi-file repository debugging — they are mostly unusable for it. On the other hand, Haiku 5.5 reaches 77.3% on the 1-bug tier, so there are budget models that can fight — but its ceiling is low, falling to 18.2% at 5 bugs and zero at 10. When choosing a model, "cheap" by itself says nothing; what matters is the tier at which it starts to break.

One more detail from the 50-bug tier: the authors' trained 9B agent resolved the second-highest share of individual bugs, behind only Sonnet 5.5, yet its full-repair rate was still 0%. The paper's explanation is that every 50-bug episode terminated at the turn or context limit, often before producing a valid submission. At this difficulty, the bottleneck is not only "can it fix it" but "can it finish fixing it inside the turn and context budget". That is a design lesson for agent products: context window and turn budget are part of capability. Bigger is not automatically better — but too small a window or too few turns guarantees failure.

Full-repair rates across four difficulty tiers: at 1 bug, Opus 5.5 at 95.5%, Sonnet 5.5 at 90.9%, Haiku 5.5 at 77.3% and the trained 9B model at 72.7%; at 50 bugs, only Opus 5.5 at 9.1% and Gemini 3.8 Flash at 4.5% remain, everything else at 0%

The most interesting upset: a 9-billion-parameter model catches the flagships by going to school

The paper's second contribution is worth reading even more closely than the benchmark. The authors took Qwen3.5-9B and built an easy-to-hard training curriculum from BuggyMLE: three difficulty bands across 200 additional ML repositories — easy at 1–5 hunks, medium at 10–12, hard at 30–34 — using 2,724 evaluator-verified repair trajectories from five frontier teacher models, including Gemini 3.6 Flash and Gemini 3.7 Flash, for supervised fine-tuning, followed by reinforcement learning (GRPO) stage by stage up the difficulty ladder.

The result: the 9B model fixes 72.7% on the 1-bug tier, far above the supervised-fine-tuning-only version at 50.0%; on the 10-bug tier it fixes 18.2%, beating Opus 5.5's 13.6% and trailing only Sonnet 5.5's 31.8%. Two ablations make the point cleanly. Training on easy tasks only scores 0% on both the 5- and 10-bug tiers — however well you drill easy tasks, the skill does not transfer upward. Training on hard tasks only drops to 22.7% on the 1-bug tier, worse than the 50.0% of the fine-tuning-only baseline, because the task distribution shifted and the small model could not cover both ends. Only the full easy-to-hard curriculum holds both ends.

What matters more is transfer. The authors tested the small model on general coding benchmarks that share nothing with its training data: SWE-bench Lite resolve rate rose from 24% to 33%, and LiveCodeBench v6 pass@1 rose from 69% to 71%. Skill learned fixing ML repositories carried over to fixing ordinary software repositories. The implication is that model size is not the only variable — the difficulty structure of the training matters just as much. A 9-billion-parameter model, taught with the right curriculum, can rival flagships on specific tasks; conversely, piling on parameters without thinking about what to practise may spend money in the wrong place.

What this means for people building apps

First, smaller changes are more likely to succeed on the first try. CCI says patch size and complexity predict difficulty best, and the everyday corollary is direct: split a large change into several small ones, not only as good engineering hygiene but to make the agent more likely to succeed. Asking an agent to change fifty places at once pushes the task into the tier where it is weakest; changing one or two places, running the tests, then moving on raises the odds substantially.

Second, "the metric recovered" is not "it is fixed". BuggyMLE's scoreboard repeats this gap: Opus 5.5 recovered 92.2% of the metric on the 10-bug tier but fully repaired only 13.6%. When you use an agent to fix a bug, do not stop at "tests pass" — ask it to explain each suspect site, how it located it and how it fixed it, or have a different model or session re-check the work. The paper's per-bug verification, translated into daily work, is one sentence: after the fix, walk through every bug site one by one.

Third, the training lesson transfers. Easy-only does not scale up, hard-only loses the easy end, and the full easy-to-hard run works best — and that holds for onboarding people and for code review with agents as much as for training models. Let an agent handle single-file fixes first, then gradually move to cross-file work, rather than throwing a fifty-bug repository at it on day one. One more detail from the paper: the biggest efficiency gains in reinforcement learning came during the hard stage, where the agent learned to finish repairs in far fewer turns without losing reward. Difficulty is not the goal; difficulty calibrated to teach a new skill is.

Fourth, grade evaluations by tier instead of reporting one total. BuggyMLE scores the same tasks across four difficulty tiers and reports each tier separately, so the slide is visible at a glance. Teams evaluating agents internally can copy that: prepare three tiers of tasks — single-file fixes, cross-file debugging, and repositories with several interacting bugs — and score each separately. A single "80% pass rate" mixes all three tiers together and hides where the model starts to break. With tiers, choosing a model, writing prompts and deciding whether a job can be handed to an agent all get a factual basis.

How this relates to what we have already published

Agent evaluation is not a new topic on this site, so the boundary needs to be drawn. The 15 November SWE-CoWork article is about multi-agent collaboration: tests passing 60% of the time while tasks finished under 10%, puncturing the fantasy that coding agents organise like human teams — its anchor is collaboration structure. This article's anchor is different: BuggyMLE is about difficulty itself — whether difficulty can be measured, slid up and down, and how the difficulty structure of a training curriculum shapes capability. Both pour cold water on "agents are very capable", but from different directions: SWE-CoWork pours it on "they can't cooperate", BuggyMLE on "they collapse when it gets hard".

The 11 November AWS UAEF article and the 19 October Microsoft Scope article are about evaluation frameworks and tools — products for scoring agents. This article is about a specific research paper — the research behind why scores lose meaning without a difficulty scale. Reading this one first makes the other two clearer: an evaluation without an adjustable difficulty dial produces scores that saturate quickly and stop meaning anything.

The limitations, stated up front

Several caveats apply to this paper's conclusions, and the authors state them in their own limitations section. First, CCI only sees structural features, not semantic complexity: two repositories can look identical on structure while demanding completely different domain knowledge, and CCI cannot tell them apart. Treating it as a general law needs broader coverage to confirm, and the authors note that BuggyMLE's own tasks were excluded from CCI validation to avoid circular reasoning. Second, injected bugs are synthetic: however controlled, their distribution differs from bugs people actually write in the real world. A model good at fixing synthetic bugs is not automatically good at fixing the three-year-old inherited bug in your repository. Third, the per-bug grading uses a language model as judge, and the judge itself can be wrong; and all results are the authors' own evaluation — the team comes from Google Research and DeepMind, evaluating models that include their own Gemini as well as rivals Claude and Qwen, with no independent reproduction.

One more number deserves to stand alone: even the clean repositories, with no bugs injected, sit at only around the 49th percentile on the Kaggle leaderboard. These machine learning tasks are hard to begin with; bugs make hard harder. BuggyMLE does not solve the fact that machine learning engineering is intrinsically difficult. What it does is measure the difficulty and grade it, so the next person knows which tier they are on and how far the next tier is. For people building apps, that ruler is more useful than any single score: the next time an agent fails on a repository, the first question should be — by CCI, which tier is this job actually in?

Views 0Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project

Related articles

A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts
News
Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

Developer WorkflowProduct LaunchTesting & Quality
A developer with glasses types on a laptop at a wooden desk, an external monitor to the left showing C code on a dark background, a small plant on the desk
News
Microsoft's Coding Model Stops Playing Second Fiddle: MAI Code 2 Scores 96% of Opus 5.5 for 5% of the Cost — Read the Three Caveats First

On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.

Developer WorkflowProduct LaunchModel Updates
Against a black background, a conductor seen from behind raises a baton, with swirling trails of light circling beneath him like an orchestra being conducted
News
One API for 45 Models, and It Switches Providers When One Fails: Perplexity's Orchestrator Wants to Be the Switchboard

On 17 November 2026, Perplexity launched the Orchestrator API: one interface reaching OpenAI, Anthropic, Google, xAI and open-weight models such as GLM, Kimi and DeepSeek, billed at the answering model's published rate with no platform fee. When one provider fails, the same model retries on another deployment, and preview routing assigns requests to models by task.

Developer WorkflowProduct LaunchModel Updates