AI Benchmarks Are Aiming at Air: Mercor's Map Shows 90% of US Work Hours Lack Benchmark Coverage

We check the leaderboards every day: who gained two points on SWE-bench, which new model beat GPT at math reasoning, whether Claude is still number one in the latest agent eval. Those scores shape our entire intuition about "which model is stronger." A new study from Mercor just threw cold water on that intuition — the leaderboard scores we obsess over may only measure about 10% of real work.

Mercor's own words are worth reading verbatim: "Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do."

In other words, the AI evaluation apparatus is aiming at air. The target stands at the 10% mark, while 90% of the workload isn't being measured at all.

What Mercor actually did

Mercor — the AI talent marketplace that also builds the APEX benchmarks — created a "Benchmark Map of the US economy": it took every test prompt from a small set of high-quality AI benchmarks, mapped each one onto concrete job tasks in the US Department of Labor's O*NET occupational taxonomy, and asked how much of what Americans actually do is covered by benchmarks at all.

The methodology is solid. O*NET divides the US workforce into 923 occupations, each broken down into a list of "tasks" that make up the job, totaling 18,797 tasks. As the study puts it: "O*NET divides the US workforce into 923 occupations; for each occupation, O*NET attempts to provide a comprehensive list of "tasks" which make up the occupation, totaling 18,797 tasks across all occupations."

Mercor started with an initial list of 29 benchmarks, selected for quality and economic relevance. Inclusion required clearing five screens: public availability (at least 25 public tasks), agentic tasks rather than Q&A, economic realism (tasks people are frequently asked to do in real enterprises, prioritized over academically complex ones), economic diversity (capped per job family, deliberately countering the massive overrepresentation of software engineering in popular benchmarks), and balanced difficulty (not yet saturated by frontier models). Then, following work by Wang et al., they used an LLM judge to tag each benchmark prompt with as many O*NET tasks as it plausibly matched, validating the tagging against human expert labels on Mercor's own APEX benchmarks. An O*NET task counted as "covered" only if at least 5 benchmark prompts mapped to it.

That "at least 5 prompts" bar is already generous — and the results are still brutal: of 923 occupations, only 208 have even one covered task, and 6 of the 22 job families contain no covered occupation at all. Verbatim: "only 208 of 923 occupations have a task with at least 5 benchmark prompts, and 6 of 22 job families have no such occupations."

How the map was drawn

The numbers alone undersell the work, so it's worth unpacking the methodology — it determines how much to trust that 90%.

The centerpiece is an interactive map: 923 tiles, one per O*NET occupation, grouped into 22 job families, shaded by the share of each occupation's tasks represented in the initial benchmark list. Click a family to open it. At a glance, software engineering and financial analysis glow while truck driving and plant operation sit almost entirely dark — "deep and narrow" coverage, visible in one image.

The coverage pipeline has four stages. First, benchmark selection: the 29 benchmarks passed the five screens above. Second, mapping: the LLM judge tagged every prompt with candidate O*NET tasks, checked against human expert tags. Third, the threshold: ≥5 prompts per task to count as covered. Fourth, the hours conversion: coverage estimates were combined with LLM-derived estimates of each O*NET task's share of working time (from Hatgis-Kessell et al.) to compute the share of total US working hours spent on covered tasks. That's where the 90% comes from.

The bias analysis uses task-level logistic regression — coverage probability regressed on AI-exposure and log occupational wage bill, with pointwise 95% confidence intervals clustered at the occupation-group level. Wage bills come from BLS estimates of mean wages times headcount, assuming a 40-hour week. This is serious statistics, not descriptive hand-waving.

So the 90% needs to be read precisely: it's not "90% of occupations are untested" but hours-weighted — 90% of Americans' working time goes to tasks with little or no benchmark coverage. The occupation-level figure (208/923) and the hours-level figure (90%) are different things; the latter stings more, because it directly answers "how far are leaderboard scores from the real economy."

Coverage isn't random: benchmarks favor high-pay, AI-exposed jobs

The second finding stings harder: the skew isn't random, it's systematic. Even after controlling for AI-exposure (how automatable a task is), tasks in occupations with larger wage bills are significantly more likely to be covered; and within an occupation, tasks rated "most important" by O*NET domain experts are about 6–10x more likely to be represented than those rated least important.

AI-exposure itself is a strong predictor: tasks tagged "exposed" are roughly 6x more likely to appear on benchmarks than "unexposed" ones. That sounds reasonable — benchmarks exist to measure AI progress, and progress is most plausible where exposure is highest. It explains why benchmarks overflow with software engineering and financial analysis and have almost nothing on truck driving or gas plant operation.

But Mercor offers one devastating example: "For example, benchmarks in our initial list heavily cover AI-exposed lawyers' tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks."

Lawyers and legal secretaries — similar AI-exposure, very different pay. Benchmarks picked the former and dropped the latter. Mercor's verdict: the long tail of smaller, middle-income knowledge work is systematically undercovered. Which happens to be the shape of work most ordinary people — and most vibe developers — deal with daily.

To be fair, Mercor concedes the allocation has logic: high-quality, economically relevant benchmarks are scarce as a public good, so prioritizing the most important, most AI-exposed tasks in the largest wage-bill occupations is the natural choice. Their warning: unless the number and diversity of such benchmarks grows substantially, these patterns will leave enormous blind spots in our picture of economically valuable model capabilities.

What this means for vibe developers: stop picking models by leaderboard

Three concrete takeaways for people who build with AI every day.

First, leaderboard scores and your tasks may not even be on the same channel. The SWE-bench, HumanEval, and agent leaderboards you consult measure tasks inside "software engineering" — an overrepresented job family. What you actually ask models to do — scraping for your niche business, generating docs to your company's bizarre spec, wrangling an internal DSL only three people understand — almost certainly falls inside the unmeasured 90%. A 5-point leaderboard lead tells you nothing about a 5-point lead on your tasks. It doesn't even tell you your tasks were tested.

Second, the only ruler you can trust is your own eval. Mercor's study is, accidentally, the strongest endorsement ever written for building your own evaluation set. If public benchmarks don't cover your long-tail tasks, stop letting leaderboards choose your model. Take 20–50 real tasks, run the candidates, judge actual output quality — that crude method is closer to "who's stronger on my tasks" than any leaderboard. Vibe developers have the edge here: your tasks are concrete and real, which makes evals far easier to write than academia's. Keep it lightweight: (1) pick 20 representative tasks from what you actually asked AI to do in the last two weeks, with original inputs intact; (2) write one line per task on what "good" means (it runs, the format is right, no key information dropped); (3) run 2–3 candidate models on the same set and score blind; (4) keep the set and re-run it every time you switch models or prompts. That's your own benchmark map — with one tile lit up.

Third, the blind spot is the opportunity. Benchmarks skew toward high-pay, AI-exposed work, which means agent applications for "long-tail middle-income knowledge work" sit in a measurement vacuum — your competitors can't see in either. Legal secretaries, clinic receptionists, production schedulers at small factories: AI-exposed, unbenchmarked, and exactly where vibe products can grow quietly. Leaderboards only ever show you the red ocean; the blank spaces on the map are the blue one.

Three readers, three readings

In its "Can O*NET + benchmarks tell us how good AI is at your job?" section, Mercor names three audiences, and the study lands differently for each.

Enterprise buyers should read it as a warning: choosing models by leaderboard means using "average scores on software engineering and finance tasks" to predict "model performance on your company's actual roles." If your business sits inside that 90% — it probably does — there is almost no transmission belt between leaderboard rank and your lived experience. Move some of the leaderboard-chasing budget into building your own task set.

Policymakers should read it as methodology: forecasting AI's economic impact can't rest on "models gained N points on benchmarks" — it needs "those capability gains mapped onto real occupations' tasks." O*NET weighting is step one of translating "model capability scores" into "occupation capability scores," even if the data is still too thin.

Ordinary workers can read it optimistically: your job is probably outside the benchmarks' spotlight, which means nobody currently has a standard answer to "how good is AI at my role" — including the people loudly predicting AI will replace it. No measurement, no replacement timeline. And for vibe developers, that unpriced 90% is the opportunity: everyone else is fighting over the measured 10% while you build where nobody is looking.

A letter to benchmark builders

The most constructive part of the study is that it doesn't stop at exposing the gap — it points a direction. Mercor's read: the "deep and narrow" pattern reflects the scarcity of high-quality, economically relevant benchmarks as a public good. With limited resources, covering the most important, most AI-exposed tasks in the largest wage-bill occupations first is reasonable. But without a large increase in the number and diversity of such benchmarks, the blind spots in our view of economically valuable capabilities will only grow.

There's a counterintuitive detail worth noting: to maximize breadth, they deliberately capped benchmarks per job family — which systematically understates coverage in the most-benchmarked families like software engineering. So for programmers, reality is probably somewhat better than "90% of hours uncovered" — somewhat, not much, since software engineering is only a slice of hours-weighted work.

One more layer for Chinese-speaking vibe developers: O*NET is a US taxonomy and the 29 benchmarks are overwhelmingly English-context. Draw the same map for Chinese workplaces — Chinese customer support, Chinese official-document writing, Chinese e-commerce operations — and coverage would only look worse. If English benchmarks leave 90% blank, the Chinese-context figure plausibly starts at 95%. Which means Chinese vibe products can't copy homework from English leaderboards at all: building your own Chinese-task eval isn't a nice-to-have, it's mandatory.

Limitations, the date caveat, and one reminder

Mercor states its own limits plainly: "We want to stress, therefore, that these results are provisional. We will update them as we expand the initial benchmark list."

Three layers of limitation deserve honesty. First, 29 benchmarks are an initial list, not the universe — any incomplete list systematically understates true coverage, and the bias is hard to bound. Second, capping benchmarks per job family makes economy-wide estimates more accurate at the cost of understating coverage in benchmark-dense families (e.g., software engineering). Third, excluding saturated benchmarks may understate coverage of tasks LLMs already handle easily — though the authors argue including them would bias upward even more, since benchmark saturation often reflects lower task realism.

On dating, a transparency note: the Mercor research page itself shows no visible publication timestamp. The October 15, 2026 date used here is triangulated from three independent sources — the aggregator runtimewire labels its repost "Published Oct 15th, 2026, 12:44am CT," a Hacker News discussion thread appeared for the study, and Mercor's official LinkedIn account posted around October 16 saying "Today, we're releasing the Benchmark Map." All three point to the same window, but we found no self-declared date on Mercor's own page, so the date evidence for this story is rated "medium," as disclosed.

One final judgment: the most valuable thing about this study isn't the 90% figure itself (a provisional estimate, likely to be revised) but that it turned "benchmark coverage" into a visible map — and honestly marked the blank areas. A map's value was never in what it measured; it's in showing you what hasn't been measured yet. Next time you see "model X tops leaderboard Y," ask one question: was it ever tested on tasks like mine? This map says nine times out of ten, the answer is no. And for builders, the places with no answers are usually where the opportunities are.