Back to Explore
NewsVibeFix editorialUpdated Nov 19, 2026

Same Model, Different Harness, a 24-Point Gap: Garry Tan's Evals Puncture the Agent-Picking Myth

On 8 November 2026, Y Combinator CEO Garry Tan published open harness evals: same tasks, same model, only the agent harness swapped. Pass rates sat between 71% and 79% and could not be separated — but cost per solved task ranged from $0.19 to $0.44. A second board of 28,849 trials showed a 24.1-point solve-rate gap. Keep the model, change the shell, and the bill changes more than the brain.

Close-up of colourful Python code lines on a dark screen, with syntax highlighting clearly visible

In short: stop asking which model is strongest — ask what it is wearing

On 8 November 2026, Garry Tan, the CEO of Y Combinator, posted results from an evaluation he ran himself, under a blunt title: harness-evals — which agent harness does the best work with a frontier model, for the least cost? He did not change the model: the same tasks, Claude Opus 5.5 or GPT-6.1 Sol, with only the "shell" around the task swapped — the system prompt, the tools, the memory, the verification. When the results came back, Tan poured cold water on them first: all seven setups passed between 71% and 79% of the tasks, and 14 of the 20 tasks scored identically everywhere. "This is a test problem, not a tie," he said. What actually separated the setups was the bill: for the same solved task, one setup cost $0.19 and another $0.44 — more than double.

This is worth writing about not because there is one more leaderboard, but because it hits the most expensive misunderstanding in vibe coding circles. For the past year, almost every discussion about picking a tool has been a debate about models: Opus or Sol, whose benchmark score is higher. Yet most of the money and time an agent burns in a day is not spent inside the model call — it is spent in the shell around the model: how much of the system prompt gets re-sent on every request, how many tools it loads and never uses, whether it spins in place after a failure, whether it remembers how this repository was fixed last time. Tan's evaluation is the first to weigh that shell on its own, with open methods and data anyone can re-run. In almost the same week, a second independent board went live: the Harness Leaderboard on Hugging Face, released on 5 November, ran 28,849 trials, changed only the harness and never the model, and found a 24.1-point gap between the top and bottom solve rates. Put the two data sets together and the conclusion is firmer than either one alone: keep the model the same, change the shell, and the results really do change — but which part changes, by how much, and how far you can trust it, has to be checked line by line.

First, Tan's numbers: pass rates cannot separate the setups, but the bill can

Tan's repository is garrytan/g-harness-evals, created on 5 November 2026 under the MIT licence; when checked on 19 November it already had 166 stars. The evaluation uses two task sets: 52 public Terminal-Bench 2.1 coding tasks, and 20 multi-user simulation tasks, in which the agent plays the chat bot of a simulated company, answering several people in turn. Every setup ran each task multiple times, costs were counted at public list prices, and times were reported as medians. Start with the multi-user tasks, which drew the most attention. On Opus 5.5: QM (a loop built on Pi) cost $0.19 per solved task, Claude Code $0.22, Hermes Agent $0.24, and OpenClaw $0.44. And the pass rates? QM 72.5%, Claude Code 76.3%, Hermes Agent 78.8%, OpenClaw 76.3% — four setups squeezed into six points, with statistical tests unable to separate any pair. Tan's own words: no pair could be told apart on pass rate, and a gap smaller than 7.7 to 14.2 points could be hiding in the noise. Time spread just as widely: QM and Claude Code took a median 40.1 seconds per try, Hermes Agent 64.6 seconds, OpenClaw 99.4 seconds.

On GPT-6.1 Sol the order changed: Codex cost $0.17 per solved task and took 30.1 seconds — cheapest and fastest; OpenClaw $0.22 and Hermes Agent $0.19, with pass rates from 71.3% to 75.0%, again indistinguishable. In other words, on the same tasks with the same model, swapping only the shell can double the cost per solved task and more than double the time, while the most obvious metric — "did it solve the task or not" — barely moves. That is a practical warning for anyone paying per token: picking the wrong shell will not make you noticeably dumber, but it can double your monthly bill and double your waiting time.

The 52 coding tasks tell a subtler story. On Opus 5.5, Pi cost $0.62 per solved task, Claude Code $0.63, OpenClaw and Hermes Agent $0.88 each; Claude Code had the highest pass rate at 88.5%, Pi 81.4%, Hermes Agent 81.7%, OpenClaw 76.0% — OpenClaw was the only one judged "behind" Claude Code in the preregistered test, but Tan notes in a footnote that the call is fragile: under a different outage rule, or with provider-refused tries removed, it becomes "can't tell". On Sol, Pi was cheapest at $0.23, Codex $0.34 ($0.31 after correction), Hermes Agent $0.43 ($0.37 after correction), OpenClaw $0.61 ($0.50 after correction). Hermes Agent had the highest pass rate, 85.6%, Codex 83.7%, Pi 80.4%, OpenClaw 76.0%. One detail matters: the two task sets cannot be compared side by side — different tasks, slightly different cost accounting — and Tan marks that in bold in the README. Some online summaries already mix the two tables' numbers, which is wrong.

Comparison chart: in Tan's multi-user tasks on the same Opus 5.5 model, cost per solved task ranged from $0.19 to $0.44; on the Harness Leaderboard's 28,849 trials, solve rates ranged from 51.4% for mini-swe-agent to 27.3% for OpenClaw

Where OpenClaw's money goes: about 80% of it is re-sending the same text

A bill that doubles needs an explanation, and Tan's repository has a separate cost breakdown with a specific answer: on Opus 5.5, about 81% of OpenClaw's extra cost over Claude Code was cache writes — it re-sends a much bigger system prompt and much bigger tool results on every request. In other words, it is not thinking longer; it is re-reading the instruction manual out loud on every turn. On Sol the gap took a different shape: OpenClaw averaged 282,492 output tokens per task against Codex's 32,226 — nearly nine times as many, and output tokens are the most expensive part of metered pricing. Hermes Agent's extra cost on Sol is also spelled out: 423 kB more tool descriptions per try and 16.7 requests per try against Codex's 11.6. What is interesting about these numbers is that they are all engineering problems, not model-capability problems: how long the prompt is, how wordy the tool descriptions are, whether results are re-sent in full every time, how often failures are retried. All of it is decided by the shell's design — and all of it is something a person building with agents can change themselves.

The cheapness of the lightweight contenders, Pi and QM, also has a name. Pi was the cheapest on the coding tasks on both models, $0.62 and $0.23; QM shared first place with Claude Code on the multi-user tasks at $0.19. What they share is a thin shell: fewer rituals, shorter prompts, fewer round trips. But cheapness has a price, and Tan's data states it plainly: Pi's pass rate on the coding tasks was about seven points below Claude Code (81.4% against 88.5%), and QM's multi-user comparisons were never put through a statistical test — plus it ran on Docker's default network, so its network conditions were not held fixed against the others, and its ranking should be read with a discount. Picking a shell is never "lighter is better" or "heavier is better"; it is a question of what your tasks are paying for: pass rate, or speed and cost.

The second data set: 28,849 trials, and the winner is a bare bash loop

Tan's evaluation is small — 20 multi-user tasks and 52 coding tasks. The Harness Leaderboard, launched almost the same day, scaled the experiment up by more than a hundred times: Hugging Face researcher Joël Niklaus released the board on 5 November, running 28,849 trials at $4,746 of inference cost, comparing nine harnesses — Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw. The models were fixed to three open models (GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint), the tasks were Terminal-Bench 2.1 and DeepSWE, and every task got three attempts. The result: the highest average solve rate belonged to mini-swe-agent at 51.4% — a minimal agent that is little more than a loop plus a bash tool; the lowest was OpenClaw at 27.3%. Pi solved 49.4% at just $0.29 per solved task; OpenCode cost $3.35 per solved task, 11.7 times Pi's cost. The board also notes that five harnesses were statistically tied for first, including Kimi Code, Hermes Agent, Pi and Claude Code.

The most thought-provoking part of this board is not who came first, but what the winner is: mini-swe-agent has almost no fancy design, just a loop and bash. It wins not on features, but on waste avoided. Another detail matters just as much: the best harness changes with the model — mini-swe-agent was best on GLM-5.3-Flash, Hermes Agent on DeepSeek-V4.1-Flash, Kimi Code on Flint. In other words, there is no universal answer to "which harness is best"; it depends on the model you run. The independent checking site honesty.tools verified the headline numbers against the board's live API on 6 November: 28,849 trials, about $4,746, mini-swe-agent 51.4%, OpenClaw 27.3%, Pi 49.4% at $0.29 — all matching the live data. But it added a caveat: this was a numbers check, not a re-run — the claim that "model, provider and reasoning effort were held fixed" is the board's own design statement, not something audited trial by trial.

The board's limits are stated in the open, too: it used only three open Flash-tier models, no Claude, no GPT; OpenClaw's last place is partly because its 8,192-token limit per response cut off answers on Flint and DeepSeek, and a re-run with a newer version is being prepared. A follow-up in The New Stack on 13 November added another telling detail: DeepSeek's own Harness tied for first across the board's five scored evaluations, yet ranked only sixth on Terminal-Bench alone, because of one built-in behaviour — it shuts down background processes when a task finishes, and some tasks need those processes to stay alive. A single small default can move a ranking by several places. That is the truth about the shell: what decides success or failure is often not the big boxes on the architecture diagram, but these unremarkable default settings.

That 18.4-point gap — don't forward it yet

The most widely shared number from this whole debate is the 18.4-point gap on the public Terminal-Bench 2.0 board: the same Claude Opus 4.6 scored 76.4% in Stanford IRIS Lab's Meta-Harness and 58.0% in Claude Code. Many people treat it as proof that "the harness matters more than the model", but it is exactly the number that most needs a second look. Developer Quang Lee analysed the public board and found that, keeping only verified rows, the Opus 4.6 gap shrinks from 18.4 points to 4.9 — public submissions come from different teams with different budgets, prompts and run configurations, so comparing them mixed together was never a fair fight. Terminal-Bench 2.1 later fixed 28 of its 89 tasks, and Claude Code with Opus 4.6 rose to 70.1% on 2.1. In other words, a large part of the 18.4 points came from broken tasks and mismatched configurations, not from the harness itself.

An earlier comparison points the same way: ByteDance Seed's HarnessDev study found GPT-5 scoring 35.2% on Terminal-Bench 2.1 in one harness and 49.6% in another. And a study published on arXiv in July 2026 (arXiv:2609.00006) examined the source code of eleven production coding agents and broke the harness into seven common subsystems: the agent loop, LLM integration, tools and actions, memory and context, safety and permissions, orchestration, and extensibility. The harness is not one thing; it is a whole set of things. Taken together, the reasonable reading is this: the harness can move scores by a few points or by more than twenty, but how much depends on task difficulty, model tier, and how clean the board itself is. Treating 18.4 points as a universal law is the same mistake as treating the 71%-to-79% "tie" as proof that the harness does not matter: both use one number in place of the whole measurement.

Beyond the rankings, three findings about how to use a harness are worth more

The most useful part of Tan's repository may not be the ranking tables, but the sections on what each harness is like to use. The first is about where to put a rule. A "check gbrain first" rule, placed in the system prompt or in a hook that runs every turn, was followed in 30 of 30 turns; through Claude Code's startup hook it was 26 of 30; with no rule at all — relying only on the skill file saying so — it was 18 of 30. Codex was more extreme: the rule placed in a hook, the system prompt or the startup file was followed 30 of 30 times in each case, and 6 of 30 with no rule. The conclusion is plain: put important rules where the agent sees them every turn; writing them only in a skill document is the same as not writing them — Claude Code never once opened the unmentioned skill file in 30 turns.

The second finding concerns Tan's own product. Opus 5.5 read gstack's skills (/investigate, /qa, /review, /ship) every time, but followed none of their steps — 0 of 46 sessions, at about twice the cost per try. Tan's conclusion is honest: the tasks were too easy for Opus, so this only proves the steps were not followed, not whether following them would help. An evaluation willing to publish negative results about its author's own product is more credible than a leaderboard that only reports good news. The third finding is about shared memory: when gbrain ran with one shared token for the whole bot, it returned records the asking person was not allowed to see — 33, 20, 17 and 9 times out of 51 tries across four setups; with per-person access rules, all four setups dropped to 0 of 51. For people building with agents, that finding is worth as much as any ranking: before wiring shared memory into an agent, sort out permissions first, or the tokens you save will eventually be paid back in a single unauthorised read.

These three data sets do not add up to "the harness matters more than the model"

Now for the brake pedal. None of these three data sets ran a controlled experiment crossing models with harnesses: Tan fixed the model and swapped the harness, the leaderboard fixed the model and swapped the harness, and the Terminal-Bench public board is a collection of self-submitted runs with different configurations. So no data set here can answer "which matters more, the harness or the model" — they can only answer "what happens when you swap the harness and hold the model still". Tan's own wording stays cautious throughout: pass rates "can't tell apart", cost differences "no statistical test", and the tasks are public "dev" results, not confirmation on hidden tasks. His repository also publishes two post-hoc corrections — on 9 November the cost accounting was fixed to stop counting reservation charges, and on 10 November a problem with the egress proxy cutting off slow requests was fixed — plus a GitHub outage that interrupted 353 tries, 351 of which were re-run. An evaluation that writes its corrections and its accidents into the README is harder to read, but more trustworthy.

The conflicts of interest should also be on the table: QM is a Y Combinator project and Tan runs YC; YC is an investor in Hermes; OpenClaw's creator works at OpenAI; gstack and gbrain are Tan's own projects. Tan discloses all of this in the repository's disclosures section, so Hermes's and QM's good results should be read with a small discount — but that does not mean the data is unusable. On the contrary, the fuller the disclosure, the better readers can judge for themselves. One more note: this site has already covered the harness from several angles — Stack Overflow's October essay on "agent equals model plus scaffolding" as a concept, Google's Harness² on scaffolds that evolve themselves, Vercel's durable-harness on surviving crashes, Microsoft's Copilot Harness on swapping models without rewriting the base. This article is about something else: in the week of 5 to 8 November 2026, three new sets of measurements quantified the harness's differences for the first time, in pass rates, dollars and seconds. Concept pieces and measurement pieces are read differently; do not mix them up.

For people who build with agents: compare the bill first, the leaderboard second

Applied to everyday work, these numbers yield a few rules you can take away directly. First, compare cost before pass rate when choosing a harness. In Tan's data, pass rates across 20 tasks could not separate the setups at all, while costs differed by a factor of two; on the leaderboard they differed by 11.7 times. How much your project burns in tokens each month depends more on the shell than on the model. Run a few dozen real tasks from your own repository under each setup, record each one's "cost per solved task", and then decide — copying a leaderboard ranking means letting somebody else's tasks decide for your repository.

Second, put rules where the agent sees them every turn. The system prompt, a per-turn hook, or the startup file — any of the three works; do not hide them in a skill document alone. Tan's data shows 30 out of 30 turns against 0 — that is not folklore, it is a measurement. Third, sort out permissions before turning on shared memory. gbrain's lesson is already written down: dozens of unauthorised reads with a shared token, zero with per-person rules. Fourth, do not switch harnesses for the sake of a top ranking. The best harness follows the model, follows the task, and follows the default settings — a single default that shuts down background processes can drop DeepSeek's Harness several places on one sub-board. Finally, remember Tan's most practical advice: "For now, choose on cost and speed: Claude Code on Opus, Codex on Sol." Not because they are the smartest, but because in his measurements they were the cheapest and fastest on their own models. When the harder Terminal-Bench re-run lands, that advice may change — and when it does, follow the new numbers rather than guarding the old ranking.

Conclusion: the harness race has only just started keeping score

In the first week of November 2026, the competition between coding agents quietly changed tracks: it used to be about whose model was smarter; now it is about whose shell manages money better. Tan's evaluation, Niklaus's leaderboard and the 18.4-point gap on the Terminal-Bench public board are all saying the same thing: the model is the engine, and the shell is the gearbox, the fuel consumption and the dashboard — same engine, and the whole car can behave very differently. But the other side of that sentence matters just as much: none of the three data sets proves the shell matters more than the model. They only prove that the shell's differences are real, measurable, and show up mainly in cost and speed. For people who write code with agents every day, that is actually good news: you cannot change the model or control it, but the shell is something you can work on yourself. A shorter prompt, a few fewer tools loaded, no spinning in place after a failure, rules placed where every turn can see them — none of those changes cost anything, and what they save is real money. A harder round of evaluations is already running; when it lands, we will see whose shell is best at making ends meet.

Views 0Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project

Related articles

A developer with glasses types on a laptop at a wooden desk, an external monitor to the left showing C code on a dark background, a small plant on the desk
News
Microsoft's Coding Model Stops Playing Second Fiddle: MAI Code 2 Scores 96% of Opus 5.5 for 5% of the Cost — Read the Three Caveats First

On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.

Developer WorkflowProduct LaunchModel Updates
A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts
News
Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

Developer WorkflowProduct LaunchTesting & Quality
Against a black background, a conductor seen from behind raises a baton, with swirling trails of light circling beneath him like an orchestra being conducted
News
One API for 45 Models, and It Switches Providers When One Fails: Perplexity's Orchestrator Wants to Be the Switchboard

On 17 November 2026, Perplexity launched the Orchestrator API: one interface reaching OpenAI, Anthropic, Google, xAI and open-weight models such as GLM, Kimi and DeepSeek, billed at the answering model's published rate with no platform fee. When one provider fails, the same model retries on another deployment, and preview routing assigns requests to models by task.

Developer WorkflowProduct LaunchModel Updates