Back to Explore
NewsVibeFix editorialUpdated Nov 19, 2026

63% of Tests Passed, Under 8% of Jobs Done: SWE-CoWork Punctures the Coding Agent's Organizational Illusion

On 15 November 2026, TokenWave AI posted SWE-CoWork on arXiv: coding agents placed in a simulated company with 26 digital colleagues. Across 101 real development tasks, the two best models passed about 63% of test items but completed under 8% of assignments. Handing over the full spec up front added about 16 points of test pass rate — and zero completed tasks. The bottleneck is not the requirements. It is finishing.

Three developers lean over screens showing code, discussing together in a bright office, with colleagues working in the background

In short: 63% of the tests passed, and under 8% of the jobs got done

On 15 November 2026, a paper went up on arXiv titled "SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations". It comes from TokenWave AI, with 26 authors led by Xinping Lei. Its conclusion fits in one sentence, and it is enough to pause every claim that "AI is about to replace programmers": across 101 real development assignments, the two strongest models passed about 63% of the test items, yet completed fewer than 8% of the assignments end to end. Six in ten test items passed, and fewer than one in ten jobs finished — and the gap in between is what the paper is really about.

This is worth writing about not because there is another benchmark leaderboard, but because it measures something different from the old ones. For two years, coding-agent evaluation has worked like this: hand the agent a written specification and a repository, let it change the code, run the tests, and count a pass as a win. Scores climbed from single digits toward saturation, and it began to look as if agents could soon work on their own. SWE-CoWork asks a different question: real development work is never "change code to a written spec". It is "get the job done inside a company". Requirements live in colleagues' heads, on project boards, in versioned documents, chat logs and merge-request histories, and some of them contradict each other. You have to ask the right people, ask the right questions, then write the code, verify it yourself, and hand over an accurate report. The paper drops agents into such a simulated company, with 26 digital colleagues who each hold only part of the picture, and watches how far they get. The result: code-writing ability is already strong, but the ability to "finish a job inside an organization" is still far behind.

First, what this simulated company looks like: 101 assignments, 26 colleagues, 40,000 chat messages

The scale matters, because it decides how much weight the conclusion carries. The benchmark holds 101 development assignments drawn from 101 real open-source repositories, spanning 11 programming languages and 12 application domains. The median assignment adds 2,955 lines of production code across 60 files, graded by 32 executable verifier test nodes. Each assignment starts with only a brief handoff — something like "finish the 2026.09 upgrade; the previous task owner is on leave" — with no complete specification and no ready-made task list.

Each assignment comes with a simulated company of 26 people: module maintainers, compatibility liaisons, release coordinators and more. Every digital colleague has distinct responsibilities and only part of the information, and behaves by rules — disclosing a requirement only when the agent provides sufficient code context, asking for a reproduction first, or referring the agent to another owner. Across the benchmark there are 2,626 employee profiles and 40,836 chat messages. The median assignment has six accountable owners, meaning the requirements for one job have to be collected from six people on average. Scoring uses two test sets: F2P — fail-to-pass nodes that fail before the change and should pass after it, 2,699 in total — and P2P — pass-to-pass nodes that already pass and must not break, 562 in total. An assignment counts as complete only when every F2P and P2P node passes in the same run. That is a strict standard, but it is the standard of real work: shipping means fixing the new feature without breaking the old ones.

The paper also built a control condition called Oracle-Spec: hand the agent the complete requirement set up front, remove all colleagues, and keep the repositories, tests and execution budget exactly the same. That separates two questions: is the job unfinished because "the requirements were never obtained", or because "they were obtained but not implemented correctly"? It is the cleverest design in the paper, and the rest of the analysis keeps coming back to it.

All eleven models fail at "finished": even the best reaches only 7.92%

Now the main table. Of the eleven models, Claude-Opus-5.5 scores 63.84% F2P and GPT-6 Astra 62.91%, with complete-task rates of 6.93% and 7.92% respectively. Third place, Seed Evolving, drops straight to 39.09% F2P and 0.99% complete — one assignment out of 101. The remaining eight models reach at most 35.70% F2P (Qwen3.8-Max) and as low as 13.04% (DeepSeek V4 Pro), with complete-task rates of at most 0.99%, and five of them at 0.00% — not a single assignment finished out of 101.

The P2P column is the interesting one: every model scores between 81.85% and 95.20% on P2P, in a different ranking order from F2P. In other words, "don't break what already works" is something the models do reasonably well. The hard part is "deliver the new requirements". The paper puts it directly: preserving existing behavior and delivering new requirements pose distinct challenges, with the latter remaining a major bottleneck in organizational software engineering. For daily development, that is a practical warning: when you ask an agent to change code, it will probably not wreck the repository — but it will very likely get only part of the job right, and you will not notice the rest unless you verify item by item.

One more number deserves to be pulled out on its own: the gap between models is not a slope, it is a cliff. The top two sit around 63% F2P, the third drops to 39%, and the other eight are all below 36%. On complete tasks, the top two are around 7%, the third is 1%, and the rest are essentially zero. That says there is a threshold between "can work inside an organization" and "can only hover at its edges", and the top two have only half crossed it. What that threshold consists of is what the rest of the paper peels apart, layer by layer.

Comparison chart: F2P pass rate versus complete-task pass rate for the top four of eleven models — Claude-Opus-5.5 at 63.84% versus 6.93%, GPT-6 Astra at 62.91% versus 7.92%; in the Oracle-Spec control, Seed Evolving F2P rises from 39.09% to 55.76% and GPT-6 Astra from 62.91% to 78.84%, while complete-task rates do not move

Handing over the full spec did not help: the Oracle-Spec control exposes the real bottleneck

The first reaction many readers will have is: it failed because the requirements were never fully obtained, right? Hand over the complete spec and the problem goes away? The control condition was built to test exactly that intuition, and it breaks it in half. With complete requirements, Seed Evolving's F2P rises from 39.09% to 55.76%, a gain of 16.67 percentage points; GPT-6 Astra's rises from 62.91% to 78.84%, a gain of 15.93 points. A full spec genuinely helps — about 16 points more test items pass. But complete tasks? Seed Evolving stays at 0.99%, GPT-6 Astra stays at 7.92%. Not one point of movement.

That is the single most memorable set of numbers in the paper. Even with every requirement in hand, Seed Evolving still leaves 44% of F2P test nodes unsatisfied, and GPT-6 Astra still leaves 21%. Those failures happen after the requirements are available — the problem sits in implementation and verification, not in requirement acquisition. In other words, "not knowing what to do" and "knowing what to do but not finishing it" are two different bottlenecks, and SWE-CoWork proves the second one exists and is just as fatal. Fixing only the first — writing more complete specifications, wiring up a better knowledge base — will not move the complete-delivery rate at all.

That has a direct implication for vibe coding workflows. Many people's current habit is to write the specification as precisely as possible, spelling out every field and every edge case in the prompt. The SWE-CoWork data says a detailed spec buys about 16 points of test pass rate, but it will not turn "60% done" into "done". The real shortfall is downstream: after writing the code, does the agent verify item by item? Does it notice the requirements it missed? Does it report honestly in the handoff what is unfinished? The next section follows the strongest model's trajectories, requirement by requirement, to answer exactly those questions.

Following the strongest model's trail: it found the right people, then lost the requirements

The paper traces all 101 Claude-Opus-5.5 trajectories requirement by requirement — the most detailed section in the paper, and the most useful for daily work. The good news first: finding people is something Opus 5.5 does well. Its first question goes to the accountable owner a mean of 93% of the time; it rarely asks the wrong person. The trouble starts after contact: of 2,176 colleague-held requirements, it obtains only 64%. How is the other 36% lost? The paper breaks it down: 16% are raised but never disclosed by the colleague, 17% are never raised at all — the owner was already contacted about another card, but the agent never brought up the requirement on that card — and only 3% belong to owners never contacted. Most missing requirements, in other words, come not from failing to identify the right colleagues, but from incomplete clarification with colleagues already contacted.

And obtaining a requirement does not mean delivering it. Deferred questions often simply die: of 450 deferred threads, 153 are abandoned, with a median of 7.1 hours of execution budget still remaining — not out of time, but out of follow-through. Of the failed test nodes linked to requirements, 56% trace to implementation after the requirement had already reached the agent: the requirement arrived, and the implementation still failed. The most telling figure is the last link in the chain: 31% of the requirements the agent reported as "done" in its final handoff fail their tests. It is not lying. It genuinely believes it finished — it wrote "done" without verifying item by item.

Put together, the failure portrait of Opus 5.5 is clear: accurate at finding people, incomplete at asking; discounted at implementation after acquisition; silent on deferred questions; overconfident at handoff. This is not a bug in one step, but a chain from asking to doing to verifying to handing over, leaking a little at every link, until the complete-delivery rate is under 7%. The paper calls the missing capability "sustained requirement tracking": keeping hold of every requirement's state from clarification through implementation, verification and handoff. For a human engineer, that is daily routine. For an agent, it is the lesson most missing right now.

More work does not mean more output: tool-call counts barely track the score

The paper holds one more counterintuitive finding worth pulling out on its own: the volume of work and the result of the work are barely related. Across the eleven models, requirement acquisition tracks F2P very strongly (Spearman rho = 0.94), but median tool-call counts track F2P at only 0.08 — essentially nothing. Some models burn through their eight-hour execution budget with hundreds or thousands of tool calls, yet obtain fewer requirements than the leading models. DeepSeek V4 Pro is the extreme case: a median of 0 colleague messages per run, only 1.8% of colleague-held requirements obtained, and 13.04% F2P. It is not "thinking independently". It is building behind closed doors — it does not even ask.

The strongest models, conversely, are not the busiest. Another analysis finds that workplace queries before the first edit, and total workplace queries, show little rank association with F2P (rho = -0.10 and -0.17); what does correlate is a lower tool-call index for the first code edit (rho = -0.62, meaning earlier first edits go with higher scores) and more frequent colleague communication (rho = 0.71). The paper cautions that these are associations, not causes — "edit earlier" will not by itself raise a score. But combined with the earlier analysis, the portrait is consistent: strong models are not the ones that investigate most or call tools most, but the ones that start implementing early, communicate often, and ask while they build. Weak models do the opposite: either they research endlessly and never start, or they write code in silence and never ask anyone.

That finding carries a direct suggestion for how to use agents: stop treating "how many tool calls it made, how many tokens it burned" as proof of work. The paper's usage analysis confirms it: across the nine models with available telemetry, model calls, token usage and context compactions show no consistent relationship with delivery; large input-token totals often reflect repeated processing of retained context rather than more extensive investigation. The numbers worth watching are how many requirements it obtained, how many items it verified, and how many of the "done" claims in its handoff actually passed. SWE-CoWork reports all three. People using agents every day can count them too.

The paper's limits: a simulated company is not a real company, and a v1 preprint is not a verdict

Before the conclusions are stated too firmly, the paper's limits need stating too. First, it is a v1 preprint posted on arXiv on 15 November 2026, without peer review. The numbers are the authors' own runs; the code and dataset are public (swe-cowork-bench on GitHub and the dataset on Hugging Face), but as of writing no independent team has reproduced the results. When citing this paper, the honest phrasing is "the paper reports", not "it has been proven".

Second, a simulated company is still a simulation. The 26 digital colleagues are rule-governed language models: they disclose a requirement only when the agent provides enough code context, ask for a reproduction, or refer the agent to another owner. Real colleagues are messier — they have moods, hold meetings, agree verbally and forget to write it down, and change their minds about requirements halfway through. The simulated environment's advantage is control: it is reproducible, and every requirement can be traced to a specific test. Its cost is that it measures "working in a simulated organization with explicit rules", not "working in a real office". The conclusions hold inside the simulation; applied to real teams, they are directional guidance, not precise predictions.

Third, the model list itself deserves attention. Several of the eleven names appear in public evaluation for the first time: GPT-6 Astra, Seed Evolving, Qwen3.8-Max, Kimi K3, GLM 5.3. The paper does not publish detailed version information or sampling parameters for these models, and the execution budget is eight hours per assignment. Whether scores can be compared directly across models depends on conditions being identical — the paper states they are (same tasks, same repositories, same tests, same scaffold) — but how friendly the scaffold itself is to different models is not explored in the paper. One more point: the background figures the paper cites — Anthropic's report that as of May 2026 over 80% of code merged into its internal codebase was authored by Claude, and that the typical engineer merged eight times as many lines per day in Q2 2026 as in 2024 — are the Anthropic Institute's claims cited by the paper, not measurements from SWE-CoWork itself, and should be attributed as such rather than presented as this paper's findings.

Drawing the line against the site's two earlier evaluation stories: three different things measured

The site has already published two evaluation stories recently, and readers could easily read all three as "yet another benchmark saying agents are not good enough". The lines are worth drawing. The cycle224 Garry Tan harness evals measured the "shell": the same model with only the agent harness swapped, where pass rates could not be separated but cost and speed differed by more than double. Its conclusion is about selection — the shell decides the bill, not the brain. The cycle221 OpenAI research-acceleration report measured "research work": agents doubling their success rate on research tasks, while the hardest work still belonged to humans. Its conclusion is about the boundary of research automation.

SWE-CoWork measures a third thing: "organizational work" — an agent starting from one brief handoff inside a simulated company, finding people, asking for requirements, writing code, verifying, and handing over, all the way through. Its bottleneck is neither the shell nor research ability, but sustained requirement tracking. All three conclusions look like they point at "agents are not that strong yet", but "strong" means something different in each: the harness story says "changing the shell doesn't change the brain", the research story says "the hard jobs still belong to humans", and this story says "finishing a job is not the same as finishing it correctly". Taken together, the three form a fuller picture: model capability, harness, and organizational collaboration are three separate weak links, and fixing one does not lengthen the other two by itself.

For people who vibe-code: stop watching the pass rate, watch these three numbers

The most useful thing in this paper for daily work is not the eleven-model ranking, but the three checks it implies. The first: how many requirements were obtained. Before letting an agent work, ask it one question — "whose hands are this job's requirements in, and who will you ask first?" If it cannot answer, or answers and then starts writing code immediately, it is what the paper calls a passive "spec executor": starting work before the requirements are assembled, which guarantees rework. The better order is reversed: first have the agent list the requirements, marking the source of each (which colleague, which document, which chat record), and flagging the missing ones — then start coding.

The second: how many items were verified. When the agent says "done", do not take it on trust — ask it to show evidence item by item: which tests correspond to each requirement, whether they ran, whether they passed. The paper's data is right there: 31% of the requirements the strongest model claimed as complete in its handoff failed their tests. The same applies to your project — when the agent says "all fixed", spot-check the test results for a few key requirements. The more thorough habit is to have the agent report only what it has verified, and mark the rest "unverified" rather than "done". That one change in habit cuts rework in half.

The third: whether deferred questions got followed up. In the paper, 153 deferred discussion threads were simply abandoned, with a median of 7.1 hours of budget left — the agent did not run out of time, it forgot. A human engineer who gets stuck in a meeting writes it down and follows up afterwards; an agent asked by a colleague to "provide a reproduction first" goes off to do something else and never comes back. A simple rule for daily agent use: before each conversation ends, have it list "unresolved questions and who to ask next", and start the next conversation by going through that list. The list itself is the minimum viable version of "sustained requirement tracking" — no new tooling needed, one sentence adds it.

One final suggestion for teams: stop grading agents by tool-call counts and token usage. The paper's data already shows those numbers barely track delivery. What deserves grading is requirement acquisition rate, item-by-item verification rate, and handoff accuracy — the three numbers above. Change the metrics, and the agent's behavior changes with them: it will ask people more, start earlier, verify more often, instead of quietly racking up tool calls.

Conclusion: what agents lack is not the ability to write code, but the ability to finish a job

For two years, the story of coding agents has been "scores are rising": SWE-bench climbing from single digits toward saturation, new leaderboards appearing constantly. SWE-CoWork changes the ruler, and the story changes with it: 63% of test items can pass, while fewer than 8% of jobs get finished. With a different ruler, the weak link shows: not writing code, but "finishing a job inside an organization" — requirements scattered everywhere and never fully asked for; implementations discounted after acquisition; deferred questions never followed up; handoffs that overstate what was done.

For people who vibe-code, that is both good news and bad news. The good news is that the weak link can be trained: a requirements list, item-by-item verification, and a deferred-questions follow-up — none of it waits for the next generation of models, all of it pays off immediately. The bad news is that the weak link is in your hands, not the model's: an agent will not grow a "sustained requirement tracking" habit on its own. You have to build the scaffold — make it list requirements before coding, show verification evidence item by item, and hand over an unresolved-questions list every round. The paper's title says coding agents should be evaluated as "digital employees in organizations"; read in reverse, it says the same thing: manage it like a new hire who needs onboarding before it can genuinely finish a job for you. Whether the next generation of models turns 7.92% into 30% or 50% will have to wait for the next evaluation. Until then, start with the three numbers in your own hands — the return is immediate.

Views 1Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project

Related articles

A developer with glasses types on a laptop at a wooden desk, an external monitor to the left showing C code on a dark background, a small plant on the desk
News
Microsoft's Coding Model Stops Playing Second Fiddle: MAI Code 2 Scores 96% of Opus 5.5 for 5% of the Cost — Read the Three Caveats First

On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.

Developer WorkflowProduct LaunchModel Updates
A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts
News
Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

Developer WorkflowProduct LaunchTesting & Quality
Against a black background, a conductor seen from behind raises a baton, with swirling trails of light circling beneath him like an orchestra being conducted
News
One API for 45 Models, and It Switches Providers When One Fails: Perplexity's Orchestrator Wants to Be the Switchboard

On 17 November 2026, Perplexity launched the Orchestrator API: one interface reaching OpenAI, Anthropic, Google, xAI and open-weight models such as GLM, Kimi and DeepSeek, billed at the answering model's published rate with no platform fee. When one provider fails, the same model retries on another deployment, and preview routing assigns requests to models by task.

Developer WorkflowProduct LaunchModel Updates