Back to Explore
NewsVibeFix editorialUpdated Nov 20, 2026

1,700 Investigations, Only 15 Fixes Merged As Written: OpenAI's Performance Factory Tells the Honest Truth About Agents

On 19 November 2026, OpenAI disclosed its internal Performance Factory: Codex agents automatically investigate ChatGPT performance regressions, running 1,700+ investigations in three months and finding 600+ root causes. Yet only about 15 fixes were merged exactly as Codex wrote them — agents can take over the investigating, but fixing still needs a human.

An engineer wearing glasses sits with his back to the camera, studying two monitors showing server metrics and a four-golden-signals dashboard, latency at 150 ms and errors at 0.2%

In short: when ChatGPT slows down, let an agent investigate — and let a person decide

On 19 November 2026, OpenAI disclosed an internal system called the Performance Factory. The news came out through a VentureBeat exclusive, based on an engineering report OpenAI shared with the publication plus comments from OpenAI's performance team, with a post from OpenAI's developer account @OpenAIDevs the same day. The job it does is specific: using Codex agents to automatically investigate and mitigate performance regressions — a service getting slower after an update. It has been running internally for just over three months, keeping ChatGPT fast.

Three sets of numbers first, because the whole judgment of this story hangs on them. In just over three months, the system tracked more than 1,700 investigations, about 17 a day; it found the root cause in more than 600 of them; about 400 of those progressed to a proposed fix; roughly 50 fixes that eventually shipped were influenced by or matched agent suggestions; and only around 15 shipped fixes were the exact pull requests authored by Codex. 1,700, 600, 400, 50, 15 — that funnel is the heart of this story: agents can already take over the investigation work, but handing over a fix exactly as written is still a long way off. OpenAI admits as much itself: the results fall short of its ambitions for automated remediation.

How it grew: from manual log-reading to agents running the process

The Performance Factory was not created out of nothing; it grew step by step out of the performance team's daily grind. It started fully manual: when a service slowed down, engineers examined logs and traces, looked at code changes, and debugged live systems. To make that work less painful, the team built more than 30 specialized tools. Then Codex arrived, and engineers could ask it why a service had become slower, letting the agent investigate logs and use tools to provide the answer — but a person still had to steer every step.

The next step was writing the investigation method itself down. The team turned the performance team's procedures into a reusable Codex skill called perf-forensics, so an agent could reproduce an investigation the same way every time. Once that worked, the team found it could hand Codex a performance alert and get a diagnosis back in a single run. But at that point engineers had become intermediaries: copying alerts into Codex and pasting the findings into Slack. So the team simply connected the alerts to cloud-based Codex sessions, and the intermediary step disappeared.

Even then, the investigation ended with a diagnosis that needed an engineer to review and implement. Meanwhile, the ChatGPT performance team was dealing with regressions almost daily and struggling to keep track of them. The team first built a regression tracking system to manage the workload, and then realized that each tracked regression could become a task that Codex would automatically investigate and advance through different stages — identifying the root cause, proposing a fix, and verifying recovery. The Performance Factory's task-based architecture grew out of that realization.

Four parts: tasks, dispatch, cloud execution, subscriptions

The system has four main components, and the division of labor between engineers and agents is written into the architecture. The first is the task service, which stores tasks and their progress; engineers define workflows through configuration files and write Markdown prompts that determine what the agents must do at each stage. The second is the dispatch loop, which periodically checks on tasks and, when one is ready to advance, starts a Codex session and assigns the task to it; the agent records its findings and updates the task when it completes its work.

The third is an internal API for running Codex in the cloud. Agents operate in environments that have access to OpenAI's code repositories and observability infrastructure, with permissions controlled and actions tracked. The fourth is subscriptions, which connect tasks to external events: if an agent opens a pull request and needs an engineer's review, the task can pause until it receives the engineer's feedback on GitHub; when a deployment completes, the next step is triggered.

The way tasks record their own history deserves a paragraph of its own, because it solves the most common defect of multi-agent systems: context that grows fatter with every handoff. A task record contains a free-form field where agents leave notes, along with structured metadata such as when a regression began and when its root cause was identified. Instead of copying large amounts of profiling data into the task, agents record the relevant metrics, queries, and high-level findings; the next agent receives the task record and can rerun those queries if it needs to verify the results. The evidence is preserved, the context is not overloaded, and the next agent can still check the previous one's conclusions. OpenAI lists this as one of its lessons for enterprise teams: preserve evidence without overloading agent context.

Adversarial review: letting another agent pick holes

The most interesting design in the system is adversarial review. The team found that agents sometimes stopped at the first plausible explanation, even when the evidence did not fully support it. So they added a step in which a separate Codex session challenges the original diagnosis. The reviewer receives the entire task record but starts with a fresh context, and its instructions explicitly tell it to find gaps in the diagnosis, verify the supporting evidence, and test alternative explanations. Its whole session is devoted to challenging the original agent's conclusions.

The report gives a concrete example. In one investigation, an agent attributed failing ChatGPT conversations to errors in image downloads. The reviewing agent noticed that the internal service responsible for locating files had already recovered while image requests continued failing, so the original diagnosis did not explain the continuing failures, and the investigation was reopened. When adversarial review first went live, it sent back a substantial number of diagnoses for further investigation; the team used that feedback to improve the original investigator's instructions, and the return rate fell. Currently, adversarial review sends back around 25% of diagnoses.

There is a second meaning to this design: it admits that agents can be confidently wrong. Instead of waiting for a user to discover the mistake, the process includes a role whose job is to disbelieve. The team later added adversarial review to proposed fixes as well, because agents sometimes propose overly complicated fixes or introduce new problems. The report's example is adding a cache: response times improve, but the cache can serve stale data. Reviewing the fix is protection against fixing one thing and breaking another.

The honest funnel: 1,700 investigations, 15 fixes merged as written

Laid out flat, the numbers show what the Performance Factory is really worth. More than 1,700 investigations in just over three months, about 17 a day — the investigation step is already automated and running. Root causes found in more than 600 cases, about 35% — meaning close to two-thirds of investigations never pinned down a defensible root cause. About 400 progressed to a proposed fix, roughly 24% of all investigations. About 50 fixes that shipped were influenced by or matched agent suggestions, around 3%; and only about 15 shipped fixes were the exact pull requests Codex wrote, under 1% of investigations.

Funnel chart of the Performance Factory's first three-plus months: more than 1,700 investigations, root causes found in more than 600 (about 35%), proposed fixes in about 400 (about 24%), shipped fixes influenced by agent suggestions in about 50 (about 3%), and exact Codex-authored pull requests merged in about 15 (under 1%)

Read the funnel together with OpenAI's own words. The performance team acknowledges that these results fall short of its ambitions for automated remediation — but also that much of the work of resolving regressions, detecting them, identifying their causes, tracking their progress, and verifying recovery, has been automated, while improving the quality of proposed fixes remains a priority. In other words, the system is best at "finding out" and weakest at "fixing". Codex performs best when the cause of a regression is clear and the fix is relatively contained: the report's example is a change that caused an application to serialize the same conversation several times while processing a request, where Codex could identify the redundant work and propose serializing it once and reusing the result. With complex changes — a new feature adding latency and requiring architectural modifications — agents sometimes propose overly complicated fixes or introduce new problems, which is exactly why proposed fixes go through adversarial review too.

Two details at the tail of the funnel are worth recording. First, after an engineer deploys a fix, another agent checks whether the performance issue has been addressed; if the problem persists, the workflow sends the task back for further investigation — verifying recovery is automated as well, not "case closed once a fix is proposed". Second, the team has completed a workflow evaluation suite that it plans to use to improve agent-generated fixes. OpenAI's attitude toward the system is not "shipped and done" but "get it running, then keep grinding it better with an evaluation suite".

Evaluating the system itself is another part of the report worth a closer look. OpenAI backtests investigations against historical incidents: it has agents re-examine real performance events from the past and measures how accurately they identify when problems started, their impact, and potential causes. That method has a built-in trap — information collected after an incident can reveal the correct answer, and historical evidence can expire — so backtest results need careful interpretation and cannot be read directly as accuracy in production. The team also evaluates agent-generated summaries separately, checking whether engineers can understand the evidence and proposed actions from the summary alone. Whether a summary can be understood is its own metric, counted separately from whether the diagnosis is right.

That evaluation thinking is useful for ordinary teams too. Before letting an agent do a job, decide how you will measure whether it does the job well: take problems that really happened, have the agent redo them, and score it against what actually happened; grade summaries separately from diagnoses instead of mixing the two. Without that, improvement is guesswork — it feels smarter today and dumber tomorrow, with nothing reproducible to show. OpenAI building a dedicated workflow evaluation suite to grind fix quality better is itself the point: in an agent system, evaluation is not something you remember after launch; it is part of the system.

Where people still sit: decisions, deployments, and an agent that watches the agents

People have not left the process; they have moved to a few specific points in it. Once a diagnosis passes adversarial review, Codex proposes a fix and opens a pull request, but an engineer reviews the code and decides whether to merge it. After a fix is deployed, an agent checks whether the problem is really gone, but the deployment itself remains an engineer's job. Will that change? OpenAI says it is investigating whether agents could eventually deploy some low-risk fixes without human approval — "investigating", not "already able to".

Another problem people still have to handle is volume. When the system first ran, engineers were given an interface full of investigation results and had to determine which findings mattered. The system now makes it easier to navigate results by grouping related incidents, ranking their estimated impact, and generating concise reports with graphs and metrics. OpenAI lists this as a lesson too, under the heading "avoid AI fatigue": where humans must review an agent's output, make sure the most pertinent information is presented to them, instead of overwhelming them with content.

One more link in the chain is easy to miss: who watches the agents themselves. OpenAI runs a daily agent that examines recent agent logs for recurring tool failures and permission problems; when it finds an issue that repeatedly blocks investigations, the Performance Factory can create a task and propose a fix for an engineer to review. Evaluations then check whether the change resolves the problem. In one sentence: there is an agent in the system whose job is watching whether the other agents are tripping over themselves.

One more number belongs in the background: according to The Pragmatic Engineer in September, OpenAI's vice president of engineering, Venkat Venkataramani, said the load on the systems for building, testing, and rolling out software had grown about tenfold in roughly six months. The faster agents write code, the more change the performance team has to absorb — and the Performance Factory addresses exactly the gap where agent-driven development outruns performance maintenance. That background is second-hand reporting, included here to explain why the system exists, not to vouch for how well it works.

Boundary with what VibeFix already published: an internal ops system, not a Codex product or a benchmark paper

VibeFix has already published several OpenAI and Codex articles, so this one needs to be separated from them first. The first was Pando on 2 November: OpenAI rewrote its Git backend after Codex overwhelmed its Git servers — an infrastructure release. The second was the ChatGPT and Codex shared allowance from December (2 November), about a product interface merge and shared usage pools. The third was OpenAI's bug bounty rewards (2 November), about security bounty rules. All three touch Codex, but none of them covers how Codex maintains ChatGPT itself inside OpenAI.

Another easy confusion is cycle225's SWE-CoWork: a benchmark paper that used 487 messages and 36,411 tool calls to test whether coding agents work as "digital employees" — concluding that tests passed more than 60% of the time while tasks were completed under 10% of the time. That was a paper, an evaluation, a judgment about "whether it works"; this article is about a deployed internal system, three-plus months of production numbers, a record of "how it is actually done". One measures in the lab, the other runs in production — they are read differently.

Dots and Duet are different again. Those are user-facing, always-on agents that approach people proactively and handle everyday work; the Performance Factory is an invisible internal system with no user interface and no chat entry point. Its users are OpenAI's own performance engineers. Comparing it with Dots and Duet would be comparing a backstage operations system with a front-stage assistant product — not the same thing.

For people building with agents: four things worth copying directly

The value of the Performance Factory for ordinary teams is not in building your own "performance factory", but in four practices it writes down clearly enough to copy. The first is "write the investigation method down as a skill". OpenAI's perf-forensics is a written investigation procedure that agents reproduce. For a small team, this is the easiest one to adopt: write down the steps you follow when a performance problem appears — which logs to read, which traces to check, which deployment records to compare — as a Markdown document, and have the agent run it. The next time the same kind of problem appears, the agent does not have to start from scratch, and neither do you.

The second is "let another agent pick holes". An adversarial review costs one extra session and buys a filter that stops "the first plausible explanation" before it gets merged. A small team may not be able to afford two sets of agents, but it can add a human review at the key conclusion: when the agent hands over a diagnosis, have it also hand over which alternative explanations it ruled out and what evidence ruled each one out. What was ruled out matters as much as what was found — in the image-download example from the report, the mistake was that nobody asked "were the other explanations excluded?"

The third is "keep only reproducible evidence in the task record". OpenAI's practice is to record metrics, query statements, and high-level conclusions, not copy large amounts of raw data; the next agent that wants to verify reruns the queries itself. Agent handoffs inside a team can follow the same rule: a handoff document should say "what was checked, what the conclusion was, and what reproduces it", not paste thousands of lines of logs. Context is saved, and conclusions can still be checked.

The fourth is "rank agent output by priority". The investigation results engineers see each day are grouped, ranked by estimated impact, and summarized in short reports with charts, instead of raw output poured straight at people. This applies to any team that has agents producing work in bulk: agents can produce more in a day than people can read in a day, so without ranking and summarizing first, people get drowned by agent output. OpenAI calls it "AI fatigue" — a term worth remembering for every team that uses agents.

Conclusion: the next stop for agents is maintenance, not creation

What is really worth noticing about the Performance Factory is not the 1,700 investigations or the 15 fixes merged as written, but the direction it marks. For the past year, the coding-agent story has starred "creation": writing code, building apps, shipping products. The Performance Factory stars "maintenance": finding why things got slower, tracking regressions, verifying recovery, and turning what engineers learn into reusable workflows. Creation is visible; maintenance is invisible — but once a system is running, maintenance is where most of the time goes.

It is also worth noticing that OpenAI did not oversell this disclosure. 1,700 investigations, more than 600 root causes, about 400 proposed fixes, about 50 shipped fixes influenced by agents, 15 merged as written — those numbers are honest. They say agents are already useful at "finding out" and still need a person standing behind them at "fixing". For people doing vibe coding, that judgment has a practical meaning: what you build with an agent will sooner or later need someone — or another agent — to maintain it. Getting it written is only the first step; keeping it running, finding out why it slowed down, and fixing it when it does is what decides whether a product survives. The Performance Factory does not answer "how fast can agents write", but "when what agents wrote gets slower, what then?" — a question that will sooner or later land on everyone who builds with agents.

Views 1Comments 0

Comments (0)

ME
0/1000
Loading comments...

Sources

Browse projectsPublish your project

Related articles

A developer with glasses types on a laptop at a wooden desk, an external monitor to the left showing C code on a dark background, a small plant on the desk
News
Microsoft's Coding Model Stops Playing Second Fiddle: MAI Code 2 Scores 96% of Opus 5.5 for 5% of the Cost — Read the Three Caveats First

On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.

Developer WorkflowProduct LaunchModel Updates
A developer types at a dual-monitor workstation; the left screen shows a dark code editor, the right a testing dashboard with green checkmarks and bar charts
News
Did Your Agent Get Better or Worse? AWS Open-Sources UAEF to Measure Agents from Every Framework with One Ruler

On 11 November 2026, AWS open-sourced UAEF: adapters turn traces from LangGraph, LangChain, Strands and Bedrock AgentCore into one format, then 36 metrics across 7 dimensions score them, flagging regressions past 5%. The judge model is hardened too — scores must cite verbatim evidence, and never decide safety.

Developer WorkflowProduct LaunchTesting & Quality
Against a black background, a conductor seen from behind raises a baton, with swirling trails of light circling beneath him like an orchestra being conducted
News
One API for 45 Models, and It Switches Providers When One Fails: Perplexity's Orchestrator Wants to Be the Switchboard

On 17 November 2026, Perplexity launched the Orchestrator API: one interface reaching OpenAI, Anthropic, Google, xAI and open-weight models such as GLM, Kimi and DeepSeek, billed at the answering model's published rate with no platform fee. When one provider fails, the same model retries on another deployment, and preview routing assigns requests to models by task.

Developer WorkflowProduct LaunchModel Updates