OpenAI Teaches AI to Speak COBOL: Open-Source MultiLangBench Makes Models Rewrite SQLite from Scratch as Migration Accuracy Jumps from 5.4% to 97.9%

On October 19, 2026, OpenAI published a research post with an unsexy title: "Advancing coding in low-resource programming languages." No new model launch, no keynote, not even a decent demo video. But if your work touches real-world legacy systems, this may be the most important technical announcement OpenAI has made this year: for the first time, it turned "can AI actually handle ancient languages like COBOL?" into something measurable — and open-sourced the benchmark to prove it. It's called MultiLangBench, and it asks models to reimplement SQLite from the documentation alone, from scratch.

What's interesting here isn't the score. It's the uncomfortable truth the post exposes: the "AI coding revolution" we keep talking about has really only happened in a handful of resource-rich languages — Python, JavaScript, TypeScript. Step outside that comfort zone and AI performance falls off a cliff. And the languages AI is worst at happen to run banks, tax systems, weather forecasting, and nuclear reactor safety analysis. OpenAI choosing to confront that gap head-on deserves serious attention — with one caveat nailed down first: every number on the scorecard comes from OpenAI itself. Keep the words "vendor-reported" in mind; we'll discount each claim below.

Your bank still runs on COBOL. Your AI assistant can't read it.

Start with the basics: COBOL is not a museum piece. OpenAI's post cites the numbers — COBOL supports more than 40% of the world's online banking systems, 80% of in-person card transactions, and 95% of ATM transactions. Fortran still runs weather models and nuclear safety analysis. Visual Basic keeps countless internal enterprise systems alive. These languages share one trait: the amount of public code and documentation available for them is tiny compared to Python or JavaScript. OpenAI calls them "low-resource languages" — little training data, little exposure, and naturally, little learned.

The contrast is absurd. On one side, coding agents from every major lab race each other up the SWE-bench leaderboard. On the other, the real-world plight of bank IT departments: the engineers who wrote that COBOL have mostly retired, documentation is fragmentary, and the most critical business rules — how exactly interest is computed, at which decimal place rounding happens — live only inside the code itself. Point the strongest AI coding assistant at 40-year-old COBOL and it will most likely confabulate with great confidence. That's not the model being "dumb"; it simply never saw much decent COBOL during training.

The other side of the gap is people — and the people problem is more urgent. A 2019 workforce study found the average COBOL developer was 58 years old, with roughly 10% retiring each year; a Forrester survey commissioned by Deloitte found 71% of mainframe teams report being understaffed. An ACT-IAC report on legacy code modernization estimated 84,000 unfilled mainframe positions in the United States alone. The script already ran once during the pandemic: in 2020, New Jersey's unemployment insurance system collapsed under a flood of claims, and the state publicly appealed for retired COBOL programmers to come back and help. "A government agency begging retirees to fix critical infrastructure" — that image is becoming banking's daily reality at a much larger scale.

Worse, these systems aren't waiting to be replaced — they're growing. A Micro Focus global survey estimated roughly 800 billion lines of COBOL code running in production every day, with 92% of respondents calling COBOL a strategic asset and over half expecting their COBOL applications to keep running for at least another decade. The IRS, the Social Security Administration, the Treasury, the Department of Veterans Affairs — all run on COBOL. If COBOL stopped working tomorrow, large parts of the American economy would go dark within hours. This isn't nostalgia; it's a load-bearing wall — and its maintenance crew is retiring at 10% a year.

So the problem OpenAI set out to solve isn't "make AI write prettier Python." It's "make AI useful in languages it has barely seen." That matters for the vibe coding era too: today you build new projects on mainstream stacks, but tomorrow, when you want an agent to take over a decade-old internal system or migrate a client's legacy code, the model's low-resource language ability determines whether it's a helper or a hazard.

Two new exams: build a database from scratch, then move the old code without breaking it

OpenAI designed two evaluations for two distinct skills: building from zero, and migrating without loss. The first is MultiLangBench; the second is a COBOL migration test. Both are public, and the first is fully open source (Apache-2.0, under OpenAI's GitHub organization).

MultiLangBench: docs only, reimplement SQLite

The idea is blunt: hand the model the official SQLite 3.40.1 documentation — no source code — and ask it to implement a SQLite-compatible database engine in one of 20 programming languages. Grading works like this: run the model's executable against test cases converted from SQLite's own TCL test suite, roughly 24,000 of them (a 272-test subset is recommended for routine, cheaper runs). The clever part is the judge: instead of checking whether the model's text output looks like SQLite, the real SQLite 3.40.1 reads the database file the model produced. Get the on-disk file format right or fail — much harder to fake through memorization.

OpenAI also wrote three explicit anti-cheating rules into the repo README: first, scrub every trace of SQLite from the eval environment (libraries, executables, even the sqlite3 bundled with Python) so the model can't just import its way to full marks; second, no internet access, so it can't download the SQLite source tree; third, write in the wrong language — or mix languages — and the pass rate is set to zero. The score itself is a tuple — (pass rate, API cost) — writing "how much did this cost" into the grade. A model scoring 60% for a few dollars and one scoring 62% for ten times the money are different products. Every team evaluating agents for their own codebase should steal this structure: public spec, independent oracle, shortcuts explicitly banned, cost reported alongside the score.

The list of 20 languages is worth a closer look too: besides the four low-resource headliners — COBOL, Fortran, Visual Basic, OCaml — it includes C, C#, C++, Common Lisp, Go, Java, JavaScript, Lean, PHP, PowerShell, Python, Ruby, Rust, Scala, Swift, and Zig. The evaluation is language-agnostic: feed each implementation the same starting database and a SQL query, then check whether the output database matches the reference implementation — one test logic stretched across 20 languages, no per-language test harness needed. That's also its most ambitious claim: the next vendor declaring "our model understands COBOL best" can skip the argument and just run the same exam. The cheaper GPT-6 Luna deserves its own glance: it rose only from 10.9% to 17.1% on COBOL, 25.6% to 34.1% on OCaml, 21.4% to 31.4% on Fortran — at $0.77 per sample versus $3.48. Translation: budget-tier models are still far from usable in low-resource languages; if you want low-resource migration today, you're still paying for reasoning budget. The cost dividend hasn't reached the middle tier yet.

The COBOL migration test: translate legacy code, but not one byte may differ

The second test hits closer to real-world pain: translate COBOL programs into C++, Java, Kotlin, Python, or Rust. Grading uses hidden tests demanding byte-for-byte parity on observable outputs — a migration counts only if every test passes. OpenAI highlights one deliberate trap: COBOL fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. So a "helpful" rewrite that rounds for you runs without a single error yet produces different financial numbers. In a banking system, that's an incident. Making "behavioral equivalence" rather than "looks right" the sole criterion is the hardest-core part of this test.

The scorecard: from "barely works" to "nearly saturates" (all vendor-reported)

Every number below comes from OpenAI's October 19 post. No independent third party has reproduced them. Read the scores first, then the discounts.

• MultiLangBench (20-language average, max reasoning): GPT-6 Astra hit an 87.4% pass rate; the previous generation, GPT-5.6 Sol, managed only 26.8%. Astra scored above 83% in 19 of the 20 languages — COBOL was the lone exception, though it still jumped from 10.9% to 72.1%.
• By language: Fortran rose from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4%, OCaml from 25.6% to 85.8%. Note that COBOL remains the hardest of the 20 languages — its "best" score of 72.1% is itself evidence that the ceiling for low-resource languages is still far away.
• The cheap one: GPT-6 Luna scored 40.3% on the same test at about $0.77 per sample versus $3.48 for the older model — roughly 78% cheaper. A usable, affordable middle tier is forming; not every job needs the priciest model.
• COBOL migration (low reasoning effort): GPT-6.1 Sol reached 86.6% accuracy against 5.4% for GPT-5.6 Sol at the same setting — from "basically unusable" to "usable" in one generation — while costing 39% less per API call.
• COBOL migration (higher reasoning effort): GPT-6.1 Sol hit 97.9%, which OpenAI says "nearly saturates the eval." The older model has no published number at this setting, so no comparison is possible.

The jump from 5.4% to 97.9% tells its own story: the old "it just can't do it" for low-resource languages was largely a matter of these languages never being seriously trained on, not of the task being unsolvable. Once data, training, and measurement arrive, the curve gets very steep. That's good news for every niche stack — your obscure language isn't doomed to be left behind by AI; it just hasn't had its turn at the training-data trough yet.

Three judgments: why this matters more than another leaderboard

Judgment one: the eval design is more worth copying than the scores. AI coding leaderboards have multiplied past the point of numbness, but MultiLangBench gets three things genuinely right: first, a judge independent of the model — the real SQLite acts as oracle instead of another model handing out grades; second, cost inside the score — the (pass rate, API cost) tuple forces the question "what did this score cost to buy"; third, anti-cheating rules stated up front — offline, scrubbed environments, zero for language violations. Many leaderboards won't even discuss whether the model saw the test questions during training. If you're evaluating agents for your own team, copy this structure directly.

Judgment two: low-resource languages are the "dark matter" of model capability. High scores in mainstream languages mean less and less — Python code piles up in training corpora by the mountain, and a high score may just be recitation. Low-resource languages are a mirror: little data, nothing to memorize, so what comes out is closer to true generalization. By putting COBOL, Fortran, and VB on stage, OpenAI is conceding a fact: for agents to enter the real world, they must survive the "legacy system" gauntlet. Banks, governments, and manufacturers won't rewrite their old systems just because AI only speaks Python. Whoever cracks low-resource languages first gets the customers with the deepest budgets and the strongest motivation.

One sobering number to add: Gartner's June 2026 mainframe forecast says that more than 70% of mainframe exit projects initiated in 2026 will fail to produce their intended benefits, with overestimation of generative AI tooling as the primary cause. That doesn't contradict OpenAI's scores — it's about something else: translation accuracy and migration success are two different problems. Anthropic's February 2026 post on COBOL modernization put it more honestly: what AI really changes is the economics of the discovery and analysis phase — making "figuring out what the old system actually does" cheaper — not making translation magically correct. Cognition, the company behind Devin, reported in April 2026 on its work at Itaú Unibanco (Latin America's largest private bank): 5–6x faster refactoring, zero production errors — while explicitly drawing the line that fully autonomous migration of transactional systems is "out of reach for agents today." So the pragmatic posture is to use AI as a knowledge-transfer accelerator: extract the tacit knowledge out of the system and into tests while the engineers who understand the business are still around, instead of betting on one fully automatic translation. The window is closing; 84,000 empty positions won't fill themselves.

Judgment three: open-sourcing the eval is the hardest currency in this post. Scores can be self-reported, but releasing MultiLangBench under Apache-2.0 means anyone can run Claude, Gemini, or DeepSeek through the same exam. OpenAI doing this signals either supreme confidence in its own models or a calculation that defining the arena beats winning one leaderboard. For the community it's a win: the next vendor claiming "our model understands COBOL best" can be told to run MultiLangBench and skip the pitch deck.

And OpenAI isn't the only one fighting over who defines the arena. IBM's Watsonx Code Assistant for Z reached version 2.8 in March 2026, purpose-built for COBOL-to-Java migration and wired through MCP directly into job control metadata, copybook libraries, and VSAM data definitions — it has something OpenAI doesn't: production context. AWS Transform for mainframe, launched in 2025, markets itself as the first agentic AI service for end-to-end mainframe modernization, with Transamerica, Allianz, and Marriott Hotels as reference clients. GitHub Copilot has COBOL explanation, test generation for undocumented functions, and refactoring suggestions as documented capabilities. The emerging shape: OpenAI writes the public exam, IBM and AWS sell proximity to production systems, Anthropic and Cognition sell the workflow. The exam score is just the ticket in; the real contracts go to whoever holds the copybooks and the JCL. Which is also the shrewd part of OpenAI open-sourcing the eval — if you can't be closest to production, be the one who writes the test.

The discounts: five reservations

Five buckets of cold water, all grounded in public information — skepticism, not contrarianism.

First, contamination risk is real. SQLite's source and TCL test suite are public and almost certainly in training data. Hashed test names and a conversion step guard against memorized test files, but not against a model that memorized the SQLite source itself. Cross-language comparison (the same model's spread across 20 languages) is the more trustworthy reading — COBOL lagging clearly behind everything else suggests the test is measuring something real.

Second, a benchmark program is not a 40-year-old banking system. MultiLangBench is clean: complete docs, a well-defined target, ground truth available. Real core banking systems are the opposite: missing docs, business rules scattered through the code, everything coupled to everything. OpenAI itself cautions that production migrations still need expert review and testing against business rules. 97.9% is a score on a carefully designed exam — not a license to rewrite your bank's core ledger tomorrow.

Third, every number comes from the exam's author. As of publication, no independent third party has run MultiLangBench and published results. Explainx, which dissected the eval, also notes the repo launched with three commits and didn't even list the target languages in its README. The benchmark's credibility will only be established when the first independent reproductions land. Until then, mentally prepend "according to OpenAI" to every "from X% to Y%."

Fourth, the test measures the cleanest version of the problem. Tech Times' October 21 follow-up analysis lists the dirty work real migration involves: copybooks — shared data-layout definitions referenced by dozens or hundreds of programs, where one change propagates everywhere — while OpenAI's test only covers single-file programs; JCL — the job control language that governs how COBOL programs get submitted and how files are allocated on mainframes, where a translated program can pass every unit test and still fail inside the job stream it belongs to; CICS — the transaction middleware running bank teller systems, where migrating it means translating transaction semantics too. None of that is in the test. OpenAI itself cautions that production migrations still need expert review and testing against business rules. The honest reading of 97.9% is: "near-perfect on a carefully cleaned exam," not "ready to take over your bank's core ledger tomorrow."

Fifth, "looks right but computes differently" is the most dangerous failure mode. Factory.ai's Legacy-Bench ran real COBOL tasks — bug fixes in multi-program codebases — and found models producing silent wrong outputs: a payroll program passing 24 of 29 tests while computing incorrect deductions, with no error and no compilation failure. Of 44 tasks that no model solved at all, 31 were COBOL tasks. COBOL-Coder, a domain-adapted model fine-tuned on Qwen2.5-Coder and published on arXiv in April 2026, honestly admitted in its paper that the evaluation covers single-file programs only — cross-file dependencies, copybooks, and JCL untested. In other words, even a model fine-tuned specifically for COBOL hasn't dared touch the dirty work of real systems. OpenAI's 97.9% is a real step forward — just don't mistake it for the finish line. The finish line is on the copybook-and-JCL side.

Three practical takeaways

If you actually maintain legacy systems or are evaluating AI for a migration project, these matter more than the scores:

• Have the AI explain before it rewrites. Feed it a COBOL or Fortran module and ask for a section-by-section explanation of the business logic first; verify it by hand before any migration talk. A wrong explanation guarantees a wrong rewrite; a correct one gives the rewrite a foundation.
• Write tests before you migrate. The core idea of OpenAI's test — behavioral equivalence as the only standard — applies to your project too: capture the legacy system's key inputs and outputs as a test suite first, then compare item by item after migration. Migrating without tests is driving blindfolded.
• Watch the decimal points. The COBOL fixed-point trap is no isolated case — in financial, tax, and billing logic, "runs fine but computes differently" is the most common accident category. When having AI rewrite such code, list rounding rules, precision, and edge values as an explicit checklist and verify each one.

One last thought: what strikes me most about this post isn't the 97.9% — it's the timing of OpenAI choosing to do this now. While everyone races on "how fast AI writes new code," someone finally started answering "can AI take over the old world" — the world running our money, our power grids, and our weather forecasts. The second half of AI coding may not be about writing more new apps from scratch, but about safely carrying sixty years of accumulated legacy systems into the next sixty. OpenAI just handed in its exam paper — and published the questions. Now let's see which labs dare to sit the same exam.