Opinion: Legacy code may be where AI coding helps most, and open tests will build the trust it needs
OpenAI’s COBOL results are encouraging for banks and governments running old systems. Publishing MultiLangBench was a good step; independent checks are the next one.
Illustration · not a photoHow this story was checked
- ResearchedWith AI assistance · 2 sources
- AI source checkEach sentence matched to its source
- Editor approvalMayank Sahu · 20 Oct, 14:01
- Published20 Oct, 14:01
Verify this story
- 2 sources, 1 official · 2 with an archived copy · see sources
- Every fact has a label and a source number · fact ledger
- What we searched, read and could not verify · research note
- Machine-readable: facts as JSON · text as Markdown
How this was made: researched, drafted and source-checked by AI (Claude, made by Anthropic), then published under the editor’s standing approval. A person did not read it line by line before it went live. AI policy
In short
We think legacy systems are one of the most useful places for AI coding tools. OpenAI’s reported jump in COBOL migration accuracy, from 5.4% to 86.6% at low effort in its own test, is encouraging. Releasing MultiLangBench publicly was a welcome step, and independent results will help organisations trust these tools with critical code.
Why legacy code matters
Many banks, insurers and government departments still run core systems in COBOL and Fortran. The people who wrote that code are retiring, and moving it to newer languages is slow and risky. That is why OpenAI’s new results caught our attention.
In OpenAI’s own tests, GPT-6.1 Sol migrated COBOL programs correctly 86.6% of the time at low reasoning effort and 97.9% at higher effort, up from 5.4% for GPT-5.6 Sol.12 GPT-6 Astra averaged 87.4% on a 20-language test of building a database engine from scratch.1
Credit where it is due
Two things stand out positively. First, OpenAI chose a strict scoring rule for migrations: a translated program only counts if every hidden test passes.12 That is the right bar for financial code. Second, OpenAI released MultiLangBench’s setup and results on GitHub, so others can check them.1 Open tests help everyone, including other labs.
OpenAI also says real migrations still need expert review.2 We agree, and it is good to see that said clearly.
What we would like to see next
- Independent results. Universities, auditors or other labs running MultiLangBench would make the numbers more useful to buyers.
- A public migration test. The COBOL migration test has not been released.2 Even a smaller public version would help.
- More model comparisons. Results from several AI providers on the same open test would help teams choose.
How teams can try this safely
Start with code that already has strong tests. Migrate small pieces, compare outputs exactly, and keep a person who knows the business rules in the loop. A rounding change that looks harmless can change financial results.2
Our view
AI will not replace the experts who understand these systems. It may help them move much faster. Open, independent testing is how that promise becomes something organisations can rely on.
Fact ledger
How labels workQuestions readers ask
Is this a news report?
No. This is Anawords’ opinion, based on our news report on OpenAI’s low-resource language results.
Should companies migrate COBOL with AI now?
We suggest starting small with well-tested code, comparing outputs exactly, and keeping experts in the loop.