Tuesday, 20 October 2026
Anawords
Research desk
Research noteResearched 20 Oct 2026 · published 20 Oct 2026, 13:51 IST

Research note: Opinion: Legacy code may be where AI coding helps most, and open tests will build the trust it needs

The facts this opinion relies on, and how we kept it fair to OpenAI and to readers.

Research led by Mayank Sahu
Compiled with AI assistance (Claude by Anthropic)
Read the story →

At a glance

2searches run
2sources read
2sources used
4facts labelled
Mixedevidence strength · 2 of 4 facts (50%)

How we rate evidence: Some facts are confirmed, others rest on one report or on the company’s own claims. We count the share of known facts labelled Confirmed or Reported: 70% or more is Strong, 40–69% Mixed, below 40% Limited.

From sources to story

How many sources we read, how many we used, and how many are official or archived.

Read2Every source we opened
Used2Relied on for the story
Cited2Linked in the story
Official1From the company or a public body
Archived2Saved on the Internet Archive

How sure we are, fact by fact

  • Confirmed · 2 — An official source says it: lab docs, pricing page, notice.
  • Company claim · 1 — Only the company says it. Not checked by anyone else yet.
  • Unconfirmed · 1 — One report only, shown with its source.

Evidence map

Each row is a fact from the story; each column is a source. A filled dot means that source supports the fact.

FactLabel1OFF2OUT
OpenAI reports GPT-6.1 Sol migrated COBOL correctly 86.6% (low effort) and 97.9% (higher effort).Company claim
A migration counts only if every hidden test passes.Confirmed
MultiLangBench is released on GitHub.Confirmed
The COBOL migration test is not public.Unconfirmed
Question

What we set out to do

Offer a constructive view on what OpenAI’s results could mean for organisations with old systems.

Note

The facts we relied on

  • OpenAI’s reported scores and its strict scoring rule.
  • The GitHub release of MultiLangBench.
  • OpenAI’s own note that expert review is still needed.
Note

How we kept it fair

  • We describe scores as OpenAI’s own measurements.
  • We credit OpenAI for publishing its test before asking for more.
  • Our suggestions apply to any provider, not just OpenAI.
Method

Labels we chose and why

  • Claim for the scores.
  • Confirmed for the scoring rule and release.
Open questions

What we could not verify

  • Independent results on these tests.

Can you help confirm one of these? Send us a source. When we find an answer, we update this note and the story, with the date.

Next steps

How we could improve this

Update this view when independent groups publish MultiLangBench results.

Research timeline

  1. Research20 Oct 2026
  2. Research note published20 Oct 2026, 13:51 IST
  3. Story published20 Oct 2026, 14:01 IST

What we searched

  • OpenAI low-resource programming languages
  • MultiLangBench GitHub

Every source we read

SourceUsed?Why
OpenAI
Archived copy		USEDScores, scoring rule and GitHub release.
CellCog blog
Archived copy		USEDCaveats that match OpenAI’s own statements.

Sources cited in the story

  1. 1OpenAI: Advancing coding in low-resource programming languagesOFFICIAL
  2. 2CellCog blog: OpenAI’s COBOL test, 5.4% to 86.6% in one model jumpOUTLET

Research notes show our working. They are not peer-reviewed papers. Data: facts as JSON · How we report