Sakana AI researchers test whether AI reviewers can catch planted errors in research papers
Their multi-agent reviewer caught about 73% of errors planted in a paper’s core claims in their tests, far more than earlier systems. It is the authors’ own result, and code is available only on request.
How this story was checked
- ResearchedWith AI assistance · 2 sources
- AI source checkEach sentence matched to its source
- Editor approvalMayank Sahu · 12 Oct, 14:06
- Published12 Oct, 14:06
In short
A paper by Sakana AI researchers, posted on arXiv on 8 October 2026, proposes judging AI peer reviewers by whether they find planted mistakes. Their Multi-Layered Review system caught 73.43% of errors in papers’ core claims, against 14.81% for the best earlier system, according to the authors.
The idea
Most tests of AI peer review check how closely AI reviews match human ones. A new paper from Sakana AI researchers asks a different question: can an AI reviewer find a mistake that has been deliberately planted in a paper?12
The paper, “Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review”, was posted on arXiv on 8 October 2026.1
The benchmark
The authors built a Contradiction Benchmark with 1,164 inserted contradictions across 257 papers from conferences including ACL, CVPR, ICML and NeurIPS, MarkTechPost reports.2
The reviewer
Their system, Multi-Layered Review (MLR), uses three agents built on off-the-shelf Claude models: one reads the appendix, an optional one checks related literature, and a main reviewer reads the paper in three passes.2
Results reported by the authors
According to MarkTechPost’s summary of the paper:2
- With four reviews, MLR caught 73.43% of errors planted in a paper’s core claims, against 14.81% for the best earlier system tested.
- Across all planted errors it caught 40.95%.
- On real ICLR 2025 submissions, its scores correlated 0.586 with human scores; two human reviewers correlate 0.742 with each other.
- Each review cost about $0.47, excluding the literature agent.
These are the authors’ own results. The paper is a preprint and the arXiv abstract itself gives no figures.1
Limits
Planted contradictions are not the same as the subtle problems real reviewers look for. MarkTechPost lists the code as available on request rather than public.2
Disclosure: the system tested uses Claude models. Anawords also uses Claude, made by Anthropic, as a research tool.
Fact ledger
How labels workQuestions readers ask
Can AI replace peer reviewers?
This paper does not claim that. It tests whether AI reviewers can catch planted errors, and reports its own results.
How accurate was the AI reviewer?
The authors report it caught 73.43% of errors planted in core claims and 40.95% of all planted errors.
Is the code public?
MarkTechPost lists it as available on request.