---
title: "Sakana AI researchers test whether AI reviewers can catch planted errors in research papers"
url: https://anawords.com/story/sakana-ai-llm-peer-review-contradiction-benchmark
published: 2026-10-12T14:06:35+05:30
updated: 2026-10-12T14:06:35+05:30
section: Research
author: Mayank Sahu
checked: sources checked by AI, approved by editor
version: 1
---

# Sakana AI researchers test whether AI reviewers can catch planted errors in research papers

Their multi-agent reviewer caught about 73% of errors planted in a paper’s core claims in their tests, far more than earlier systems. It is the authors’ own result, and code is available only on request.

**In short:** A paper by Sakana AI researchers, posted on arXiv on 8 October 2026, proposes judging AI peer reviewers by whether they find planted mistakes. Their Multi-Layered Review system caught 73.43% of errors in papers’ core claims, against 14.81% for the best earlier system, according to the authors.

## The idea

Most tests of AI peer review check how closely AI reviews match human ones. A new paper from Sakana AI researchers asks a different question: can an AI reviewer find a mistake that has been deliberately planted in a paper?[^1][^2]

The paper, “Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review”, was posted on arXiv on 8 October 2026.[^1]

## The benchmark

The authors built a Contradiction Benchmark with 1,164 inserted contradictions across 257 papers from conferences including ACL, CVPR, ICML and NeurIPS, MarkTechPost reports.[^2]

## The reviewer

Their system, Multi-Layered Review (MLR), uses three agents built on off-the-shelf Claude models: one reads the appendix, an optional one checks related literature, and a main reviewer reads the paper in three passes.[^2]

## Results reported by the authors

According to MarkTechPost’s summary of the paper:[^2]

- With four reviews, MLR caught 73.43% of errors planted in a paper’s core claims, against 14.81% for the best earlier system tested.
- Across all planted errors it caught 40.95%.
- On real ICLR 2025 submissions, its scores correlated 0.586 with human scores; two human reviewers correlate 0.742 with each other.
- Each review cost about $0.47, excluding the literature agent.

These are the authors’ own results. The paper is a preprint and the arXiv abstract itself gives no figures.[^1]

## Limits

Planted contradictions are not the same as the subtle problems real reviewers look for. MarkTechPost lists the code as available on request rather than public.[^2]

*Disclosure: the system tested uses Claude models. Anawords also uses Claude, made by Anthropic, as a research tool.*

## Fact ledger

- [Confirmed] The paper was posted on arXiv on 8 October 2026. (source 1)
- [Company claim] The benchmark has 1,164 planted contradictions in 257 papers. (source 2)
- [Company claim] MLR caught 73.43% of core-claim errors, against 14.81% for the best earlier system. (source 2)
- [Company claim] Its scores correlated 0.586 with human reviewers’ scores. (source 2)
- [Not known] Whether the results hold on real-world review errors.

## Sources

1. [arXiv: Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review](https://arxiv.org/abs/2610.11087) — official
2. [MarkTechPost: Sakana AI’s LLM peer review system catches 73% of core-claim errors](https://www.marktechpost.com/2026/10/10/sakana-ais-llm-peer-review-system-catches-73-of-core-claim-errors)