Research note
Research note: Sakana AI researchers test whether AI reviewers can catch planted errors in research papers
How we checked the Sakana AI peer-review paper, why its numbers are labelled as the authors’ own, and what the test does not show.
What we set out to check
What the paper proposes, how the test works, what results it reports, and what the results do not show.
What we found
- arXiv confirms the paper, its authors and its date.
- The numbers come from the paper as summarised by MarkTechPost; the abstract does not give them.
Numbers that differ
None between the sources. We note that the 73% figure is for errors in core claims only; across all planted errors the figure is about 41%.
Labels we chose and why
- Confirmed for the paper’s existence and date.
- Company claim (here, the authors’ claim) for all results, since no one else has reproduced them.
What we could not verify
- The full paper text directly.
- Independent reproductions, since the code is available only on request.
What we searched
- Sakana AI Multi-Layered Review peer review
- Beyond Imitation LLM-assisted peer review arXiv
- Sakana peer review 73% core claim errors
Every source we read
| Source | Used? | Why |
|---|---|---|
| arXiv | USED | Confirms title, authors and submission date. The abstract gives no figures. |
| MarkTechPost | USED | Summarises the paper’s method and numbers. |
| RIO Rundown newsletter | NOT USED | Repeats MarkTechPost. |