Sakana's AI paper checker caught 73% of planted main-claim errors
Key takeaways
- Sakana AI's three-agent reviewer, MLR, caught 73.43% of contradictions planted in papers' main claims when four reviews were combined, against 35.40% for the best rival system tested.
- Across all 1,164 planted errors the figure falls to 40.95%, and on 211 real withdrawn arXiv papers MLR matched the stated withdrawal reason exactly 16.11% of the time.
- A separate Sakana release, MASS, raised a 27B model's win rate on held-out tasks, but its score per output token slipped on Terminal-Bench 2.0 and SWE-bench Verified.
Last updated: 12 October 2026
73.43% is the share of contradictions planted in papers' main claims that Sakana AI's paper reviewer caught when four of its reviews were combined. A single review caught 60.79%. The best rival system tested, LLM-Review running on Claude Sonnet 4, caught 35.40%, according to the paper Sakana researchers posted to arXiv on 8 October.
The system is called Multi-Layered Review, or MLR. It uses three agents: an Appendix Agent that summarises appendices with Claude Haiku 3.5, a Literature Review Agent on Claude Sonnet 4 that searches the web to judge novelty, and a Review Agent that pulls their work into a single review.
Why a benchmark score is not a verdict
The benchmark was built by planting contradictions into conference papers, 1,164 in all, so the right answer is always known. Main-claim errors are the easiest to spot. Across all 1,164, the four-review ensemble caught 40.95%, against 16.43% for the best rival.
Real papers are harder. On 211 withdrawn arXiv papers, MLR's review matched the stated reason for withdrawal exactly 16.11% of the time, or 26.07% when similar concerns count. That beats the other systems tested, but it is a long way from 73%. Peer review tools earn trust on real manuscripts, and that is the number to judge MLR on.
If you want a sense of how agents divide up a job like this, our explainer on how AI agents work covers the basics.
A second release: MASS
Sakana also published MASS, a self-improvement loop in which one model acts as executor, workflow optimiser and evaluator. The base model is Qwen3.6-27B, and the loop is fine-tuned with LoRA.
On three held-out tasks, its win rate against the base model rose from 53.9% after one round to 69.9% after two. Transfer to public benchmarks was uneven: score per output token was 0.96 times the base on Terminal-Bench 2.0 and 0.94 times on SWE-bench Verified. The authors also warn that shared blind spots in a loop where one model plays every role could let flawed results be accepted and reinforced, which they call a possible Achilles' heel.
Watch for independent tests of MLR on manuscripts nobody has checked yet. The 16% result on real withdrawn papers says more than the 73% does.
