Sakana ai's AI review system detects 73% of core claim errors in research papers

Sakana AI has introduced a novel peer review system for AI research, capable of identifying 73.43% of core-claim errors in academic papers. The system, called Multi-Layered Review (MLR), employs three AI agents to analyze research papers, generating a structured critique that includes strengths, weaknesses, and recommendations. The system was tested against a Contradiction Benchmark, where errors were intentionally planted in 257 research papers from top conferences like ACL, AISTATS, CVPR, and ICML 2025. MLR outperformed existing baselines, catching 73.43% of distance-0 contradictions, compared to 14.81% by the best baseline, AgentReview. The system also achieved a Pearson correlation of 0.586 with human scores on ICLR 2025 submissions, close to the human-to-human benchmark of 0.742. While MLR emphasizes validity and experiments, human reviewers prioritize clarity and novelty. The system costs about $0.47 per review, with a single-prompt variant reducing costs by two-thirds, though detection accuracy drops slightly. All results are detailed in Sakana AI's paper, 'Beyond Imitation'.