Elamur Research⌕
☰
home / papers / 2025 / Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

Paper · 2025 · Joeran Beel, Min-Yen Kan and Moritz Baumgart

Summary

Sakana.ai has promoted its open-source AI Scientist as a system that can carry out the whole machine-learning research cycle alone: generating ideas, running experiments, writing papers and reviewing them. Until this paper, those claims had not been independently tested. Beel, Kan and Baumgart, writing for the IR community, installed and ran the system themselves. They then examined each stage: idea generation and novelty checking, code execution, and the quality of the resulting manuscripts. They found serious weaknesses. Literature search relies on shallow keyword lookups, experiments often crash or produce misleading conclusions, and the papers contain few, mostly old citations, structural defects and invented numbers. The authors judge the output comparable to a hurried, unmotivated undergraduate. Still, they note that the system produces complete papers very cheaply and with little human effort. They see it as a real step toward what they call Artificial Research Intelligence (ARI). They urge the research community to respond now with pilot projects, competitions and attribution standards.

Key points

  • Novelty checks were unreliable: well-known ideas, such as micro-batching for SGD, were wrongly labelled as novel.
  • Of 12 proposed experiments, 5 (42%) failed because of coding errors. One experiment aimed at energy efficiency reported better accuracy while using more compute, which contradicts its own goal.
  • Code changed very little between iterations, with each round adding only about 8% more characters on average.
  • The generated papers had a median of 5 citations, and only 5 of 34 citations dated from 2020 or later. Problems included missing figures, duplicated sections, placeholder text and hallucinated results.
  • Producing one paper cost about $6–$15 and needed roughly 3.5 hours of human involvement. The authors propose research logs and markup languages as frameworks for attributing AI contributions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Institut Polytechnique de ParisPalaiseau · France16 works · 1,623 citations🔬Laboratoire Traitement et Communication de l’InformationPalaiseau · France9 works · 389 citations🎓Télécom ParisPalaiseau · France12 works · 837 citations🔬Centre Inria de SaclayPalaiseau · France4 works · 245 citations🔬SODA: Méthodes computationnelles et mathématiques pour comprendre la société et la santé à partir de donnéesSaclay · France2 works · 34 citations

Also on the byline: Centre d’Économie et de Décision Publique (CEDP) · Institut Interdisciplinaire de l'Innovation.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksInformation Systems · 1,888 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Joeran Beel, Min-Yen Kan and Moritz Baumgart. 2025. Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?. arXiv:2502.14297 · SIGIR Forum 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3769733.376974 · arXiv 2502.14297

@misc{beel2025evaluating,
  title = {{Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?}},
  author = {Joeran Beel, Min-Yen Kan and Moritz Baumgart},
  year = {2025},
  doi = {10.1145/3769733.376974},
  eprint = {2502.14297},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3769733.376974}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moritz Baumgart, Lukas Wegmeth, Tobias Vente, and Joeran Beel. 2024. e-Fold Cross-Validation for Recommender-System Evaluation. In International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys). https://isg.… ↗
  2. Joeran Beel. 2024. Our use of AI-tools for writing research papers. In Intelligent Systems Group, Blog. https://isg.beel.org/blog/2024/08/19/our-use-of-ai-tools-for-writing-research-papers/ ↗
  3. Joeran Beel, Alan Said, Tobias Vente, and Lukas Wegmeth. 2024. Green Recommender Systems: A Call for Attention. ACM SIGIR Forum 58, 2 (Dec. 2024), 1–5. https://doi.org/10.31219/osf.io/5ru2g — [doi:10.31219/osf.io/5ru2g](https://doi.org/10.31219/osf.io/5ru2g) ↗
  4. Joeran Beel, Lukas Wegmeth, and Tobias Vente. 2024. E-fold Cross-validation: A Computing and Energy-efficient Alternative to K-fold Cross-validation with Adaptive Folds [Proposal]. OSF Preprints (2024). https://doi.org/10.31219/osf.io/exw3j — [doi:10.31219/osf.io/exw3j](https://doi.org/10.31219/osf.io/exw3j) ↗
  5. Matthew Berman. 2024. "AI Scientist" Can Discover New Science! (Self-Improving AI = AGI). YouTube (2024). https://www.youtube.com/watch?v=LTM_3EUwUmE ↗
  6. Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. 2018. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research 18, 223 (2018), 1–42. find
  7. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. 2024. Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. arXiv:2410.13185 (2024). https://ar…
  8. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292 [cs.AI] https://arxiv.org/abs/2408.06292 ↗
  9. Christopher Mahlich, Tobias Vente, and Joeran Beel. 2024. From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation. In International Conference on Artificial Intelligence and Machine Learning Research (CAIMLR). https://isg.beel.org/blog/2024/09/16/e-fold-cross-validation/ ↗
  10. James Manyika. 2024. A new era of discovery. Google Blog (2024). https://blog.google/technology/ai/a-new-era-of-discovery/ ↗
  11. Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S. Weld. 2025. Scideator: Human-LLM Scientific Idea Generation Grounded in Research-Paper Facet Recombination. arXiv:2409.14634 (2025). https://arxiv.org/abs/2409.14634 — [arXiv:2409.14634](https://arxiv.org/abs/2409.14634) ↗
  12. Sakana. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. Sakan.ai Blog (2024). https://sakana.ai/ai-scientist/ ↗
  13. Sakana.AI. 2024. Sakana “AI Scientist”. Sakana.ai Website (2024). https://www.sakanaai.org/sakana-ai-scientist/ ↗
  14. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. arXiv:2409.04109 (2024). https://arxiv.org/abs/2409.04109 ↗
  15. Haoyang Su, Renqi Chen, Shixiang Tang, Xinzhe Zheng, Jingzhe Li, Zhenfei Yin, Wanli Ouyang, and Nanqing Dong. 2024. Two Heads Are Better Than One: A Multi-Agent System Has the Potential to Improve Scientific Idea Generation. arxiv:2410.09403 (2024). https://arxiv.org/abs/2410.09403 — [arXiv:2410.09403](https://arxiv.org/abs/2410.09403) ↗
  16. TheAIGRID. 2024. SakanaAI New AI AGENT Researcher STUNS The ENITRE INDUSTRY! YouTube (2024). https://www.youtube.com/watch?v=hP-IzCZAZDc ↗
  17. Aidan Toner-Rodgers. 2024. Artificial Intelligence, Scientific Discovery, and Product Innovation. arXiv:2412.17866 (2024). https://arxiv.org/abs/2412.17866 ↗
  18. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, Dov Te’eni, and Iddo Drori. 2024. AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews. 2408.10365 (2024). https://arxiv.org/… ↗
  19. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review. arXiv:2412.01708 (2024). https://arxiv.org/abs/2412.01708 ↗
  20. ChengXiang Zhai. 2024. Large Language Models and Future of Information Retrieval: Opportunities and Challenges. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (Washington DC, USA) (SIGIR ’24). Association for Computing Machinery, Ne… — [doi:10.1145/3626772.3657848](https://doi.org/10.1145/3626772.3657848) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beel-2025-evaluating-sakana-s-ai/