When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Paper · 2025 · Guijin Son, Jiwoo Hong, Honglu Fan et al.
Summary
Large language models are increasingly promoted as "AI Co-Scientists" that generate hypotheses, code, and manuscripts. Far less is known about whether they can check scientific work for mistakes. The authors address this gap with SPOT (Scientific Paper Error Detection), a multimodal benchmark of 83 papers containing 91 errors serious enough to have triggered an erratum or a retraction.
The papers were drawn from WithdrarXiv and PubPeer and published in 2024 or later, which reduces the risk of training-data contamination. GPT-4o filtered the candidates, and only errors the original authors explicitly acknowledged were kept. Two independent groups of human annotators then audited the selection. Each paper was converted into Markdown text plus images using Llama-Parse, followed by a GPT-4.1 correction pass and a manual check.
When frontier models were tested on SPOT, the best one, o3, reached only about 21% recall and 6% precision. Models also rarely found the same errors across repeated runs. Expert case studies traced the failures to gaps in long-tail domain knowledge and to unstated multi-step calculations. The results suggest current LLMs cannot yet be relied on to verify scientific research.
Key points
- SPOT contains 83 manuscripts with 91 errors across ten fields, including Mathematics, Physics, Biology, Materials Science, and Computer Science. Of these errors, 59 led to errata and 32 to retractions.
- The six error categories are Equation/proof (37), Figure duplication (27), Data inconsistency (18), Statistical reporting (4), Reagent identity (3), and Experiment setup (2).
- Inputs are long: an average of roughly 12,900 tokens (GPT-4o tokenizer) and 17.5 images per paper.
- Pass@1 scores were 18.4% for o3, 7.3% for Gemini 2.5 Pro, and 0.9% for Llama-4-Maverick. Confidence was low, and models seldom rediscovered the same errors across eight runs.
- The metadata is released on Hugging Face as
amphora/SPOT-MetaData.
Fields, topics and concepts
📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksDecision Sciences · 306 worksInformation Systems and Management · 96 worksHistory and Philosophy of Science · 83 works
🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works
Cite
Cite as Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonçalo Paulo, Youngjae Yu and Stella Biderman. 2025. When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research. arXiv:2505.11855.
@misc{son2025when,
title = {{When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research}},
author = {Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonçalo Paulo, Youngjae Yu and Stella Biderman},
year = {2025},
doi = {10.48550/arxiv.2505.11855},
eprint = {2505.11855},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2505.11855}
}References
- Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. find
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al. R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation. arXiv preprint arXiv:2505.02018, 2025. — [arXiv:2505.02018](https://arxiv.org/abs/2505.02018) find
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
- Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models on university-level physics problem solving. arXiv preprint arXiv:2503.21821, 2025. find
- Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.
- Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. Can chatgpt be used to generate scientific hypotheses? Journal of Materiomics, 10(3):578–584, 2024. find
- Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An llm agent for comprehensive academic paper search. arXiv preprint arXiv:2501.10120, 2025.
- Rishab Jain and Aditya Jain. Generative ai in writing research papers: a new type of algorithmic bias and uncertainty in scholarly work. In Intelligent Systems Conference, pages 656–669. Springer, 2024. find
- Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025.
- Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
- José R Penadés, Juraj Gottweis, Lingchen He, Jonasz B Patkowski, Alexander Shurick, Wei-Hung Weng, Tao Tu, Anil Palepu, Artiom Myaskovsky, Annalisa Pawlosky, et al. Ai mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolution. bioRxiv, pages 2025–02, 2… find
- Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525–535, 2024. find
- Haining Pan, Nayantara Mudur, William Taranto, Maria Tikhanovskaya, Subhashini Venugopalan, Yasaman Bahri, Michael P Brenner, and Eun-Ah Kim. Quantum many-body physics calculations with large language models. Communications Physics, 8(1):49, 2025. find
- DeepMind. Alphaevolve: a gemini-powered coding agent for designing advanced algorithms. https://deepmind.google/discover/blog/ alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/, 2025. Accessed: 2025-05-15. ↗
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
- Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019.
- Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. A review on fact extraction and verification. ACM Computing Surveys (CSUR), 55(1):1–35, 2021. find
- Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, and Yunhe Feng. Poly-fever: A multilingual fact verification benchmark for hallucination detection in large language models. arXiv preprint arXiv:2503.16541, 2025. find
- Raúl Ortega and José Manuel Gómez-Pérez. Sciclaims: An end-to-end generative system for biomedical claim analysis. arXiv preprint arXiv:2503.18526, 2025.
- Sujit Kumar, Anshul Sharma, Siddharth Hemant Khincha, Gargi Shroff, Sanasam Ranbir Singh, and Rahul Mishra. Sciclaimhunt: A large dataset for evidence-based scientific claim verification. arXiv preprint arXiv:2502.10003, 2025. — [arXiv:2502.10003](https://arxiv.org/abs/2502.10003) find
- Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363, 2024.
- Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. Nlpeer: A unified resource for the computational study of peer review. arXiv preprint arXiv:2211.06651, 2022.
- Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. Peerqa: A scientific question answering dataset from peer reviews. arXiv preprint arXiv:2502.13668, 2025. find
- Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin… find
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationa…
- David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr…
- Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Moprd: A multidisciplinary open peer review dataset. Neural Computing and Applications, 35(34): 24191–24206, 2023. find
- Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. Automatically evaluating the paper reviewing capability of large language models. arXiv preprint arXiv:2502.17086, 2025. — [arXiv:2502.17086](https://arxiv.org/abs/2502.17086) find
- OpenAI. Openai o3 and o4-mini system card. https://openai.com/index/ o3-o4-mini-system-card/, April 2025. Accessed: 2025-05-12. ↗
- Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/, April 2025. Accessed: 2025-05-12. ↗
- Delip Rao, Jonathan Young, Thomas Dietterich, and Chris Callison-Burch. Withdrarxiv: A large-scale dataset for retraction study. arXiv preprint arXiv:2412.03775, 2024. — [arXiv:2412.03775](https://arxiv.org/abs/2412.03775) find
- José Luis Ortega. Classification and analysis of pubpeer comments: How a web journal club is used. Journal of the Association for Information Science and Technology, 73(5):655–670, 2022. find
- OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex… — GPT-4o System Card
- MAA. American Invitational Mathematics Examination – AIME. In American Invitational Mathematics Examination – AIME 2024, February 2024, February 2024. URL https://maa.org/ math-competitions/american-invitational-mathematics-examination-aime. ↗
- Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934, 2025.
- Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025.
- Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint arXiv:2504.17192, 2025.
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
- OpenAI. tiktoken: A fast bpe tokeniser for use with openai’s models. https://github.com/ openai/tiktoken, 2025. GitHub repository; version 0.9.0 (Feb. 14, 2025); accessed May 12, 2025. ↗
- Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565, 2024.
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
- OpenAI. GPT-4.1. https://openai.com/index/gpt-4-1/, 2025. Accessed: May 15, 2025. ↗
- Google Cloud. Gemini 2.5 pro. https://cloud.google.com/vertex-ai/ generative-ai/docs/models/gemini/2-5-pro, 2025. Accessed: 2025-05-12. ↗
- Google Cloud. Gemini 2.0 Flash Lite. https://cloud.google.com/vertex-ai/ generative-ai/docs/models/gemini/2-0-flash-lite, 2025. Accessed: May 15, 2025. ↗
- Anthropic. Claude 3.7 Sonnet System Card. https://assets.anthropic.com/ m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf, 2025. Accessed: May 15, 2025. ↗
- Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
- Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt…
- Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
- Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datase…
- Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025.
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017. find
- Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. find
- Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. find
- An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
- Dan Petersen and Orsola Tommasi. Multiplicative chow-künneth decomposition and homology splitting of configuration spaces, 2024. URL https://arxiv.org/abs/2401.06455. — [arXiv:2401.06455](https://arxiv.org/abs/2401.06455) ↗
- Xingyao Ye, Ruoyang Liu, Xinyu Mu, Shanshan Tao, Hao Yang, Xuejiao J Gao, Shuo-Wang Yang, and Donglin Jiang. Superacid in situ protected synthesis of covalent organic frameworks. Journal of the American Chemical Society, 2025. find
- Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239, 2024.
- Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017. find
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
- Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. Reviewer2: Optimizing review generation through prompt generation. arXiv preprint arXiv:2402.10886, 2024.
- Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, and Heng Ji. Scientific opinion summarization: Paper meta-review generation dataset, methods, and evaluation. In Artificial Intelligence for Research and Democracy: First International Workshop, AI4Research 2024, and 4th International Wo… find
- Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. arXiv preprint arXiv:2109.09774, 2021. — [arXiv:2109.09774](https://arxiv.org/abs/2109.09774) find
- Tomas Bonavia and Juan A Marin-Garcia. A noise audit of the peer review of a scientific article: a wpom journal case study. WPOM-Working Papers on Operations Management, 14(2): 137–166, 2023. find
- Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024.
- Andy L Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021. — [arXiv:2104.03113](https://arxiv.org/abs/2104.03113) find
- Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning. arXiv preprint arXiv:2502.17407, 2025.
- Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
- Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
- Kabir Ahuja, Melanie Sclar, and Yulia Tsvetkov. Finding flawed fictions: Evaluating complex reasoning in language models via plot hole detection. arXiv preprint arXiv:2504.11900, 2025. — [arXiv:2504.11900](https://arxiv.org/abs/2504.11900) find
- Jeehoon Park, Junyeong Park, and Philsang Yoo. Algebraic description of complex conjugation on cohomology of a smooth projective hypersurface, 2024. URL https://arxiv.org/abs/ 2402.14546. ↗
- Reem Altuijri, A Atta, E Abdeltwab, and MM Abdelhamied. Impacts of low energy argon beam on enhancing the surface wettability and electrical performance of ca/pani films. ECS Journal of Solid State Science and Technology, 13(4):043017, 2024. find
- Michael Simpson. The scientific case against net zero: Falsifying the greenhouse gas hypothesis. Journal of Sustainable Development, 17(6):137–157, 2024. doi: 10.5539/jsd.v17n6p137. URL https://ideas.repec.org/a/ibn/jsd123/v17y2024i6p137.html. — [doi:10.5539/jsd.v17n6p137](https://doi.org/10.5539/jsd.v17n6p137) ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/son-2025-ai-co-scientists-fail/