Elamur Research⌕
☰
home / papers / 2025 / The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

Paper · 2025 · Ziming Luo, Atoosa Kasirzadeh and Nihar B. Shah

Summary

AI scientist systems promise to carry out a full machine-learning research cycle, from forming hypotheses to drafting a manuscript, with no human involvement. Little attention has gone to whether their internal workings meet basic methodological standards. The Carnegie Mellon authors study four ways such pipelines can go wrong: choosing convenient benchmarks, leakage between training and evaluation data, questionable use of metrics, and reporting only the experiments that happened to look good.

Pre-training data could contaminate any existing task, so the authors build a fully synthetic one. They then set up separate controlled conditions for each pitfall, using generated candidate datasets and metrics and randomizing names and orderings.

Testing two open-source systems revealed several problems of varying severity. Most of them would be hard to spot from the final paper alone. An LLM-based auditor caught these issues much more reliably when it could inspect workflow logs and generated code. The authors therefore urge venues that review AI-generated research to require these artifacts.

Key points

  • Systems evaluated: Agent Laboratory and The AI Scientist v2. Code and data are released at github.com/niharshah/AIScientistPitfalls.
  • When choosing benchmarks, the systems favored those with strong prior state-of-the-art results or simply picked the first options listed. Neither looked at test data directly, but both sometimes built their own synthetic datasets or subsampled the provided ones without disclosing it in the paper.
  • The authors found no intentional metric cherry-picking. However, metric choice depended on the order metrics appeared in the prompt, and the systems sometimes replaced user-requested metrics with others.
  • In both systems, the internal reward mechanism that picks which experiments to report could see test-set scores. It preferred runs that did well on the test set even when training or validation results were weak, which is a form of post-hoc selection bias.
  • An auditor given only the paper reached 55% accuracy (F1 0.51). With logs and code, it reached 82% accuracy (F1 0.81). Fisher's exact test confirmed the difference is significant (p = 6.3 × 10⁻⁵).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Social Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Ziming Luo, Atoosa Kasirzadeh and Nihar B. Shah. 2025. The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems. arXiv:2509.08713.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.08713 · arXiv 2509.08713

@misc{luo2025more,
  title = {{The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems}},
  author = {Ziming Luo, Atoosa Kasirzadeh and Nihar B. Shah},
  year = {2025},
  doi = {10.48550/arxiv.2509.08713},
  eprint = {2509.08713},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.08713}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ananya. What counts as plagiarism? AI-generated papers pose new risks. Nature, 644:598–600, 2025. News Feature. find
  2. AutoScience. Meet Carl: The first AI system to produce academically peer-reviewed research. https://www.autoscience.ai/blog/ meet-carl-the-first-ai-system-to-produce-academically-peer-reviewed-research, June 2025. Accessed: 2025-06-15. ↗
  3. Peter V Coveney and Sauro Succi. The wall confronting large language models. arXiv preprint arXiv:2507.19703, 2025. — [arXiv:2507.19703](https://arxiv.org/abs/2507.19703) find
  4. Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Harshit Gupta, Manav Chaudhary, and Vasudeva Varma. Autorev: Automatic peer review system for academic research papers. arXiv preprint arXiv:2505.14376, 2025. — [arXiv:2505.14376](https://arxiv.org/abs/2505.14376) find
  5. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259, 2024.
  6. Federico Echenique and Kevin He. Screening p-hackers: Dissemination noise as bait. Proceedings of the National Academy of Sciences, 121(21):e2400787121, 2024. find
  7. Thorsten Eisenhofer, Erwin Quiring, Jonas Möller, Doreen Riepel, Thorsten Holz, and Konrad Rieck. No more reviewer\# 2: Subverting automatic paper-reviewer assignment using adversarial learning. arXiv preprint arXiv:2303.14443, 2023. — [arXiv:2303.14443](https://arxiv.org/abs/2303.14443) find
  8. Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel G. Rodriques. Robin: A multi-agent system for automating scientific discovery, 2025.
  9. Elizabeth Gibney. Scientists hide messages in papers to game AI peer review. Nature, 2025. find
  10. Tarun Gupta and Danish Pruthi. All that glitters is not novel: Plagiarism in AI generated research. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25721–25738, Vienna, Austria, July 2025. Association for Computational Linguis…
  11. Jhih-Yi Janet Hsieh, Aditi Raghunathan, and Nihar B Shah. Vulnerability of text-matching in ML/AI conference reviewer assignments to collusions. In 34th USENIX Security Symposium (USENIX Security 25), pages 5189–5208, 2025. find
  12. Intology AI. Zochi technical report. https://www.intology.ai/blog/zochi-tech-report, 2025. Accessed: 2025-06-15. ↗
  13. Shashidhar Reddy Javaji, Yupeng Cao, Haohang Li, Yangyang Yu, Nikhil Muralidhar, and Zining Zhu. Can AI validate science? benchmarking llms for accurate scientific claim → evidence reasoning, 2025. find
  14. Ross D. King, Jem Rowland, Stephen G. Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa N. Soldatova, Andrew Sparkes, Kenneth E. Whelan, and Amanda Clare. The automation of science. Science, 324(5923):85–89, 2009. find
  15. Tzu-Ling Lin, Wei-Chih Chen, Teng-Fang Hsiao, Hou-I Liu, Ya-Hsin Yeh, Yu Kai Chan, Wen-Sheng Lien, Po-Yen Kuo, Philip S Yu, and Hong-Han Shuai. Breaking the reviewer: Assessing the vulnerability of large language models in automated peer review under textual adversarial attacks. arXiv preprint arXiv… find
  16. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery, 2024.
  17. Ryan Liu and Nihar B Shah. ReviewerGPT? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
  18. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. LLM4SR: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306, 2025.
  19. Vishisht Rao, Aounon Kumar, Himabindu Lakkaraju, and Nihar B Shah. Detecting LLM-generated peer reviews. arXiv preprint arXiv:2503.15772, 2025. — [arXiv:2503.15772](https://arxiv.org/abs/2503.15772) find
  20. Andrew Sparkes, Wayne Aubrey, Emma Byrne, Amanda Clare, Muhammed N Khan, Maria Liakata, Magdalena Markham, Jem Rowland, Larisa N Soldatova, Kenneth E Whelan, et al. Towards robot scientists for autonomous scientific discovery. Automated experimentation, 2:1– 11, 2010. find
  21. Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM. Extended version available at https://www.cs.cmu.edu/~nihars/ preprints/SurveyPeerReview.pdf, June 2022. ↗
  22. Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonçalo Paulo, Youngjae Yu, et al. When AI co-scientists fail: Spot-a benchmark for automated verification of scientific research. arXiv preprint arXiv:2505.11855, 2025.
  23. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants, 2025.
  24. Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. Mind the blind spots: A focus-level evaluation framework for llm reviews. arXiv preprint arXiv:2502.17086, 2025.
  25. Mike Thelwall and Abdallah Yaghi. Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platforms. Scientometrics, pages 1–23, 2025. find
  26. NovelSeek Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, Zhilong Wang, Jinyao Liu, Runmin Ma, Tianshuo Peng, Peng Ye, Dongzhan Zhou, Shufei Zhang, Xiaosong Wang, Yilan Zhang, Meng Li, Zhongying Tu, Xiangyu Yue, Wangli Ouyang,… find
  27. Xiangru Tang, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, and Zhuosheng Zhang. Risks of AI scientists: Prioritizing safeguarding over autonomy. In Data in Generative Models-The Bad, the Ugly, and the Greats, 2025. find
  28. Richard Van Noorden and Jeffrey M Perkel. AI and science: what 1,600 researchers think. Nature, 621(7980):672–675, 2023. find
  29. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025.
  30. Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2025-more-automate-less-see/