Elamur Research⌕
☰
home / papers / 2025 / From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery

From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery

Paper · 2025 · Tianshi Zheng, Zheye Deng, Hong Ting Tsang et al.

Summary

This survey examines how large language models are changing scientific research. It focuses on how much independence these systems have, not on any one discipline. The authors argue that earlier reviews mostly catalogued domain applications or captured LLM abilities at a single point in time. Those reviews missed the steady shift from narrow helpers toward agents that run whole research pipelines. To frame this shift, the paper organizes prior work along the six classical stages of the scientific method, from defining a problem through iterating on results. It then sorts systems into a three-tier autonomy scheme:

  • Tool: narrow, human-directed tasks.
  • Analyst: goal-driven data modeling with humans framing problems and checking outputs.
  • Scientist: open-ended, multi-stage exploration with little human input.

The result is a conceptual map linking dozens of benchmarks and frameworks to these tiers. It also offers an agenda of open problems. Its value lies in giving researchers a shared vocabulary for comparing systems and judging where autonomous AI-driven science is heading. General-purpose scientific LLMs are deliberately left out of scope.

Key points

  • The scientific method is split into six stages (after Popper and Kuhn): observation and problem definition, hypothesis development, experimentation and data collection, data analysis and interpretation, drawing conclusions, and iteration and refinement.
  • Each autonomy level is described by four features: the LLM's role, the human's role, task scope (explicit, goal-oriented, or open-ended) and workflow complexity.
  • Representative Level 3 systems include The AI Scientist, The AI Scientist-v2, Agent Laboratory, AI-Researcher, Zochi and Carl. Benchmarks such as MLE-Bench, DiscoveryBench, ScienceAgentBench, LLM-SRBench and IdeaBench appear at lower levels.
  • Five future directions are proposed:
    • fully autonomous research cycles
    • robotic lab automation, especially in chemistry
    • continuous self-improvement, e.g. via online reinforcement learning
    • transparency and interpretability
    • ethical governance
  • A curated paper list is maintained at the GitHub repo HKUST-KnowComp/Awesome-LLM-Scientific-Discovery.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China183 works · 27,600 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang and Yangqiu Song. 2025. From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery. arXiv:2505.13259.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.13259 · arXiv 2505.13259

@misc{zheng2025automation,
  title = {{From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery}},
  author = {Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang and Yangqiu Song},
  year = {2025},
  doi = {10.48550/arxiv.2505.13259},
  eprint = {2505.13259},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.13259}
}

References

141 references, 76 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tejumade Afonja, Ivaxi Sheth, Ruta Binkyte, Waqar Hanif, Thomas Ulas, Matthias Becker, and Mario Fritz. 2024. Llm4grn: Discovering causal gene regulatory networks with llms – evaluation through synthetic data generation. Preprint, arXiv:2410.15828. find
  2. Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal. 2024. Litllm: A toolkit for scientific literature review. Preprint, arXiv:2402.01788.
  3. Pegah Ahadian and Qiang Guan. 2024. Ai trustworthy challenges in drug discovery. In Trustworthy Artificial Intelligence for Healthcare, pages 1–12, Cham. Springer Nature Switzerland. find
  4. Autoscience. 2025. Meet carl: The first ai system to produce academically peer-reviewed research. find
  5. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. Researchagent: Iterative research idea generation over scientific literature with large language models. Preprint, arXiv:2404.07738.
  6. Jonas Belouadi, Anne Lauscher, and Steffen Eger. 2024. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. Preprint, arXiv:2310.00367.
  7. Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles, and David Williams-King. 2025. Superintelligent agents pose catastrophic risks: Can scie…
  8. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Autonomous chemical research with large language models. Nature, 624:570–578.
  9. Nicolas Bougie and Narimasa Watanabe. 2024. Generative adversarial reviews: When llms become the critic. Preprint, arXiv:2412.10415. — [arXiv:2412.10415](https://arxiv.org/abs/2412.10415) find
  10. Markus J. Buehler. 2024. Accelerating scientific discovery with generative knowledge extraction, graph-based representation, and multimodal intelligent graph reasoning. Preprint, arXiv:2403.11996. — [arXiv:2403.11996](https://arxiv.org/abs/2403.11996) find
  11. Franck Cappello, Sandeep Madireddy, Robert Underwood, Neil Getty, Nicholas Lee-Ping Chia, Nesar Ramachandra, Josh Nguyen, Murat Keceli, Tanwi Mallick, Zilinghan Li, Marieme Ngom, Chenhui Zhang, Angel Yanguas-Gil, Evan Antoniuk, Bhavya Kailkhura, Minyang Tian, Yufeng Du, Yuan-Sen Ting, Azton Wells, a… — [arXiv:2502.20309](https://arxiv.org/abs/2502.20309) find
  12. Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceeding… find
  13. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M adry. 2025. Mle-bench: Evaluating machine learning agents on machine learning engineering. Preprint, arXiv:2410.07095.
  14. Aonan Chen, Zhaoning Yang, Ruochen Li, Jiachen Liu, Muntasir Chowdhury, Moontae Lee, Dong-Yu Du, and Tao Yu. 2025a. Xtragpt: Context-aware and controllable academic paper revision via human-ai collaboration. Preprint, arXiv:2505.11336. find
  15. Tingting Chen, Srinivas Anumasa, Beibei Lin, Vedant Shah, Anirudh Goyal, and Dianbo Liu. 2025b. Auto-bench: An automated benchmark for scientific discovery in llms. Preprint, arXiv:2502.15224.
  16. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. find
  17. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025c. Scienceagentbench: Toward rigorous assessm…
  18. Junyan Cheng, Peter Clark, and Kyle Richardson. 2025a. Language modeling by language models. Preprint, arXiv:2506.20249.
  19. Ming-Chang Cheng, Yixuan Wang, Yuchi Wang, Yair Schiff, Kevin Bello, J. Zico Kolter, Yacine Jernite, William Yang Wang, Daniel Fried, Tian-Li Yu, and Andrew Gordon Wilson. 2025b. Language modeling by language models. Preprint, arXiv:2506.20249.
  20. Ioana Ciucă, Yuan-Sen Ting, Sandor Kruk, and Kartheik Iyer. 2023. Harnessing the power of adversarial prompting and large language models for robust hypothesis generation in astronomy. Preprint, arXiv:2306.11648. — [arXiv:2306.11648](https://arxiv.org/abs/2306.11648) find
  21. Kourosh Darvish, Marta Skreta, Yuchi Zhao, Naruki Yoshikawa, Sagnik Som, Miroslav Bogdanovic, Yang Cao, Han Hao, Haoping Xu, Alán Aspuru-Guzik, Animesh Garg, and Florian Shkurti. 2025. Organa: A robotic assistant for automated chemistry experimentation and characterization. Preprint, arXiv:2401.0694… — [arXiv:2401.06949](https://arxiv.org/abs/2401.06949) find
  22. Data Intelligence Lab. 2025. Ai-researcher: Fully-automated scientific discovery with llm agents. https://github.com/HKUDS/AI-Researcher. ↗
  23. Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. 2024a. Tables as texts or images: Evaluating the table reasoning ability of llms and mllms. Preprint, arXiv:2402.12424. find
  24. Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, and Yangqiu Song. 2024b. Text-tuple-table: Towards information integration in text-to-table generation via global tuple extraction. Preprint, arXiv:2404.14215. find
  25. Fabio Dennstädt, Johannes Zink, Paul Martin Putora, Janna Hastings, and Nikola Cihoric. 2024. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Systematic Reviews, 13:158. find
  26. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, and 21 others. 2024. Llms assist nlp resea…
  27. Haoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang, Su Zhou, Anirudh Dagar, Jiani Zhang, Ali Caner Turkmen, Cuixiong Hu, Huzefa Rangwala, Ying Nian Wu, Bernie Wang, and George Karypis. 2025. Mlzero: A multi-agent system for end-to-end machine learning automation. Preprint, arXiv:2505.13941. — [arXiv:2505.13941](https://arxiv.org/abs/2505.13941) find
  28. Tao Feng, Yihang Sun, and Jiaxuan You. 2025. Grapheval: A lightweight graph-based llm framework for idea evaluation. Preprint, arXiv:2503.12600.
  29. Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. 2025. Budgetmlagent: A cost-effective llm multi-agent system for automating machine learning tasks. Preprint, arXiv:2411.07464. — [arXiv:2411.07464](https://arxiv.org/abs/2411.07464) find
  30. Muhan Gao, Jash Shah, Weiqi Wang, and Daniel Khashabi. 2025. Science hierarchography: Hierarchical organization of science literature. Preprint, arXiv:2504.13834. — [arXiv:2504.13834](https://arxiv.org/abs/2504.13834) find
  31. Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. 2024. Empowering biomedical discovery with ai agents. Cell, 187(22):6125–6151.
  32. A. Ghafarollahi and M. J. Buehler. 2024a. Protagents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning. Preprint, arXiv:2402.04268. — [arXiv:2402.04268](https://arxiv.org/abs/2402.04268) find
  33. Alireza Ghafarollahi and Markus J. Buehler. 2024b. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning. Preprint, arXiv:2409.05556.
  34. Google. 2025. Gemini deep research - your personal research assistant. find
  35. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, and 15 others. 202… — Accelerating scientific discovery with Co-Scientist
  36. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. 2024. Blade: Benchmarking language model agents for data-driven science. Preprint, arXiv:2… find
  37. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. 2024a. Ideabench: Benchmarking large language models for research idea generation. Preprint, arXiv:2411.02429. — [arXiv:2411.02429](https://arxiv.org/abs/2411.02429) find
  38. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024b. Ds-agent: Automated data science by empowering large language models with case-based reasoning. Preprint, arXiv:2402.17453.
  39. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. Preprint, arXiv:2401.13919.
  40. Jiyan He, Weitao Feng, Yaosen Min, Jingwei Yi, Kunsheng Tang, Shuai Li, Jie Zhang, Kejiang Chen, Wenbo Zhou, Xing Xie, Weiming Zhang, Nenghai Yu, and Shuxin Zheng. 2023. Control risk for potential misuse of artificial intelligence in science. Preprint, arXiv:2312.06632. — [arXiv:2312.06632](https://arxiv.org/abs/2312.06632) find
  41. Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. 2024a. Nova: An iterative planning and search approach to enhance novelty and diversity of llm generated ideas. Preprint, arXiv:2410.14255.
  42. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. 2024b. Infiagent-dabench: Evaluating agents on data analysis tasks. Preprint, arXiv:2401.05507. — [arXiv:2401.05507](https://arxiv.org/abs/2401.05507) find
  43. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  44. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024a. Mlagentbench: Evaluating language agents on machine learning experimentation. Preprint, arXiv:2310.03302.
  45. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024b. Understanding the planning of llm agents: A survey. Preprint, arXiv:2402.02716.
  46. IntologyAI. 2025. Zochi technical report: The first artificial scientist. find
  47. Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. 2024. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Preprint, arXiv:2406.06769.
  48. Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld, and Peter Clark. 2025. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation. Preprint, arXiv:2503.22708.
  49. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024a. A survey on large language models for code generation. Preprint, arXiv:2406.00515.
  50. Peiwen Jiang, Xinbo Lin, Zibo Zhao, Ruhui Ma, Yvonne Jie Chen, and Jinhua Cheng. 2024b. TKGT: Redefinition and a new way of text-to-table tasks based on real world demands and knowledge graphs augmented LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,… find
  51. Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. 2025. Aide: Ai-driven exploration in the space of code. Preprint, arXiv:2502.13138.
  52. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2024. Dsbench: How far are data science agents from becoming data science experts? Preprint, arXiv:2409.07703.
  53. Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and François Charton. 2022. End-to-end symbolic regression with transformers. Preprint, arXiv:2204.10532.
  54. Byeonghwi Kim, Minhyuk Seo, and Jonghyun Choi. 2024. Online continual learning for interactive instruction following agents. Preprint, arXiv:2403.07548. — [arXiv:2403.07548](https://arxiv.org/abs/2403.07548) find
  55. Nolan Koblischke, Hyunseok Jang, Kristen Menou, and Mohamad Ali-Dib. 2025. Gravity-bench-v1: A benchmark on gravitational physics discovery for agents. Preprint, arXiv:2501.18411. — [arXiv:2501.18411](https://arxiv.org/abs/2501.18411) find
  56. Patrick Tser Jern Kon, Jiachen Liu, Qingyun Ding, Yuxin Qiu, Zhaoning Yang, Yufan Huang, Jer-Shannassa, Moontae Lee, Muntasir Chowdhury, and Aonan Chen. 2025. Curie: Toward rigorous and automated scientific experimentation with ai agents. Preprint, arXiv:2502.16069.
  57. Thomas Samuel Kuhn. 1962. The Structure of Scientific Revolutions. University of Chicago Press, Chicago. find
  58. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. Preprint, arXiv:2211.11501. find
  59. Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. 2023. Paperqa: Retrieval-augmented generative agent for scientific research.
  60. Junyi Li, Yongqiang Chen, Chenxi Liu, Qianyi Cai, Tongliang Liu, Bo Han, Kun Zhang, and Hui Xiong. 2025. Can large language models help experimental design for causal discovery? Preprint, arXiv:2503.01139. — [arXiv:2503.01139](https://arxiv.org/abs/2503.01139) find
  61. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. 2024a. Chain of ideas: Revolutionizing research via novel idea development with llm agents. Preprint, arXiv:2410.13185.
  62. Michael Y. Li, Emily B. Fox, and Noah D. Goodman. 2024b. Automated statistical model discovery with language models. Preprint, arXiv:2402.17879.
  63. Ruochen Li, Liqiang Jing, and Xinya Du. 2024c. Learning to generate research idea with dynamic control. Preprint, arXiv:2412.14626. — [arXiv:2412.14626](https://arxiv.org/abs/2412.14626) find
  64. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024d. Mlr-copilot: Autonomous machine learning research based on large language models agents. Preprint, arXiv:2408.14033.
  65. Shimin Li, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. 2024e. Agent alignment in evolving social norms. Preprint, arXiv:2401.04620. — [arXiv:2401.04620](https://arxiv.org/abs/2401.04620) find
  66. Haowei Lin, Yacine Jernite, H. T. Kung, and Andrew Gordon Wilson. 2025. Evosld: Automated neural scaling law discovery with large language models. Preprint, arXiv:2507.21184. — [arXiv:2507.21184](https://arxiv.org/abs/2507.21184) find
  67. Chengwei Liu, Chong Wang, Jiayue Cao, Jingquan Ge, Kun Wang, Lvye Zhang, Ming-Ming Cheng, Penghai Zhao, Tianlin Li, Xiaojun Jia, Xiang Li, Xinfeng Li, Yang Liu, Yebo Feng, Yihao Huang, Yijia Xu, Yuqiang Sun, Zhenhong Zhou, and Zhengzi Xu. 2025a. A vision for auto research with llm agents. Preprint,… — [arXiv:2504.18765](https://arxiv.org/abs/2504.18765) find
  68. Ryan Liu and Nihar B. Shah. 2023. Reviewergpt? an exploratory study on using large language models for paper reviewing. Preprint, arXiv:2306.00622.
  69. Sizhe Liu, Yizhou Lu, Siyu Chen, Xiyang Hu, Jieyu Zhao, Yingzhou Lu, and Yue Zhao. 2025b. Drugagent: Automating ai-aided drug discovery programming through llm multi-agent collaboration. Preprint, arXiv:2411.15692.
  70. Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. 2025c. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition. Preprint, arXiv:2503.21248.
  71. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. Preprint, arXiv:2408.06292.
  72. Yi Luo, Linghang Shi, Yihao Li, Aobo Zhuang, Yeyun Gong, Ling Liu, and Chen Lin. 2024. From intention to implementation: Automating biomedical research via llms. Preprint, arXiv:2412.09429. — [arXiv:2412.09429](https://arxiv.org/abs/2412.09429) find
  73. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. Llm4sr: A survey on large language models for scientific research. Preprint, arXiv:2501.04306.
  74. Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. 2023. Clin: A continually learning language agent for rapid task adaptation and generalization. Preprint, arXiv:2310.10134.
  75. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2024. Discoverybench: Towards data-driven discovery with large language models. Preprint, arXiv:2407.01725.
  76. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. Preprint, arXiv:2203.10244.
  77. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. 20… — MLGym: A New Framework and Benchmark for Advancing AI Research Agents
  78. Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S. Weld, Joseph Chee Chang, and Kyle Lo. 2024. Arxivdigestables: Synthesizing scientific literature into tables using language models. Preprint, arXiv:2410.22360. find
  79. Thomas O’Brien, Joel Stremmel, Léo Pio-Lopez, Patrick McMillen, Cody Rasmussen-Ivey, and Michael Levin. 2024. Machine learning for hypothesis generation in biology and medicine: exploring the latent space of neuroscience and developmental bioelectricity. Digital Discovery, 3:249–263. find
  80. Odhran O’Donoghue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Essa Ghareeb, Justin Booth, and Samuel G Rodriques. 2023. Bioplanner: Automatic evaluation of llms on protocol planning in biology. Preprint, arXiv:2310.10632. find
  81. Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. 2025. Sparks of science: Hypothesis generation using structured paper data. Preprint, arXiv:2504.12976.
  82. OpenAI. 2024. Introducing openai o1 preview. find
  83. OpenAI. 2025. Introducing deep research. find
  84. Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, and Benjamin Van Durme. 2025. Claimcheck: How grounded are llm critiques of… find
  85. Karl R. Popper. 1935. The Logic of Scientific Discovery. Routledge, London, England. find
  86. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. Preprint, arXiv:2311.05965.
  87. Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. 2024. Large language models as biomedical hypothesis generators: A comprehensive evaluation. Preprint, arXiv:2407.08940.
  88. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models. Preprint, arXiv:2401.03601.
  89. Yansheng Qiu, Haoquan Zhang, Zhaopan Xu, Ming Li, Diping Song, Zheng Wang, and Kaipeng Zhang. 2025. Ai idea bench 2025: Ai research idea generation benchmark. Preprint, arXiv:2504.14191. — [arXiv:2504.14191](https://arxiv.org/abs/2504.14191) find
  90. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. 2025. Tool learning with large language models: a survey. Frontiers of Computer Science, 19(8).
  91. Xin Quan, Marco Valentino, Louise A. Dennis, and André Freitas. 2024. Verification and refinement of natural language explanations through llm-symbolic theorem proving. Preprint, arXiv:2405.01379.
  92. Gollam Rabby, Diyana Muhammed, Prasenjit Mitra, and Sören Auer. 2025. Iterative hypothesis generation for scientific discovery with monte carlo nash equilibrium self-refining trees. Preprint, arXiv:2503.19309.
  93. Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S. Weld. 2025. Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination. Preprint, arXiv:2409.14634. — [arXiv:2409.14634](https://arxiv.org/abs/2409.14634) find
  94. Chandan K Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27):28601– 28609. find
  95. Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, and Hao Sun. 2025. Liveideabench: Evaluating llms’ divergent thinking for scientific idea generation with minimal context. Preprint, arXiv:2412.17596.
  96. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using llm agents as research assistants. Preprint, arXiv:2501.04227.
  97. Zhenqian Shen, Yongqi Zhang, Lanning Wei, Huan Zhao, and Quanming Yao. 2024. Automated machine learning: From principles to practices. Preprint, arXiv:1810.13306. — [arXiv:1810.13306](https://arxiv.org/abs/1810.13306) find
  98. Yu-Zhe Shi, Mingchen Liu, Fanxu Meng, Qiao Xu, Zhangqian Bi, Kun He, Lecheng Ruan, and Qining Wang. 2025. Hierarchically encapsulated representation for protocol design in self-driving labs. Preprint, arXiv:2504.03810. — [arXiv:2504.03810](https://arxiv.org/abs/2504.03810) find
  99. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2025a. Llm-sr: Scientific equation discovery via programming with large language models. Preprint, arXiv:2404.18400.
  100. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025b. Llm-srbench: A new benchmark for scientific equation discovery with large language models. Preprint, arXiv:2504.10415.
  101. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. Preprint, arXiv:2409.04109.
  102. Michael Skarlinski, Tyler Nadolski, James Braza, Remo Storni, Mayk Caldas, Ludovico Mitchener, Michaela Hinks, Andrew White, and Sam Rodriques. 2025. Futurehouse platform: Superintelligent ai agents for scientific discovery. find
  103. Tao Song, Man Luo, Linjiang Chen, Yan Huang, Qing Zhu, Daobin Liu, Baicheng Zhang, Gang Zou, Fei Zhang, Weiwei Shang, Jun Jiang, and Yi Luo. 2024. find
  104. A multi-agent-driven robotic ai chemist enabling autonomous chemical research on demand. ChemRxiv. Preprint. find
  105. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. Paperbench: Evaluating ai’s ability to replicate ai research. Preprint, arXiv:2504.01848.
  106. Maojun Sun, Ruijian Han, Binyan Jiang, Houduo Qi, Defeng Sun, Yancheng Yuan, and Jian Huang. 2024. A survey on large language model-based agents for statistics and data science. Preprint, arXiv:2412.14222. — [arXiv:2412.14222](https://arxiv.org/abs/2412.14222) find
  107. Shiro Takagi, Ryutaro Yamauchi, and Wataru Kumagai. 2023. Towards autonomous hypothesis verification via language models with minimal guidance. Preprint, arXiv:2311.09706. — [arXiv:2311.09706](https://arxiv.org/abs/2311.09706) find
  108. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, and 11 others. 2024. Scicode: A research coding benchmark curated by…
  109. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, Dov Te’eni, and Iddo Drori. 2024. Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews. Preprint, arXiv:2408.10365.
  110. Silviu-Marian Udrescu and Max Tegmark. 2020. Ai feynman: a physics-inspired method for symbolic regression. Preprint, arXiv:1905.11481.
  111. Rosni Vasu, Chandrayee Basu, Bhavana Dalvi Mishra, Cristina Sarasua, Peter Clark, and Abraham Bernstein. 2025. HypER: Literature-grounded hypothesis generation and distillation with provenance. Preprint, arXiv:2506.12937.
  112. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024a. Scimon: Scientific inspiration machines optimized for novelty. Preprint, arXiv:2305.14259.
  113. Runxiang Wang, Boxiao Wang, Kai Li, Yifan Zhang, and Jian Cheng. 2025a. Drsr: Llm based scientific equation discovery with dual reasoning from data and experience. Preprint, arXiv:2506.04282.
  114. Weiqi Wang, Jiefu Ou, Yangqiu Song, Benjamin Van Durme, and Daniel Khashabi. 2025b. Can llms generate tabular summaries of science papers? rethinking the evaluation protocol. Preprint, arXiv:2504.10284. — [arXiv:2504.10284](https://arxiv.org/abs/2504.10284) find
  115. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024b. Executable code actions elicit better llm agents. Preprint, arXiv:2402.01030.
  116. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Linyi Yang, Jindong Wang, Cunxiang Wang, Kaijie Zhu, Yiqiao Jin, and Xing Xie. 2024c. Autosurvey: Large language models can automatically write surveys. Preprint, arXiv:2406.10252.
  117. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024d. Chain-of-table: Evolving tables in the reasoning chain for table understanding. Preprint, arXiv:2401.04398.
  118. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024e. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Preprint, arXiv:2406.18521.
  119. Yutai Wen, Yifan Jiang, Zitao Li, Whytnee Wade, J. D. Zamfirescu-Pereira, Xinyun Chen, Yuxin Wen, Yiming Yang, Graham Neubig, Jacob Andreas, and Daniel S. Weld. 2025. Predicting empirical ai research outcomes with language models. Preprint, arXiv:2506.00794. — [arXiv:2506.00794](https://arxiv.org/abs/2506.00794) find
  120. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. Cycleresearcher: Improving automated research via automated review. Preprint, arXiv:2411.00816.
  121. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, and 4 others. 2024. Re-benc…
  122. Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, and Zhoujun Li. 2025. Tablebench: A comprehensive and complex benchmark for table question answering. Preprint, arXiv:2408.09174. xAI. 2025. Grok 3 be… — [arXiv:2408.09174](https://arxiv.org/abs/2408.09174) find
  123. Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Peng Ye, Min Dou, Botian Shi, Junchi Yan, and Yu Qiao. 2025. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. Preprint, arXiv:2402.12185. — [arXiv:2402.12185](https://arxiv.org/abs/2402.12185) find
  124. Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. 2025. Scireplicate-bench: Benchmarking llms in agent-driven algorithmic reproduction from research papers. Preprint, arXiv:2504.00255.
  125. Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. 2024. Improving scientific hypothesis generation with knowledge grounded large language models. Preprint, arXiv:2411.02382.
  126. Baixuan Xu, Chunyang Li, Weiqi Wang, Wei Fan, Tianshi Zheng, Haochen Shi, Tao Fan, Yangqiu Song, and Qiang Yang. 2025a. Towards multi-agent reasoning systems for collaborative expertise delegation: An exploratory design study. Preprint, arXiv:2505.07313. — [arXiv:2505.07313](https://arxiv.org/abs/2505.07313) find
  127. Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, and Yunjun Gao. 2025b. Dagent: A relational database-driven data analysis report generation agent. Preprint, arXiv:2503.13269. — [arXiv:2503.13269](https://arxiv.org/abs/2503.13269) find
  128. Yinggan Xu, Hana Kimlee, Yijia Xiao, and Di Luo. 2025c. Advancing ai-scientist understanding: Making llm think like a physicist with interpretable reasoning. Preprint, arXiv:2504.01911. — [arXiv:2504.01911](https://arxiv.org/abs/2504.01911) find
  129. Eric Xue, Zeyi Huang, Yuyang Ji, and Haohan Wang. 2025. Improve: Iterative model pipeline refinement and optimization leveraging llm agents. Preprint, arXiv:2502.18530. — [arXiv:2502.18530](https://arxiv.org/abs/2502.18530) find
  130. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. Preprint, arXiv:2504.08066.
  131. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. Preprint, arXiv:2309.02726.
  132. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. 2025. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. Preprint, arXiv:2410.07076.
  133. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. 2022. Natural language to code generation in interactive data science notebooks. Preprint, arXiv:2212.09248.
  134. Naruki Yoshikawa, Marta Skreta, Kourosh Darvish, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Andrew Zou Li, Yuchi Zhao, Haoping Xu, Artur Kuramshin, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. 2023. Large language models for chemistry robotics. Autonomous Robots, 47:1057– 10… find
  135. Fatemeh Pesaran Zadeh, Juyeon Kim, Jin-Hwa Kim, and Gunhee Kim. 2025. Text2chart31: Instruction tuning for chart generation with automatic feedback. Preprint, arXiv:2410.04064. find
  136. Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Renjun Xu, Hongyang Chen, Xiaohui Fan, and 2 others. 2025a. Scientific large language models: A survey…
  137. Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. 2024. A comprehensive survey of scientific large language models and their applications in scientific discovery. Preprint, arXiv:2406.10833. find
  138. Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D. Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2025b. Mlrc-bench: Can language agents solve machine learning research challenges? Preprint, arXiv:2504.09702.
  139. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan. 2023. Large language models for scientific synthesis, inference and explanation. Preprint, arXiv:2310.07984. — [arXiv:2310.07984](https://arxiv.org/abs/2310.07984) find
  140. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024a. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340– 9351,…
  141. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024b. Hypothesis generation with large language models. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), page 117–139. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2025-automation-autonomy-survey-large/