Elamur Research⌕
☰
home / papers / 2025 / AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Paper · 2025 · Jonathan Bragg, Mike D'Arcy, Nishant Balepur et al.

Summary

Evaluating AI agents for scientific work is hard to do fairly. Existing benchmark suites rarely give every agent the same tools and information sources, and they seldom factor in inference cost. They also tie tasks to particular agent frameworks, rarely reflect how people actually use such systems, and include few baselines. The authors from Ai2 respond by stating explicit principles for rigorous agent benchmarking and building AstaBench on them. It is a suite of more than 2400 problems that covers the whole research workflow, including literature understanding, coding, data analysis and end-to-end discovery, across several scientific domains. Many tasks come from real requests to the deployed Asta agents. The suite includes the Asta Environment, which provides reproducible, production-grade search over a large corpus of scientific literature. It also includes an evaluation toolkit and a leaderboard that report accuracy alongside cost and tool usage. The authors tested 57 agents from 22 architecture classes. Some sub-skills have improved noticeably, but no agent comes close to solving scientific research assistance. The resource matters because it allows controlled, cost-aware comparisons of new agents over time.

Key points

  • AstaBench has 2400+ problems across multiple domains, and many are inspired by real user queries to Asta agents.
  • The Asta Environment is presented as the first agent environment with controlled, production-grade literature search tools. This separates reasoning ability from differences in information access.
  • The agent-eval toolkit builds on Inspect logs to give time-invariant cost accounting. The AstaBench Leaderboard controls for cost, tool access and openness.
  • The agent-baselines suite offers nine Asta agent classes tuned for science, with Asta CodeScientist and Asta Scholar QA among them, along with generic ReAct-style baselines.
  • The authors evaluated 57 agents in 22 classes, and science research assistance remains largely unsolved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States422 works · 213,539 citations🎓University of Maryland, College ParkCollege Park · United States149 works · 29,191 citations🎓Bar-Ilan UniversityRamat Gan · Israel150 works · 25,428 citations🎓University of ZurichZurich · Switzerland83 works · 12,449 citations🎓University of ArizonaTucson · United States40 works · 10,622 citations🔬Allen InstituteSeattle · United States8 works · 4,370 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Jonathan Bragg, Mike D'Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feldman, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal and Daniel S. Weld. 2025. AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite. arXiv:2510.21652.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.21652 · arXiv 2510.21652

@misc{bragg2025astabench,
  title = {{AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite}},
  author = {Jonathan Bragg, Mike D'Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feldman, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal and Daniel S. Weld},
  year = {2025},
  doi = {10.48550/arxiv.2510.21652},
  eprint = {2510.21652},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.21652}
}

References

65 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. LitSearch: A retrieval benchmark for scientific literature search. In EMNLP, 2024. URL https://aclanthology.org/2024.emnlp-main.840/. ↗
  2. All-Hands-AI. OpenHands agent hub, 2025a. URL https://github.com/All-Hands-AI/ OpenHands/tree/55d204ae1b5581b0e55ebbd6465c7e2211b26765/openhand s/agenthub. Accessed: 2025-08-25. ↗
  3. All-Hands-AI. OpenHands evaluation leaderboard, 2025b. URL https://docs.google.com/spreadsheets/d/1wOUdFCMyY6Nt0AIqF705KN4JKOWgeI4wUGUP60krXXs/ed it?gid=0\#gid=0. Accessed: 2025-08-25. ↗
  4. ArcadiaImpact / UK Government BEIS Team. Inspect Evals Dashboard, 2025. URL https://inspectevalsdashboard-vv8euilv46.streamlit.app/. Accessed: 2025-07-08; site was down on 2025-08-25. ↗
  5. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke S. Zettlemoyer, Graham Neubig, Daniel S. Weld, Doug Downey,… — OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
  6. Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. In EMNLP, 2024. URL https://aclanthology.org/2024.emnl p-main.702. ↗
  7. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mádry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In ICLR, 2025. URL https://openreview.net/fo… ↗
  8. Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. MLR-Bench: Evaluating AI agents on open-ended machine learning research. arXiv:2505.19955, 2025a. URL https://arxiv.org/abs/2505.19955. ↗
  9. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward rigorous assessment of… — ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  10. Junyan Cheng, Peter Clark, and Kyle Richardson. Language modeling by language models. arXiv:2506.20249, 2025. URL https://arxiv.org/abs/2506.20249. ↗
  11. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and E. Voorhees. Overview of the TREC 2020 deep learning track. ArXiv, abs/2102.07662, 2021. URL https://api.se manticscholar.org/CorpusId:212737158. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) ↗
  12. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, 2019. find
  13. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-One: A g… — Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
  14. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  15. Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Findings of NAACL, 2025. URL https://arxiv.org/abs/2406.08446. ↗
  16. Etash Guha, Negin Raoff, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad-Falcon, Caroline Choi, Kushal Arora, Mike Merrill, Yichuan Deng, Ashima Suvarna, Hritik Bansal, Marianna Nezhurina, Reinhard Heckel, Seewong Oh, Tatsunori Hashimoto, Jenia… find
  17. Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for LLM evaluation, 2023. URL https://github.com/hugging face/lighteval/tree/126f908a323a6d36f718076c4748e212d7275cfe. ↗
  18. Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and Weinan E. PaSa: An LLM agent for comprehensive academic paper search. In ACL, 2025. URL https://aclanthology.org/2025.acl-long.572/. ↗
  19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. In ICML, 2024.
  20. Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Daniel S. Weld, and Peter Clark. CodeScientist: End-to-end semi-automated scientific discovery with code-based experimentation. In ACL Findings, 2025.
  21. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. arXiv:2407.01502, 2024. URL https://arxiv.org/abs/2407.01502. ↗
  22. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Franck Stéphane Ndzomga, Kangheng Liu, and Arvind Narayanan. HAL: A holistic agent leaderboard for centralized and reproducible agent evaluation. https://github.com/princeton-pli/hal-harness, 2025. ↗
  23. Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, and Ang Chen. EXP-Bench: Can AI conduct AI research experiments? arXiv:2505.24785, 2025. URL https://arxiv.org/abs/2505.2… ↗
  24. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In ICML, 2023. find
  25. Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB-Bench: Measuring capabilities of language models for biology research. arXiv:2407.10362, 2024.
  26. Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas… find
  27. Zijun Liu, Kai Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang, Zhenhe Zhang, Peng Li, and Yang Liu. AIGS: Generating science from AI-powered automated falsification. ArXiv, abs/2411.11910, 2024. URL https://api.semanticscholar.org/CorpusID:274140961. — [arXiv:2411.11910](https://arxiv.org/abs/2411.11910) ↗
  28. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408.06292, 2024.
  29. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Data-driven discovery with large generative models. ICML, 2024.
  30. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards data-driven discovery with large language models. In ICLR, 2025. URL https://openreview.net/pdf?id… ↗
  31. Microsoft. AutoGen agent implementations, 2024. URL https://github.com/microsoft/autogen/tree/d4dd4a26ca5c9a7e29307cf2efef7ffec9bd23da/python/pa ckages/autogen-ext/src/autogen_ext/agents. Accessed: 2025-08-25. ↗
  32. Jordan Mitchener, Francisco Pineda, Yuxin Ye, Spyros Maniatis, Kenneth Holstein, Kam Dahlquist, James D. Braza, Andrew D. White, and Samuel G. Rodriques. BixBench: a comprehensive benchmark for llm-based agents in computational biology. arXiv:2503.00096, 2025. URL https://arxiv.org/abs/2503.00096. ↗
  33. Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S Weld, Joseph Chee Chang, and Kyle Lo. ArxivDIGESTables: Synthesizing scientific literature into tables using language models. In EMNLP, 2024. URL https://aclantholo gy.org/2024.emnlp-main.538/. find
  34. Vishakh Padmakumar, Joseph Chee Chang, Kyle Lo, Doug Downey, and Aakanksha Naik. Setting the table with intent: Intent-aware schema generation and editing for literature review tables. arXiv:2507.19521, 2025. — [arXiv:2507.19521](https://arxiv.org/abs/2507.19521) find
  35. Pritika Ramu, Aparna Garimella, and Sambaran Bandyopadhyay. Is this a bad table? a closer look at the evaluation of table generation from text. In EMNLP, 2024. URL https://aclantholo gy.org/2024.emnlp-main.1239/. find
  36. Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunis-mäki. ‘smolagents‘: a smol library to build great agentic systems. https://github.com/h uggingface/smolagents, 2025. ↗
  37. Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, and Hao Sun. LiveIdeaBench: Evaluating LLMs’ divergent thinking for scientific idea generation with minimal context. arXiv:2412.17596, 2024. URL https://arxiv.org/abs/2412.17596. ↗
  38. SAgE Team, Princeton University. HAL: Holistic agent leaderboard, 2025. URL https://hal. cs.princeton.edu/\#leaderboards. Accessed: 2025-08-25. find
  39. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM agents as research assistants. In arXiv, volume abs/2501.04227, 2025.
  40. ServiceNow. BrowserGym leaderboard, 2025. URL https://huggingface.co/spaces/ ServiceNow/browsergym-leaderboard. Accessed: 2025-08-25. ↗
  41. Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in writing Wikipedia-like articles from scratch with large language models, 2024. URL https://arxiv.org/abs/2402.14207. ↗
  42. Xiaofeng Shi, Yuduo Li, Qian Kou, Longbin Yu, Jinxin Xie, and Hua Zhou. SPAR: Scholar paper retrieval with llm-based agents for enhanced academic search. arXiv:2507.15245, 2025. URL https://arxiv.org/abs/2507.15245. — [arXiv:2507.15245](https://arxiv.org/abs/2507.15245) ↗
  43. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.
  44. Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. TMLR, 2025-January:1–31, 2025. URL https://tmlr.org/papers/v2 025/01-2025paper.pdf. ↗
  45. Amanpreet Singh, Joseph Chee Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka, Angele Zamarron, Cecile Nguyen, Jena D. Hwang, Jason Dunkleberger, Matt Latzke, Smita R Rao, Jaron Lochner, Rob Evans, Rodney Kinney, Daniel S. Weld, Doug Downey, and Sergey Feldman. Ai2 Scholar QA: Or… find
  46. Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv:2409.13740, 2024. URL https://arxiv.org/abs/2409.13740. Int… ↗
  47. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025.
  48. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous scientific innovation. arXiv:2505.18705, 2025. URL https://arxiv.org/abs/2505.18705. ↗
  49. The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025a. URL https://github.com/laude-institute/terminal-bench. ↗
  50. The Terminal-Bench Team. Terminal-Bench leaderboard, 2025b. URL https://tbench.ai/ leaderboard. Accessed: 2025-08-25. ↗
  51. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — SciCode: A Research Coding Benchmark Curated by Scientists
  52. UK AI Safety Institute and Arcadia Impact and Vector Institute. Inspect Evals: Community-contributed evaluations for inspect ai. https://github.com/UKGovernmentBEIS/ inspect_evals, 2025. Accessed: 2025-08-24. ↗
  53. UK AI Security Institute. Inspect AI: Framework for Large Language Model Evaluations, May 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai. ↗
  54. Rosni Vasu, Chandrayee Basu, Bhavana Dalvi Mishra, Cristina Sarasua, Peter Clark, and Abraham Bernstein. HypER: Literature-grounded hypothesis generation and distillation with provenance, 2025. URL https://arxiv.org/abs/2506.12937. ↗
  55. Vector Institute. Vector evaluation leaderboard, 2025. URL https://huggingface.co/spa ces/vector-institute/eval-leaderboard. Accessed: 2025-08-25. ↗
  56. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023. find
  57. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In ICML, 2024.
  58. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  59. Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. Scireplicate-bench: Benchmarking llms in agent-driven algorithmic reproduction from research papers. Proceedings of COLM, 2025.
  60. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. ResearcherBench: Evaluating deep AI research systems on the frontiers of scientific inquiry. arXiv:2507.16280, 2025. URL https://arxiv.org/abs/2507.16280. — [arXiv:2507.16280](https://arxiv.org/abs/2507.16280) ↗
  61. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Nicolaus Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-level automated scientific discovery via agentic tree search. ArXiv, abs/2504.08066, 2025. URL https://api.semanticscholar.org/CorpusID:277741107. ↗
  62. Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, et al. Lmr-bench: Evaluating llm agent’s ability on reproducing language modeling research. arXiv preprint arXiv:2506.17335, 2025.
  63. Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of LLM-based agents. arXiv:2503.16416, 2025. URL https://arxiv.org/abs/2503.16416. — [arXiv:2503.16416](https://arxiv.org/abs/2503.16416) ↗
  64. Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. Autoreproduce: Automatic ai experiment reproduction with paper lineage. arXiv preprint arXiv:2505.20662, 2025.
  65. Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, and Jiaxuan You. SafeScientist: Toward risk-aware scientific discoveries by LLM agents. arXiv:2505.23559, 2025. URL https://arxiv.org/abs/2505.23559. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bragg-2025-astabench-rigorous-benchmarking-ai-agents/