Elamur Research⌕
☰
home / papers / 2024 / ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

Paper · 2024 · Ziru Chen, Shijie Chen, Yuting Ning et al.

Summary

Claims that LLM-based agents can automate whole research pipelines are hard to verify. The authors argue that agents should first be tested on the individual steps of a scientific workflow. For this purpose they introduce ScienceAgentBench, which collects 102 tasks from 44 peer-reviewed papers spanning four fields: Psychology & Cognitive Neuroscience, Geographical Information Science, Computational Chemistry, and Bioinformatics.

Each task combines a short instruction, a dataset with a structural preview, optional hints written by domain experts, and a reference solution adapted from the paper's open code. The agent must produce one standalone Python file. Outputs are judged on whether the code executes, whether its results (predictions, figures) meet task-specific criteria, and how much the run costs. Per-task rubrics allow partial credit. Nine subject experts and repeated review rounds check scientific soundness, and two measures reduce pre-training contamination.

Five LLMs tested under three agent frameworks solve only about a third of the tasks at best. Current agents can help scientists draft code, but they are far from conducting research autonomously.

Key points

  • Benchmark scope: 102 tasks from 44 publications across four disciplines, validated by nine subject matter experts. Reference programs were adapted from 31 permissively licensed GitHub repositories, such as deepchem, scanpy, geopandas and NeuroKit.
  • Evaluation setup: five open-weight and proprietary LLMs, each run with direct prompting, OpenHands CodeAct and self-debug, with three attempts per task.
  • Best results: the strongest agent solved 32.4% of tasks unaided and 34.3% when given expert knowledge. OpenAI o1-preview reached 42.2%, but at over ten times the cost of the other models.
  • Cost vs. complexity: Claude-3.5-Sonnet with simple self-debug solved 10.8% more tasks than with OpenHands CodeAct while costing roughly 17× less in API fees. A heavier agent framework is not automatically better.
  • Practical value: adapting a task's program took a trained annotator about 2.5–3 hours, whereas an agent typically produces a usable draft within 10 minutes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The Ohio State UniversityColumbus · United States90 works · 19,112 citations🎓University of Wisconsin–MadisonMadison · United States54 works · 49,089 citations🎓Oregon State UniversityCorvallis · United States11 works · 1,422 citations🎓Oklahoma State UniversityStillwater · United States4 works · 81 citations

Also on the byline: Institute of Geography · Institute of Geography of the Slovak Academy of Sciences.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksDecision Sciences · 249 worksInformation Systems and Management · 52 works

Cite

Cite as Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su and Huan Sun. 2024. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv:2410.05080.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.05080 · arXiv 2410.05080

@misc{chen2024scienceagentbench,
  title = {{ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery}},
  author = {Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su and Huan Sun},
  year = {2024},
  doi = {10.48550/arxiv.2410.05080},
  eprint = {2410.05080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.05080}
}

References

101 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mathijs Andeweg and Tom Kuijpers. Model how land subsidence affects flooding, April 2024. URL https://learn.arcgis.com/en/projects/model-how-land-subsidence-affects-flooding/. ↗
  2. Anthropic. Claude 3.5 sonnet. Jun 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet. ↗
  3. Gordon Bell, Tony Hey, and Alex Szalay. Beyond the data deluge. Science, 323(5919):1297–1298, 2009. doi: 10.1126/science.1170411. URL https://www.science.org/doi/abs/10.1126/science.1170411. — [doi:10.1126/science.1170411](https://doi.org/10.1126/science.1170411) ↗
  4. Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conferenc… find
  5. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624:570–578, 2023. doi: https://doi.org/10.1038/s41586-023-06792-0. ↗
  6. Cédric Bouysset and Sébastien Fiorucci. ProLIF: a library to encode molecular interactions as fingerprints. Journal of Cheminformatics, 13(1):72, 2021. doi: 10.1186/s13321-021-00548-6. — [doi:10.1186/s13321-021-00548-6](https://doi.org/10.1186/s13321-021-00548-6) find
  7. Daniel Brand, Nicolas Riesterer, Hannah Dames, and Marco Ragni. Analyzing the differences in human reasoning viajoint nonnegative matrix factorization. Proceedings of the Annual Meeting of the Cognitive Science Society, 42, 2020. URL https://escholarship.org/uc/item/0br9k22g. ↗
  8. Danila Bredikhin, Ilia Kats, and Oliver Stegle. Muon: multimodal omics analysis framework. Genome Biology, 23(1), 2022. doi: 10.1186/s13059-021-02577-8. — [doi:10.1186/s13059-021-02577-8](https://doi.org/10.1186/s13059-021-02577-8) find
  9. O. J. M. Béquignon, B. J. Bongers, W. Jespers, A. P. IJzerman, B. van der Water, and G. J. P. van Westen. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Journal of Cheminformatics, 15(3), 2023. doi: 10.1186/s13321-022-00672-x. — [doi:10.1186/s13321-022-00672-x](https://doi.org/10.1186/s13321-022-00672-x) find
  10. Longbing Cao. Data science: A comprehensive overview. ACM Comput. Surv., 50(3), jun 2017. ISSN 0360-0300. doi: 10.1145/3076253. URL https://doi.org/10.1145/3076253. — [doi:10.1145/3076253](https://doi.org/10.1145/3076253) ↗
  11. Anne E. Carpenter, Thouis R. Jones, Michael R. Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A. Guertin, Joo Han Chang, Robert A. Lindquist, Jason Moffat, Polina Golland, and David M. Sabatini. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Bi… — [doi:10.1186/gb-2006-7-10-r100](https://doi.org/10.1186/gb-2006-7-10-r100) find
  12. Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL http://osc.edu/ark:/19495/f5s1ph73. ↗
  13. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  14. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=KuPixIqPiq. ↗
  15. Ziru Chen, Michael White, Ray Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for LLM planning? it depends on the discriminator. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum…
  16. Simone Ciuti, Tyler B. Muhlym, Dale G. Paton, Allan D. McDevitt, Marco Musiani, and Mark S. Boyce. Human selection of elk behavioural traits in a landscape of fear. Proceedings of the Royal Society B, 279:4407–4416, 2012. doi: 10.1098/rspb.2012.1483. — [doi:10.1098/rspb.2012.1483](https://doi.org/10.1098/rspb.2012.1483) find
  17. Andrew Dawson. eofs: A library for eof analysis of meteorological, oceanographic, and climate data. Journal of Open Research Software, 4(1), 2016. doi: https://doi.org/10.5334/jors.122. — [doi:10.5334/jors.122](https://doi.org/10.5334/jors.122) ↗
  18. Pierre-Paul De Breuck, Matthew L Evans, and Gian-Marco Rignanese. Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on modnet. Journal of Physics: Condensed Matter, 33(40):404002, jul 2021a. doi: 10.1088/1361-648X/ac1280. URL https://dx.doi.org/10.1088/1361… — [doi:10.1088/1361-648x/ac1280](https://doi.org/10.1088/1361-648x/ac1280) ↗
  19. Pierre-Paul De Breuck, Geoffroy Hautier, and Gian-Marco Rignanese. Materials property prediction for limited datasets enabled by feature selection and joint learning with modnet. npj Computational Materials, 7(1), 2021b. doi: 10.1038/s41524-021-00552-2. — [doi:10.1038/s41524-021-00552-2](https://doi.org/10.1038/s41524-021-00552-2) find
  20. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=kiYqbO3wqw. ↗
  21. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — The Llama 3 Herd of Models
  22. ESRI. Predict deforestation in the amazon rain forest, January 2024a. URL https://learn.arcgis.com/en/projects/predict-deforestation-in-the-amazon-rain-forest/. ↗
  23. ESRI. Assess access to public transit, February 2024b. URL https://learn.arcgis.com/en/projects/assess-access-to-public-transit/. ↗
  24. ESRI. Build a model to connect mountain lion habitat, August 2024c. URL https://learn.arcgis.com/en/projects/build-a-model-to-connect-mountain-lion-habitat/. ↗
  25. ESRI. Assess burn scars with satellite imagery, June 2024d. URL https://learn.arcgis.com/en/projects/assess-burn-scars-with-satellite-imagery/. ↗
  26. Sunny Fleming. Model animal home range, March 2024. URL https://learn.arcgis.com/en/projects/model-animal-home-range/. ↗
  27. Adam Gayoso, Romain Lopez, Galen Xing, Pierre Boyeau, Valeh Valiollah Pour Amiri, Justin Hong, Katherine Wu, Michael Jayasuriya, Edouard Mehlman, Maxime Langevin, Yining Liu, Jules Samaran, Gabriel Misrachi, Achille Nazaret, Oscar Clivio, Chenling Xu, Tal Ashuach, Mariano Gabitto, Mohammad Lotfollah… — [doi:10.1038/s41587-021-01206-w](https://doi.org/10.1038/s41587-021-01206-w) find
  28. David E. Graff, Eugene I. Shakhnovich, and Connor W. Coley. Accelerating high-throughput virtual screening through molecular pool-based active learning. Chemical Science, 12(22):7866–7881, 2021. doi: 10.1039/D0SC06805E. — [doi:10.1039/d0sc06805e](https://doi.org/10.1039/d0sc06805e) find
  29. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. Blade: Benchmarking language model agents for data-driven science, 2024. URL https://arxiv… find
  30. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Comp…
  31. Tony Hey, Stewart Tansley, Kristin Tolle, and Jim Gray. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, October 2009. ISBN 978-0-9825442-0-4. URL https://www.microsoft.com/en-us/research/publication/fourth-paradigm-data-intensive-scientific-discovery/. ↗
  32. Tom Hourigan. NOAA Deep Sea Corals Research and Technology Program, 1 2023. URL https://www.gbif.org/dataset/df8e3fb8-3da7-4104-a866-748f6da20a3c. ↗
  33. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=IkmD3fKBPQ. ↗
  34. Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitnik, Cao Xiao, and Jimeng Sun. Deeppurpose: A deep learning library for drug-target interaction prediction. Bioinformatics, 2020. find
  35. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id=1Fs1LvjYQW. ↗
  36. Ryan Jacobs, Tam Mayeshiba, Ben Afflerbach, Luke Miles, Max Williams, Matthew Turner, Raphael Finkel, and Dane Morgan. The materials simulation toolkit for machine learning (mast-ml): An automated open source toolkit to accelerate data-driven materials research. Computational Materials Science, 176:… — [doi:10.1016/j.commatsci.2020.109544](https://doi.org/10.1016/j.commatsci.2020.109544) find
  37. Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-[in]correct: Llms struggle with discriminating self-generated responses, 2024. URL https://arxiv.org/abs/2404.04298. — [arXiv:2404.04298](https://arxiv.org/abs/2404.04298) ↗
  38. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  39. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žı́dek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub… — Highly accurate protein structure prediction with AlphaFold
  40. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter, 2024. URL https://arxiv.org/abs/2407.01502. ↗
  41. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd…
  42. Eric Krause. Analyze urban heat using kriging, July 2024. URL https://learn.arcgis.com/en/projects/analyze-urban-heat-using-kriging/. ↗
  43. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. BioMistral: A collection of open-source pretrained large language models for medical domains. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computatio…
  44. Zekun Li, Wenxuan Zhou, Yao-Yi Chiang, and Muhao Chen. GeoLM: Empowering language models for geospatially grounded language understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5227–5240, Singa… find
  45. Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. Can gpt-4 replicate empirical software engineering research? Proc. ACM Softw. Eng., 1(FSE), jul 2024. doi: 10.1145/3660767. URL https://doi.org/10.1145/3660767. — [doi:10.1145/3660767](https://doi.org/10.1145/3660767) ↗
  46. Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, and Jiaxuan You. Paper copilot: A self-evolving and efficient llm system for personalized academic assistance, 2024. URL https://arxiv.org/abs/2409.04593. — [arXiv:2409.04593](https://arxiv.org/abs/2409.04593) ↗
  47. Anika Liu, Moritz Walter, Peter Wright, Aleksandra Bartosik, Daniela Dolciami, Abdurrahman Elbasir, Hongbin Yang, and Andreas Bender. Prediction and mechanistic analysis of drug-induced liver injury (dili) based on chemical structure. Biology Direct, 16(6), 2021. doi: 10.1186/s13062-020-00285-0. — [doi:10.1186/s13062-020-00285-0](https://doi.org/10.1186/s13062-020-00285-0) find
  48. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https://arxiv.org/abs/2408.06292. ↗
  49. Zitong Lu and Julie Golomb. Generate your neural signals from mine: individual-to-individual eeg converters. Proceedings of the Annual Meeting of the Cognitive Science Society, 45, 2023. URL https://escholarship.org/uc/item/5xn0885t. ↗
  50. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Position: Data-driven discovery with large generative models. In Forty-first International Conference on Machine Learning, 2024a. URL https://openreview.net/forum?id=5SpjhZNXtt. ↗
  51. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models, 2024b. URL https://arxiv.org/abs/2407.01725. ↗
  52. Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods, 53(4):1689–1696, February 2021. doi: 10.3758/s13428-020-01516-y. — [doi:10.3758/s13428-020-01516-y](https://doi.org/10.3758/s13428-020-01516-y) find
  53. Mariia Matveieva and Pavel Polishchuk. Benchmarks for interpretation of qsar models. Journal of Cheminformatics, 41(13), 2021. doi: 10.1186/s13321-021-00519-x. — [doi:10.1186/s13321-021-00519-x](https://doi.org/10.1186/s13321-021-00519-x) find
  54. F. Maussion, A. Butenko, N. Champollion, M. Dusch, J. Eis, K. Fourteau, P. Gregor, A. H. Jarosch, J. Landmann, F. Oesterle, B. Recinos, T. Rothenpieler, A. Vlug, C. T. Wild, and B. Marzeion. The open global glacier model (oggm) v1.1. Geoscientific Model Development, 12(3):909– 931, 2019. doi: 10.519… — [doi:10.5194/gmd-12-909-2019](https://doi.org/10.5194/gmd-12-909-2019) find
  55. Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materia… — [doi:10.1016/j.commatsci.2012.10.028](https://doi.org/10.1016/j.commatsci.2012.10.028) find
  56. OpenAI. Hello gpt-4o. May 2024a. URL https://openai.com/index/hello-gpt-4o. ↗
  57. OpenAI. Introducing openai o1-preview. Sep 2024b. URL https://openai.com/index/introducing-openai-o1-preview/. ↗
  58. Bharath Ramsundar, Bowen Liu, Zhenqin Wu, Andreas Verras, Matthew Tudor, Robert P. Sheridan, and Vijay Pande. Is multitask deep learning practical for pharma? Journal of Chemical Information and Modeling, 57(8):2068–2076, 2017. doi: 10.1021/acs.jcim.7b00146. URL https://doi.org/10.1021/acs.jcim.7b00… — [doi:10.1021/acs.jcim.7b00146](https://doi.org/10.1021/acs.jcim.7b00146) ↗
  59. Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O’Reilly Media, 2019. https://www.amazon.com/Deep-Learning-Life-Sciences-Microscopy/dp/1492039837. ↗
  60. Sebastian Raschka, Anne M. Scott, Mar Huertas, Weiming Li, and Leslie A. Kuhn. Automated Inference of Chemical Discriminants of Biological Activity, pp. 307–338. Springer New York, New York, NY, 2018. ISBN 978-1-4939-7756-7. doi: 10.1007/978-1-4939-7756-7 16. URL https://doi.org/10.1007/978-1-4939-7… — [doi:10.1007/978-1-4939-7756-7](https://doi.org/10.1007/978-1-4939-7756-7) ↗
  61. Robert Richer, Arne Küderle, Martin Ullrich, Nicolas Rohleder, and Bjoern M. Eskofier. Biopsykit: A python package for the analysis of biopsychological data. Journal of Open Source Software, 6 (66):3702, 2021. doi: 10.21105/joss.03702. URL https://doi.org/10.21105/joss.03702. — [doi:10.21105/joss.03702](https://doi.org/10.21105/joss.03702) ↗
  62. URL https://mistral.ai/news/Nicolas Riesterer, Daniel Brand, Hannah Dames, and Marco Ragni. Modeling human syllogistic reasoning:the role of “no valid conclusion”. Proceedings of the Annual Meeting of the Cognitive Science Society, 41, 2019. URL https://escholarship.org/uc/item/5xm1m8h8. ↗
  63. AShlee Robinson. Chart coral and sponge distribution factors with python, October 2023. URL https://learn.arcgis.com/en/projects/chart-coral-and-sponge-distribution-factors-with-python/. ↗
  64. Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 3 edition, 2010. find
  65. Jason Schatz and Christopher J Kucharik. Urban climate effects on extreme temperatures in madison, wisconsin, usa. Environmental Research Letters, 10(9):094024, 2015. find
  66. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://ope…
  67. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URL https://openreview.net/forum?id=ZUbraGNpAq. ↗
  68. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv.org/abs/2409.04109. ↗
  69. Gregor Sturm, Tamas Szabo, Georgios Fotakis, Marlene Haider, Dietmar Rieder, Zlatko Trajanoski, and Francesca Finotello. Scirpy: a Scanpy extension for analyzing single-cell T-cell receptor-sequencing data. Bioinformatics, 36(18):4817–4818, 07 2020. ISSN 1367-4803. doi: 10.1093/bioinformatics/btaa61… — [doi:10.1093/bioinformatics/btaa611](https://doi.org/10.1093/bioinformatics/btaa611) find
  70. Kyle Swanson, Parker Walther, Jeremy Leitz, Souhrid Mukherjee, Joseph C Wu, Rabindra V Shivnaraine, and James Zou. ADMET-AI: a machine learning ADMET platform for evaluation of large-scale chemical libraries. Bioinformatics, 40(7):btae416, 06 2024. ISSN 1367-4811. doi: 10.1093/bioinformatics/btae416… — [doi:10.1093/bioinformatics/btae416](https://doi.org/10.1093/bioinformatics/btae416) find
  71. Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, Arman Cohan, Zhiyong Lu, and Mark Gerstein. Prioritizing safeguarding over autonomy: Risks of llm agents for science, 2024a. URL https://arxiv.org/abs/2402.04247. — [arXiv:2402.04247](https://arxiv.org/abs/2402.04247) ↗
  72. Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gers… — ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
  73. Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40 (Supplement 1):i266–i276, 2024c. ISSN 1367-4811. doi: 10.1093/bioinformatics/btae230. URL https://doi.org/10.1093… find
  74. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — SciCode: A Research Coding Benchmark Curated by Scientists
  75. Isaac Virshup, Danila Bredikhin, Lukas Heumos, Giovanni Palla, Gregor Sturm, Adam Gayoso, Ilia Kats, Mikaela Koutrouli, Philipp Angerer, Volker Bergen, Pierre Boyeau, Maren Büttner, Gokcen Eraslan, David Fischer, Max Frank, Justin Hong, Michal Klein, Marius Lange, Romain Lopez, Mohammad Lotfollahi,… — [doi:10.1038/s41587-023-01733-8](https://doi.org/10.1038/s41587-023-01733-8) find
  76. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for…
  77. Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. LLMs in the imaginarium: Tool learning through simulated trial and error. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  78. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsund… find
  79. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id=jJ9BoXAfFa. ↗
  80. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  81. Logan Ward, Alexander Dunn, Alireza Faghaninia, Nils E.R. Zimmermann, Saurabh Bajaj, Qi Wang, Joseph Montoya, Jiming Chen, Kyle Bystrom, Maxwell Dylla, Kyle Chard, Mark Asta, Kristin A. Persson, G. Jeffrey Snyder, Ian Foster, and Anubhav Jain. Matminer: An open source toolkit for materials data mini… — [doi:10.1016/j.commatsci.2018.05.018](https://doi.org/10.1016/j.commatsci.2018.05.018) find
  82. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Process…
  83. F. Alexander Wolf, Philipp Angerer, and Fabian J. Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome Biology, 19(15), 2018. doi: 10.1186/s13059-017-1382-0. — [doi:10.1186/s13059-017-1382-0](https://doi.org/10.1186/s13059-017-1382-0) find
  84. Felix Wong, Erica J. Zheng, Jacqueline A. Valeri, Nina M. Donghia, Melis N. Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos-Ruiz, Aarti Krishnan, Wengong Jin, Abigail L. Manson, Jens Friedrichs, Ralf Helbig, Behnoush Hajian, Dawid K. Fiejtek, Florence F. Wagner, Holly H. Sout-ter, Ashlee M. Earl… — [doi:10.1038/s41586-023-06887-8](https://doi.org/10.1038/s41586-023-06887-8) find
  85. Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots, 2024. URL https://arxiv.org/abs/2405.07990. ↗
  86. Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., 9:513–530, 2018. doi: 10.1039/C7SC02664A. URL http://dx.doi.org/10.1039/C7SC02664A. — [doi:10.1039/c7sc02664a](https://doi.org/10.1039/c7sc02664a) ↗
  87. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents, 2024. URL https://arxiv.org/abs/2407.01489. ↗
  88. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software engineering language models, 2024a. find
  89. Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. Matplotagent: Method and evaluation for llm-based agentic scientific data visualization, 2024b. URL https://arxiv.org/abs/2402.11453. ↗
  90. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. ↗
  91. Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. LlaSMol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=lY6XTF9tPv. ↗
  92. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=yLClGs770I. ↗
  93. Paul Zandbergen. Run geoprocessing tools with python, March 2024. URL https://learn.arcgis.com/en/projects/run-geoprocessing-tools-with-python/. ↗
  94. Amanda J Zellmer and Barbara S Goto. Urban wildlife corridors: Building bridges for wildlife and people. Frontiers in Sustainable Cities, 4:954089, 2022. find
  95. Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery, 2024. URL https://arxiv.org/abs/2406.10833. ↗
  96. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Mac…
  97. Zhiling Zheng, Oufan Zhang, Ha L. Nguyen, Nakul Rampal, Ali H. Alawadhi, Zichao Rong, Teresa Head-Gordon, Christian Borgs, Jennifer T. Chayes, and Omar M. Yaghi. Chatgpt research group for optimizing the crystallinity of mofs and cofs. ACS Central Science, 9(11): 2161–2170, 2023. doi: 10.1021/acscen… — [doi:10.1021/acscentsci.3c01087](https://doi.org/10.1021/acscentsci.3c01087) find
  98. Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. CodeBERTScore: Evaluating code generation with pretrained models of code. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13921–13937, Singapore,… find
  99. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 202…
  100. Abubakr Ziedan, Cassidy Crossland, Candace Brakewood, Philip Pugliese, and Harrison Ooi. Investigating the preferences of local residents toward a proposed bus network redesign in chattanooga, tennessee. Transportation Research Record, 2675(10):825–840, 2021. find
  101. A PPENDICES We provide more details omitted in the main text as follows: • Appendix A: Limitations and Future Directions • Appendix B: Related Work • Appendix C: Example Task Instructions – Table C.1: Example Instructions for Bioinformatics and Computational Chemistry Tasks – Table C.2: Example Inst… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2024-scienceagentbench-rigorous-assessment-language/