Elamur Research⌕
☰
home / papers / 2025 / ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

Paper · 2025 · Yujie Liu, Zonglin Yang, Tong Xie et al.

Summary

Scientific hypothesis generation has had no dedicated benchmark for LLMs. ResearchBench addresses this by building on a decomposition from MOOSE-Chem. In that view, a hypothesis comes from combining a research background with one or more inspirations. Discovery can then be split into three sub-tasks: finding inspirations, composing a hypothesis from them, and ranking candidate hypotheses. Solving all three perfectly would amount to solving discovery itself.

The authors collected 1,386 papers from 12 disciplines, all published in 2024 or later in top venues, to limit pretraining contamination. An LLM-based agentic pipeline breaks each paper into its question, background survey, inspirations and main hypothesis. It uses self-refinement and checks that the extracted inspirations are necessary and sufficient. Because the pipeline is automated, the benchmark can be refreshed as model cutoffs advance.

Experiments show LLMs are surprisingly good at the out-of-distribution task of retrieving inspirations, even with hard negatives. They do only moderately at composition and ranking, and ranking suffers from strong order bias. The authors argue that LLMs could serve as "research hypothesis mines."

Key points

  • The corpus has 1,386 papers across 12 disciplines, including Chemistry, Physics, Law, Math and Astronomy, all from 2024 onward.
  • Five experts reviewed 62 sampled papers. Extraction accuracy was 91.9% counting only major issues and 82.3% when minor issues were included.
  • When GPT-4o picks the top 4% of inspiration candidates, a ground-truth inspiration is among them 45.7% of the time.
  • Pairwise ranking is distorted by input order. Llama-3.1-8B gives self-contradictory judgments 91.67% of the time, compared with 19.17% for Claude 3.5 Sonnet.
  • In the error analysis, 56% of retrieval failures came from matching surface title or abstract overlap, and 52% of composition failures came from omitting key elements.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore221 works · 39,953 citations🎓Fudan UniversityShanghai · China213 works · 22,907 citations🎓Nanyang Technological UniversitySingapore · Singapore193 works · 27,739 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China164 works · 13,544 citations🎓Wuhan UniversityWuhan · China52 works · 1,456 citations🎓UNSW SydneySydney · Australia40 works · 1,630 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksInformation Systems · 1,888 works

Cite

Cite as Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria and Dongzhan Zhou. 2025. ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition. arXiv:2503.21248.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.21248 · arXiv 2503.21248

@misc{liu2025researchbench,
  title = {{ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition}},
  author = {Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria and Dongzhan Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2503.21248},
  eprint = {2503.21248},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.21248}
}

References

32 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mathias Benedek, Tanja Könen, and Aljoscha C Neubauer. 2012. Associative abilities underlying creativity. Psychology of Aesthetics, Creativity, and the Arts, 6(3):273. find
  2. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2024. Scienceagentbench: Toward rigorous assessme…
  3. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on…
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  6. Aniketh Garikaparthi, Manasi Patwardhan, Aditya Sanjiv Kanade, Aman Hassan, Lovekesh Vig, and Arman Cohan. 2025. Mir: Methodology inspiration retrieval for scientific research problems. arXiv preprint arXiv:2506.00249. find
  7. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565.
  8. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. 2024. Ideabench: Benchmarking large language models for research idea generation. CoRR, abs/2411.02429. — [arXiv:2411.02429](https://arxiv.org/abs/2411.02429) find
  9. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131.
  10. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  11. Arthur Koestler. 1964. The act of creation. London: Hutchinson. find
  12. Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. 2024. Can large language models unlock novel scientific research ideas? CoRR, abs/2409.06185.
  13. Byung Cheol Lee and Jaeyeon Chung. 2024. An empirical investigation of the impact of chatgpt on creativity. Nature Human Behaviour, pages 1–9. find
  14. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  15. Wanhao Liu, Zonglin Yang, Jue Wang, Lidong Bing, Di Zhang, Dongzhan Zhou, Yuqiang Li, Houqiang Li, Erik Cambria, and Wanli Ouyang. 2025. Moose-chem3: Toward experiment-guided hypothesis ranking via simulated experimental feedback. arXiv preprint arXiv:2505.17873. find
  16. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. Llm4sr: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306.
  17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  18. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2024. Discoverybench: Towards data-driven discovery with large language models. CoRR, abs/2407.01725.
  19. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475.
  20. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2025a. Llm-sr: Scientific equation discovery via programming with large language models. In The Thirteenth International Conference on Learning Representations.
  21. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025b. Llm-srbench: A new benchmark for scientific equation discovery with large language models. arXiv preprint arXiv:2504.10415.
  22. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109.
  23. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. 2025. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848.
  24. Anthropic Team. 2024. The claude 3 model family: Opus, sonnet, haiku. find
  25. Qwen Team et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2:3. find
  26. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. Scimon: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 279– 299.…
  27. Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. 2025. Deepscientist: Advancing frontier-pushing scientific findings progressively. arXiv preprint arXiv:2509.26603.
  28. Zonglin Yang and Lidong Bing. 2026. Moose-star: Unlocking tractable training for scientific discovery by breaking the complexity barrier. arXiv preprint arXiv:2603.03756. find
  29. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13545–13565, Bangkok, Thailand. Association for Comput…
  30. Zonglin Yang, Wanhao Liu, Ben Gao, Yujie Liu, Wei Li, Tong Xie, Lidong Bing, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. 2025a. Moose-chem2: Exploring llm limits in fine-grained scientific hypothesis discovery via hierarchical search. In Advances in Neural Information Processing Systems. find
  31. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. 2025b. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. In Proceedings of the International Conference on Learning Representations (ICLR)…
  32. James Webb Young. 1975. A technique for producing ideas. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-researchbench-benchmarking-llms-scientific/