Elamur Research⌕
☰
home / papers / 2025 / Can Large Language Models Unlock Novel Scientific Research Ideas?

Can Large Language Models Unlock Novel Scientific Research Ideas?

Paper · 2025 · Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal et al.

Summary

Assessing whether an LLM can suggest worthwhile research directions after reading a paper is difficult. There is no reference answer, and expert review is slow, costly and cannot keep pace with new model releases. The authors build a benchmark of 1,250 recent papers from S2ORC covering Computer Science, Economics, Physics, Chemistry and Medicine. Annotators found each author-stated future research idea (FRI) and removed it from the text. These ideas, grouped by topic, form the AP-FRI corpus. Gemini, Claude-2, GPT-3.5 and GPT-4 then propose FRIs from the cleaned papers. Two automatic metrics are introduced. IAScore measures agreement with the authors' own plans and acts as a lower bound on quality. The Idea Distinctness Index uses cosine similarity to quantify diversity. Experts also rated 660 Computer Science ideas for novelty, relevance and feasibility. Claude and GPT-4 gave more varied and better-aligned ideas. Models often drift toward generic suggestions but still produce a fair share of novel, relevant ones. The metrics offer a scalable complement to expert review.

Key points

  • The dataset has 250 papers per domain, published January 2023 – February 2024 and retrieved through the Semantic Scholar Academic Graph API. Only English papers with full text and an explicit future-work section were kept.
  • Annotators separated Direct FRI sentences, which were deleted entirely, from Mixed FRI sentences, where only the future-oriented parts were cut. All removed material went into the AP-FRI corpus.
  • According to the authors, IAScore correlates strongly with expert judgments. However, it cannot credit good ideas that the original authors never mentioned.
  • Claude and GPT-4 scored higher on the Idea Distinctness Index than Gemini and GPT-3.5. Claude and GPT-4 also matched author-written ideas more closely.
  • To reduce memorization risk, the authors used only recent papers. They also probed the models for paper titles, venues and authors and found no sign of memorization. Dataset and code are released at github.com/sandeep82945/Future-Idea-Generation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology PatnaPatna · India21 works · 374 citations🔬Oak Ridge National LaboratoryOak Ridge · United States3 works · 44 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal and Asif Ekbal. 2025. Can Large Language Models Unlock Novel Scientific Research Ideas?. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1704

@inproceedings{kumar2025can,
  title = {{Can Large Language Models Unlock Novel Scientific Research Ideas?}},
  author = {Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal and Asif Ekbal},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1704},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1704}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,… — Gemini: A Family of Highly Capable Multimodal Models
  2. Anthropic. 2023. Model card for claude 2. find
  3. Mathias Benedek, Emanuel Jauk, Andreas Fink, Karl Koschutnig, Gernot Reishofer, Franz Ebner, and Aljoscha C. Neubauer. 2014. To create or to recall? neural mechanisms underlying the generation of creative new ideas. NeuroImage, 88:125–133. find
  4. Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D Hwang, Xiang Lorraine Li, Hirona J Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2023. : Making small language models better procedural knowledge models for (counterfactual) planning. arXiv preprint arXiv:2305.19472.
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, ab…
  6. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. ArXiv, abs/2202.07646. find
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — Evaluating Large Language Models Trained on Code
  8. David W. Graham, F.Q. Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler C. Murray, Christopher Newell, Smita R Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivash… find
  9. Laura J. Kornish and Jeremy Hutchison-Krupat. 2017. Research on idea generation and selection: Implications for management of technology. Production and Operations Management, 26:633–651. find
  10. Thomas S. Kuhn and David Hawkins. 1963. The structure of scientific revolutions. American Journal of Physics, 31:554–555. find
  11. R. Kumar. 1996. Research Methodology: A Step-by Step Guide for Beginners. Longman Australia. find
  12. Sandeep Kumar, Tirthankar Ghosal, and Asif Ekbal. 2023. When reviewers lock horns: Finding disagreements in scientific peer reviews. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16693–16704, Singapore. Association for Computational Linguistics. find
  13. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Informatio…
  14. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 – 46. find
  15. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B. Dolan. 2015. A diversity-promoting objective function for neural conversation models. ArXiv, abs/1510.03055.
  16. Wenke Hao, Zhicheng Li, Yuchen Qian, Yuzhuo Wang, and Chengzhi Zhang. 2020. The ACL FWS-RC: A dataset for recognition and classification of sentence about future works. In JCDL ’20: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Virtual Event, China, August 1-5, 2020, pag… find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  18. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023. Benchmarking large language models as ai research agents. ArXiv, abs/2310.03302.
  19. Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, Kyle Lo, Lucy Lu… — S2ORC: The Semantic Scholar Open Research Corpus
  20. J.J. Louviere, T.N. Flynn, and A.A.J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge books online. Cambridge University Press. find
  21. Rahul Nadkarni, David Wadden, Iz Beltagy, Noah A. Smith, Hannaneh Hajishirzi, and Tom Hope. 2021. Scientific language models for biomedical knowledge base completion: An empirical study. ArXiv, abs/2106.09700. — [arXiv:2106.09700](https://arxiv.org/abs/2106.09700) find
  22. OpenAI. 2022. text-embedding-ada-002. https://platform.openai.com/docs/guides/ embeddings. OpenAI embedding model used via API. ↗
  23. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  24. CoRR, Plato, G.R.F. Ferrari, and T. Griffith. 2000. Plato: ’The Republic’. Cambridge Texts in the History of Political Thought. Cambridge University Press. find
  25. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. ArXiv, abs/2311.05965. find
  26. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, Alhussein Fawzi, Josh Grochow, Andrea Lodi, Jean-Baptiste Mouret, Talia Ringer, and Tao Yu. 2023. M… — Mathematical Discoveries from Program Search with Large Language Models
  27. Serhad Sarica, Binyang Song, Jianxi Luo, and Kristin L. Wood. 2021. Idea generation with technology semantic network. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 35:265 – 283. find
  28. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In ICER 2022: ACM Conference on International Computing Education Research, Lugano and Virtual Event, Switzerland, August 7 - 11, 2022, Volum… find
  29. W. Scott Spangler, Angela D. Wilkins, Benjamin J. Bachman, Meena Nagarajan, Tajhal Dayaram, Peter J. Haas, Sam Regenbogen, Curtis R. Pickering, Austin Comer, Jeffrey N. Myers, Ioana Stanoi, Linda Kato, Ana Lelescu, Jacques J. Labrie, Neha Parikh, Andreas Martin Lisewski, Lawrence A. Donehower, Ying… find
  30. Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Unsupervised word embeddings capture latent knowledge from materials science literature. Nat., 571(7763):95–98. find
  31. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2023. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259. find
  32. Qingyun Wang, Lifu Huang, Zhiying Jiang, Kevin Knight, Heng Ji, Mohit Bansal, and Yi Luan. 2019. PaperRobot: Incremental draft generation of scientific ideas. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1980–1991, Florence, Italy. Association for… find
  33. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview…
  34. Yi Xu, Shuqian Sheng, Bo Xue, Luoyi Fu, Xinbing Wang, and Chenghu Zhou. 2023. Exploring and verbalizing academic ideas by concept co-occurrence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13001–13027, Toronto, Canada. Ass… find
  35. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  36. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  37. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. 2023. A survey of large l…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumar-2025-can-large-language-models-unlock/