Elamur Research
☰
home / papers / 2024 / Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2409.04109

Summary

Do current LLMs produce research ideas that rival those of human specialists? Earlier agent papers mostly relied on small reviewer pools or LLM-as-a-judge, so this question had no answer. Si, Yang and Hashimoto ran a large controlled experiment with more than 100 NLP researchers. Experts wrote project proposals on prompting-based research. An LLM agent produced proposals on the same matched topics. The agent used retrieval augmentation and generated many candidates, then ranked them. All proposals followed a shared template and passed through an LLM-based style normalizer, so reviewers could not tell the source. Other experts then scored them blind on novelty, excitement, feasibility, expected effectiveness and an overall score.

AI proposals received significantly higher novelty ratings than human ones (p < 0.05). This result held across several statistical tests. Feasibility was rated slightly lower. The authors also found two weaknesses in the agent: its outputs repeat as generation scales up, and it judges its own ideas poorly. The study offers a reusable baseline and protocol for evaluating research agents.

Key points

  • Design: 79 expert reviewers blindly assessed 49 ideas in each of three conditions (human, AI, AI reranked by a human), producing nearly 300 reviews on 1–10 scales.
  • Novelty scores: Human 4.84, AI 5.64, AI + Human Rerank 5.81. Significance was tested with two-tailed Welch's t-tests and Bonferroni correction.
  • Topics: Seven topics were taken from recent NLP calls for papers such as COLM: Bias, Coding, Safety, Multilinguality, Factuality, Math, Uncertainty. Human writers picked a topic, and the LLM idea was matched to it.
  • Style control: After normalization, expert judges distinguished AI from human ideas at only about 50% accuracy, i.e., chance level.
  • Resources and next steps: The agent code and all review scores are released at github.com/NoviScl/AI-Researcher. A follow-up study is planned in which researchers carry out the ideas as full projects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.04109 · arXiv 2409.04109

@misc{si2024can,
  title = {{Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers}},
  author = {Si, Chenglei and Yang, Diyi and Hashimoto, Tatsunori},
  year = {2024},
  doi = {10.48550/arxiv.2409.04109},
  eprint = {2409.04109},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.04109}
}

References

56 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. LitSearch: A Retrieval Benchmark for Scientific Literature Search. ArXiv, abs/2407.18940, 2024. find
  2. Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization Effects of Large Language Models on Human Creative Ideation. In Proceedings of the 16th Conference on Creativity & Cognition, 2024. find
  3. Joshua Ashkinaze, Julia Mendelsohn, Qiwei Li, Ceren Budak, and Eric Gilbert. How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experiment. ArXiv, abs/2401.13481, 2024. — [arXiv:2401.13481](https://arxiv.org/abs/2401.13481) find
  4. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. ArXiv, abs/2404.07738, 2024. find our card
  5. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sandra Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe,… find
  6. Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The neurips 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/ the-neurips-2021-consistency-experiment, 2021. Neural Information Processing Systems blog post. ↗
  7. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R’e, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. ArXiv, abs/2407.21787, 2024. find our card
  8. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or Artifice? Large Language Models and the False Promise of Creativity. In CHI, 2024. find
  9. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR, 2023. find our card
  10. Katherine M. Collins, Albert Qiaochu Jiang, Simon Frieder, Li Siang Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wenda Li, Adrian Weller, and Mateja Jamnik. Evaluating language models for mathematics through interactions. Proceeding… find
  11. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. MARG: Multi-Agent Review Generation for Scientific Papers. ArXiv, abs/2401.04259, 2024. find
  12. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-Verification Reduces Hallucination in Large Language Models. ArXiv, abs/2309.11495, 2023. find our card
  13. Shizhe Diao, Pengcheng Wang, Yong Lin, Xiang Liu, and Tong Zhang. Active Prompting with Chain-of-Thought for Large Language Models. In ACL, 2024. find
  14. Esin Durmus, Faisal Ladhak, and Tatsunori B. Hashimoto. Spurious Correlations in Reference-Free Evaluation of Text Generation. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://api.semanticscholar.org/CorpusID:248300077. ↗
  15. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking Language Model Agents for Data-Driven Science. ArXiv, abs/2408.09667,… find
  16. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. In ICML, 2024. find
  17. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. Predicting Results of Social Science Experiments Using Large Language Models. Preprint, 2024. URL https://docsend. com/view/ity6yf2dansesucf. find
  18. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks. In ICML, 2024. find
  19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In ICML, 2024. find our card
  20. Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. Autonomous LLM-driven research from data to human-verifiable research papers. ArXiv, abs/2404.17605, 2024. find
  21. Hao Kang and Chenyan Xiong. ResearchArena: Benchmarking LLMs’ Ability to Collect and Organize Information as Research Agents. ArXiv, abs/2406.10291, 2024. — [arXiv:2406.10291](https://arxiv.org/abs/2406.10291) find
  22. Michelle S. Lam, Janice Teoh, James Landay, Jeffrey Heer, and Michael S. Bernstein. Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM. In CHI, 2024. find
  23. Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R. Banaji. ChatGPT as Research Scientist: Probing GPT’s Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor. Proceedings of the National Academy of Sciences of the United States of America, 121 3… find
  24. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS, 2020. find
  25. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents. ArXiv, abs/2408.14033, 2024. find
  26. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey, Cherepanov, Jame… find our card
  27. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis. NEJM AI, 1(8), 2024. find
  28. Yiren Liu, Si Chen, Haocong Cheng, Mengxia Yu, Xiao Ran, Andrew Mo, Yiliu Tang, and Yun Huang. How AI Processing Delays Foster Creativity: Exploring Research Question Co-Creation with an LLM-based Agent. In CHI, 2024. find
  29. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery . ArXiv, abs/2408.06292, 2024. find our card
  30. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback. In NeurIP… find
  31. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discovery-Bench: Towards Data-Driven Discovery with Large Language Models. ArXiv, abs/2407.01725, 2024. find
  32. Benjamin S. Manning, Kehang Zhu, and John J. Horton. Automated Social Science: Language Models as Scientist and Subjects. SSRN Electronic Journal, 2024. find
  33. Vishakh Padmakumar and He He. Does Writing with Language Models Reduce Content Diversity? In ICLR, 2024. find
  34. Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. CiteME: Can Language Models Accurately Cite Scientific Claims? ArXiv, abs/2407.12861, 2024. find
  35. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. In ACL, 2024. find our card
  36. Nils Reimers and Iryna Gurevych. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In EMNLP, 2020. find our card
  37. Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson C. Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander… find
  38. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. REPLUG: Retrieval-Augmented Black-Box Language Models. In NAACL, 2024. find our card
  39. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber, and Lijuan Wang. Prompting GPT-3 To Be Reliable. In ICLR, 2023. find our card
  40. Müge Simsek, Mathijs de Vaan, and Arnout van de Rijt. Do grant proposal texts matter for funding decisions? a field experiment. Scientometrics, 129:2521–2532, 2024. find
  41. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Min Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng… find
  42. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 – 482, 2024. find our card
  43. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty. In ACL, 2024. find
  44. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR, 2023. find our card
  45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022. find our card
  46. Orion Weller, Marc Marone, Nathaniel Weir, Dawn J Lawrie, Daniel Khashabi, and Benjamin Van Durme. “According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data. In EACL, 2023. find
  47. Jason Weston and Sainbayar Sukhbaatar. System 2 Attention (is something you might need too). ArXiv, abs/2311.11829, 2023. — [arXiv:2311.11829](https://arxiv.org/abs/2311.11829) find
  48. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and E. Cambria. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. ACL Findings, 2024. find
  49. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS, 2023. find
  50. Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed Huai hsin Chi, and Denny Zhou. Large Language Models as Analogical Reasoners. In ICLR, 2024. find
  51. Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, and Qiaozhu Mei. MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows. ArXiv, abs/2406.06357, 2024. find
  52. Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed Huai hsin Chi, Quoc V. Le, and Denny Zhou. Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. In ICLR, 2024. find
  53. Ruiqi Zhong, Charles Burton Snell, Dan Klein, and Jacob Steinhardt. Describing Differences between Text Distributions with Natural Language. In ICML, 2022. find
  54. Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal Driven Discovery of Distributional Differences via Language Descriptions. In NeurIPS, 2023. find
  55. Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In ICLR, 2023. find our card
  56. Yilun Zhou, Caiming Xiong, Silvio Savarese, and Chien-Sheng Wu. Shared Imagination: LLMs Hallucinate Alike. ArXiv, abs/2407.16604, 2024. find