Elamur Research⌕
☰
home / papers / 2025 / The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas

The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas

Paper · 2025 · Chenglei Si, Tatsunori Hashimoto and Diyi Yang

Summary

Earlier work (Si et al., 2025) found that expert reviewers rated research ideas from an LLM agent built on Claude-3.5-Sonnet as more novel than ideas written by human NLP experts. This paper asks whether that advantage holds once the ideas are actually implemented. The authors ran a pre-registered, blinded randomized controlled trial. They gave 43 qualified researchers either a human or an AI idea, drawn from the earlier study's pool and covering seven NLP topics. Each executor spent about three months turning the idea into a codebase and a short ACL-format paper, and 58 expert reviewers then scored the results without knowing their source. AI ideas lost far more ground between the idea stage and the execution stage than human ideas, with significantly larger drops on novelty, excitement, effectiveness and overall score (p < 0.05). After execution, human ideas actually scored higher on several metrics, although that reversal was not statistically significant. The study shows that judging ideas in isolation can overrate LLM output. It argues for evaluating ideas by their execution outcomes and for feeding those outcomes back into idea generation.

Key points

  • Design: 66 people were onboarded and 43 finished, executing 19 human and 24 AI ideas across Bias, Coding, Safety, Multilingual, Factuality, Math and Uncertainty. Executors could change experimental details such as datasets and baselines, but not the core method. One project was dropped because its idea was too vague to implement.
  • Effort: Executors spent an average of about 103 hours per project. Reviewers spent around 52 minutes per review.
  • Effectiveness scores: AI ideas fell from 6.00 before execution to 4.13 after. Human ideas barely moved, from 4.83 to 4.78.
  • Reviewer behavior: When judging executed projects, reviewers weighed a broader set of factors, which exposed weaknesses in LLM ideas that were missed at the idea stage.
  • Resources: The data are released on GitHub (NoviScl/AI-Researcher). The pre-registration is on OSF, and the study was approved by Stanford IRB (protocol 74,246).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Chenglei Si, Tatsunori Hashimoto and Diyi Yang. 2025. The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. arXiv:2506.20803.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.20803 · arXiv 2506.20803

@misc{si2025ideation,
  title = {{The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas}},
  author = {Chenglei Si, Tatsunori Hashimoto and Diyi Yang},
  year = {2025},
  doi = {10.48550/arxiv.2506.20803},
  eprint = {2506.20803},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.20803}
}

References

28 references, 21 of them with a card of ours. ↗ leads to the work.

  1. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. In NAACL, 2025.
  2. Zikui Cai, Shayan Shabihi, Bang An, Zora Che, Brian R. Bartoldson, Bhavya Kailkhura, Tom Goldstein, and Furong Huang. AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security. ICLR Workshop BuildingTrust, 2025. URL https://arxiv.org/abs/2504.20965. — [arXiv:2504.20965](https://arxiv.org/abs/2504.20965) ↗
  3. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mkadry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In ICLR, 2025.
  4. Sharad Duwal. MKA: Leveraging Cross-Lingual Consensus for Model Abstention. ICLR Workshop BuildingTrust, 2025. URL https://arxiv.org/abs/2503.23687. — [arXiv:2503.23687](https://arxiv.org/abs/2503.23687) ↗
  5. Tao Feng, Yihang Sun, and Jiaxuan You. GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation. In ICLR, 2025.
  6. Ali Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel Rodriques. Robin: A multi-agent system for automating scientific discovery. ArXiv, 2025. URL https://arxiv.org/abs/2505.13400. ↗
  7. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andr… — Accelerating scientific discovery with Co-Scientist
  8. Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas. ArXiv, 2024. URL https://arxiv.org/abs/2410.14255. ↗
  9. Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code. arXiv, 2025. URL https://arxiv.org/abs/2506.02314. ↗
  10. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xinxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Li Bing. Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. In ICLR, 2025.
  11. Zexi Liu, Jingyi Chai, Xinyu Zhu, Shuo Tang, Rui Ye, Bo Zhang, Lei Bai, and Siheng Chen. ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering. arXiv, 2025. URL https://arxiv.org/abs/2505.23723. — [arXiv:2505.23723](https://arxiv.org/abs/2505.23723) ↗
  12. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery . ArXiv, 2024. URL https://arxiv.org/abs/2408.06292. ↗
  13. Alexander Novikov, Ngân V~u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav M. Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, Matej Bal… — AlphaEvolve: A coding agent for scientific and algorithmic discovery
  14. Charles O’Neill, Tirthankar Ghosal, Roberta Raileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuca. Sparks of Science: Hypothesis Generation Using Structured Paper Data. ArXiv, abs/2504.12976, 2025. URL https://arxiv.org/abs/2504.12976. ↗
  15. Nathan Roll. PolyPrompt: Automating Knowledge Extraction from Multilingual Language Models with Dynamic Prompt Generation. ArXiv, abs/2502.19756, 2025. URL https://arxiv.org/ abs/2502.19756. — [arXiv:2502.19756](https://arxiv.org/abs/2502.19756) ↗
  16. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants. ArXiv, abs/2501.04227, 2025. URL https://arxiv.org/abs/2501.04227. ↗
  17. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. In ICLR, 2025. URL https://arxiv.org/abs/2409.04109. ↗
  18. Müge Simsek, Mathijs de Vaan, and Arnout van de Rijt. Do grant proposal texts matter for funding decisions? a field experiment. Scientometrics, 129:2521–2532, 2024. URL https://link.springer.com/article/10.1007/s11192-024-04968-7. — [doi:10.1007/s11192-024-04968-7](https://doi.org/10.1007/s11192-024-04968-7) ↗
  19. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s Ability to Replicate AI Research. ArXiv, 2025. URL — PaperBench: Evaluating AI's Ability to Replicate AI Research https://arxiv.org/ab… ↗
  20. Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou, and Nanqing Dong. Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System. In ACL, 2025.
  21. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty. In ACL, 2024a.
  22. Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, and Jieping Ye. SciPIP: An LLM-based Scientific Paper Idea Proposer. ArXiv, 2024b. URL https://arxiv.org/abs/2410.23166. — [arXiv:2410.23166](https://arxiv.org/abs/2410.23166) ↗
  23. Jiaxin Wen, Chenglei Si, Yueh han Chen, He He, and Shi Feng. Predicting Empirical AI Research Outcomes with Language Models. ArXiv, 2025. URL https://arxiv.org/abs/2506.00794. — [arXiv:2506.00794](https://arxiv.org/abs/2506.00794) ↗
  24. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. CycleResearcher: Improving Automated Research via Automated Review. In ICLR, 2025.
  25. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. ArXiv, 2025. URL https://arxiv.org/abs/2504.08066. ↗
  26. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and E. Cambria. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. ACL Findings, 2024.
  27. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents. ArXiv, 2025. URL https://arxiv.org/abs/2505. 22954. ↗
  28. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process. ArXiv, abs/2503.08569, 2025. URL https://arxiv.org/abs/2503.08569. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/si-2025-ideation-execution-gap-execution/