Elamur Research⌕
☰
home / papers / 2024 / Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas

Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas

Paper · 2024 · Xiang Hu, Hongyu Fu, Jinge Wang et al.

Summary

LLM-based systems for generating research ideas often produce shallow, repetitive proposals. The authors attribute this to undirected knowledge retrieval, such as plain keyword or entity search. Nova addresses this with a three-stage pipeline. First, an LLM drafts 15 seed ideas from an input (seed) paper, its references and trending recent publications, which are identified through engagement on social media, forums and GitHub. Ten scientific discovery methods inspired by Kuhn guide this step, and self-check, self-critique and reflection filter the results. Second, over several rounds, the model writes a search plan describing which literature would make the current ideas more novel. It then retrieves those papers, generates fresh ideas and prunes them. Third, the surviving ideas are expanded with methodological detail. The evaluation used 170 recent LLM papers from venues such as ACL, ICLR and CVPR, with LLM-driven Swiss Tournament ranking plus human review. Nova produced clearly more top-rated and unique ideas than AI-Scientist, ResearchAgent and AI-Researcher. The work suggests that planning what to retrieve, rather than simply retrieving more, is what broadens LLM ideation.

Key points

  • Iterative planning raises the count of unique novel ideas 3.4× compared with the same system without it. Against prior state-of-the-art agents, Nova yields at least 2.5× more ideas reaching the top Swiss Tournament score (5).
  • In each iteration, every idea spawns 10 candidates, and self-reflection trims them to 3, which replace the previous seed ideas. Gains level off after about 3 rounds.
  • Ablation with K = 5 retrieved papers per method: without planning, unique ideas barely grow from step 2 to step 3 (42.4 → 44.1). Without both planning and retrieval, the count stalls around 31.
  • Human reviewers and the automatic LLM reviewer rank the systems in the same order: Nova, then AI-Scientist, ResearchAgent and AI-Researcher.
  • Stated limitations include the absence of a reward function to guide planning and no further benefit beyond three iterations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations🎓University of OxfordOxford · United Kingdom138 works · 161,448 citations🎓Zhejiang UniversityHangzhou · China236 works · 14,072 citations🎓Westlake UniversityHangzhou · China60 works · 8,829 citations🎓University of Electronic Science and Technology of ChinaChengdu · China39 works · 2,118 citations🎓Southeast UniversityNanjing · China33 works · 483 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan and Zhenzhong Lan. 2024. Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas. arXiv:2410.14255.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.14255 · arXiv 2410.14255

@misc{hu2024nova,
  title = {{Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas}},
  author = {Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan and Zhenzhong Lan},
  year = {2024},
  doi = {10.48550/arxiv.2410.14255},
  eprint = {2410.14255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.14255}
}

References

25 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, and Sanjiv Kumar. 2023. REST MEETS REACT: SELF-IMPROVEMENT FOR MULTI-STEP REASON-ING LLM AGENT. Preprint, arXiv:2312.100… — [arXiv:2312.10003](https://arxiv.org/abs/2312.10003) find
  2. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. Preprint, arXiv:2404.07738.
  3. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. Preprint, arXiv:2305.11738.
  4. Xuemei Gu and Mario Krenn. 2024. Generation and human-expert evaluation of interesting research ideas using knowledge graphs and large language models. Preprint, arXiv:2405.17044. — [arXiv:2405.17044](https://arxiv.org/abs/2405.17044) find
  5. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In ICML 2024.
  6. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML 2022, volume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR.
  7. Thomas S Kuhn. 1997. The structure of scientific revolutions, volume 962. University of Chicago press Chicago. find
  8. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024. Mlr-copilot: Autonomous machine learning research based on large language models agents. Preprint, arXiv:2408.14033.
  9. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. Preprint, arXiv:2408.06292.
  10. Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023. SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning. Preprint, arXiv:2308.00436.
  11. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475.
  12. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS 2023.
  13. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. Preprint, arXiv:2409.04109.
  14. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In ACL 2023, pages 10014–10037, Toronto, Canada. Association for Computational Linguistics.
  15. Haiming Wang, Huajian Xin, Chuanyang Zheng, Lin Li, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, et al. 2024a. Lego-prover: Neural theorem proving with growing libraries.
  16. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023a. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60. find
  17. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In ACL 2023, pages 2609–2634, Toronto, Canada. Association for Computational Linguistics. find
  18. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024b. Scimon: Scientific inspiration machines optimized for novelty. Preprint, arXiv:2305.14259.
  19. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR 2023. OpenReview.net.
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS 2022.
  21. Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. Preprint, arXiv:2405.00451. find
  22. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. Preprint, arXiv:2309.02726.
  23. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS 2023. find
  24. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023b. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR, 2023.
  25. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In ICLR 2023. OpenReview.net.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2024-nova-iterative-planning-search-approach/