Elamur Research⌕
☰
home / papers / 2025 / Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

Paper · 2025 · Haoyang Su, Renqi Chen, Shixiang Tang et al.

Summary

Most LLM tools for automated research ideation use a single agent or rely on artificial collaboration setups, such as invented researcher profiles or synthetic co-author networks. This paper asks whether a team of LLM agents grounded in real scholarly data can produce more novel ideas. The authors build a "Scientific Research Ecosystem", a snapshot of a research community at a chosen date. It contains agents cloned from real scientists' backgrounds and publications, a database of papers from before the cutoff, and a database of papers from after it.

On top of this ecosystem they propose VirSci, a five-stage pipeline:

  1. A leader agent picks collaborators.
  2. The team agrees on a topic.
  3. Members propose and refine ideas, with discussion inside the team and with invited outside agents.
  4. Candidate ideas are checked for novelty against existing literature and voted on.
  5. The chosen idea is written up as an abstract.

Generated abstracts are scored by comparing them to the past and contemporary paper databases. VirSci beats a single-agent setup and the prior state of the art. It also reproduces known Science of Science patterns, which makes it a possible testbed for studying collaboration.

Key points

  • Evaluation metrics: three novelty measures cover distance from earlier work (Historical Dissimilarity), fit with emerging research trends (Contemporary Dissimilarity), and likely influence on contemporary research (CI). They are combined into Overall Novelty (ON).
  • Main result: compared with a single agent, the multi-agent system improves alignment with contemporary research by +13.8% on average and potential impact by +44.1%.
  • Data: experiments use two public AMiner datasets, one single-discipline and one multi-discipline (Open Academic Graph). Scientists' names are masked for privacy.
  • Team diversity: novelty follows an inverted-U curve, peaking at roughly 50–75% diversity. Fresher teams also tend to produce more innovative ideas, consistent with Science of Science findings.
  • Code and limitations: code is released at github.com/open-sciencelab/Virtual-Scientists. The authors note that the system cannot model researchers working in several teams at once, inherits LLM biases, and could generate plausible but incorrect claims.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations

Also on the byline: Shanghai Institute for Science of Science.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou and Nanqing Dong. 2025. Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1368

@inproceedings{su2025many,
  title = {{Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System}},
  author = {Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou and Nanqing Dong},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1368},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1368}
}

References

76 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. 2024. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1–3. find
  2. Andrei V Alexandrov and Michael G Hennerici. 2007. Writing good abstracts. Cerebrovascular Diseases, 23(4):256–259. find
  3. Bedoor K AlShebli, Talal Rahwan, and Wei Lee Woon. 2018. The preeminence of ethnic diversity in scientific collaboration. Nature communications, 9(1):5163. find
  4. Leila Amgoud and Henri Prade. 2009. Using arguments for making and explaining decisions. Artificial Intelligence, 173(3-4):413–436. find
  5. Tanveer Ansar and Gregory Talavera. 2025. Bias detection and robustness testing in large language models: An experimental framework. find
  6. Kshitij Bakliwal, Maharshi Harshadbhai Dhada, Adria Salvador Palau, Ajith Kumar Parlikad, and Bhupesh Kumar Lad. 2018. A multi agent system architecture to implement collaborative learning for social industrial assets. IFAC-PapersOnLine, 51(11):1237–1242. find
  7. Kevin W Boyack, Richard Klavans, and Katy Börner. 2005. Mapping the backbone of science. Scientometrics, 64(3):351–374. find
  8. Zhaoqing Cong, Songsong Tang, Leiming Xie, Ming Yang, Yangyang Li, Dongdong Lu, Jiahong Li, Qingxin Yang, Qiwei Chen, Zhiqiang Zhang, et al. 2022. Magnetic-powered janus cell robots loaded with oncolytic adenovirus for active and targeted virotherapy of bladder cancer. Advanced Materials, 34(26):220… find
  9. Ning Ding, Shang Qu, Linhai Xie, Yifei Li, Zaoqu Liu, Kaiyan Zhang, Yibai Xiong, Yuxin Zuo, Zhangren Chen, Ermo Hua, et al. 2024. Automating exploratory proteomics research via language models. arXiv preprint arXiv:2411.03743. find
  10. Ali Dorri, Salil S Kanhere, and Raja Jurdak. 2018. Multi-agent systems: A survey. Ieee Access, 6:28573–28593. find
  11. Zhuoyun Du, Chen Qian, Wei Liu, Zihao Xie, Yifei Wang, Yufan Dang, Weize Chen, and Cheng Yang. 2024. Multi-agent software development through cross-team collaboration. arXiv preprint arXiv:2406.08979. — [arXiv:2406.08979](https://arxiv.org/abs/2406.08979) find
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Barbara Dunin-Keplicz and Rineke Verbrugge. 2011. Teamwork in multi-agent systems: A formal approach. John Wiley & Sons. find
  14. Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojević, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. 2018. Science of science. Science, 359(6379):eaao0185. find
  15. Suzanne Fricke. 2018. Semantic scholar. Journal of the Medical Library Association: JMLA, 106(1):145. find
  16. Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao, Hongyi Peng, Ze Yu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Agentscope: A flexible yet robust multi-agent platform.… — [arXiv:2402.14034](https://arxiv.org/abs/2402.14034) find
  17. Hugh G Gauch. 2003. Scientific method in practice. Cambridge University Press. find
  18. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  19. Roger Guimera, Brian Uzzi, Jarrett Spiro, and Luis A Nunes Amaral. 2005. Team assembly mechanisms determine collaboration network structure and team performance. Science, 308(5722):697–702. find
  20. Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Lu Wang, and Aakanksha Naik. 2024. Chime: Llm-assisted hierarchical organization of scientific studies for literature review support. arXiv preprint arXiv:2407.16148. find
  21. Kaixuan Huang, Yuanhao Qu, Henry Cousins, William A Johnson, Di Yin, Mihir Shah, Denny Zhou, Russ Altman, Mengdi Wang, and Le Cong. 2024. Crispr-gpt: An llm agent for automated design of gene-editing experiments. arXiv preprint arXiv:2404.18021. find
  22. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.
  23. Irving L Janis. 1972. Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. find
  24. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.
  25. Claire Kayacik, Sherol Chen, Signe Noerly, Jess Holbrook, Adam Roberts, and Douglas Eck. 2019. Identifying the intersections: User experience+ research scientist collaboration in a generative machine learning interface. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing S… find
  26. J Kühnisch, O Meyer, M Hesenius, R Hickel, and V Gruhn. 2022. Caries detection on intraoral images using artificial intelligence. Journal of dental research, 101(2):158–165. find
  27. P Langley. 1987. Scientific discovery: Computational explorations of the creative processes. MIT Press. find
  28. Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. Open source strikes bread - new fluffy embeddings model. find
  29. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459– 9474.
  30. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008.
  31. Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. Avalonbench: Evaluating llms playing the game of avalon. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  32. Julie S Linsey, Matthew G Green, Jeremy T Murphy, Kristin L Wood, and Art B Markman. 2005. “collaborating to success”: An experimental study of group idea generation techniques. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume… find
  33. Lu Liu, Benjamin F Jones, Brian Uzzi, and Dashun Wang. 2023. Data, measurement and empirical methods in the science of science. Nature human behaviour, 7(7):1046–1058. find
  34. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  35. James G March. 1991. Exploration and exploitation in organizational learning. Organization science, 2(1):71–87. find
  36. Jack Mezirow. 2003. How critical reflection triggers transformative learning. Adult and Continuing Education: Teaching, learning and research, 4:199– 213. find
  37. Santiago Miret and NM Krishnan. 2024. Are llms ready for real-world materials discovery? arXiv preprint arXiv:2402.05200. find
  38. Henrique Muzzio and Manuella Gama. 2024. Collaborative idea generation: An experience of open creativity in the public sector. VINE Journal of Information and Knowledge Management Systems, 54(1):176– 194. find
  39. OpenAI. 2023. GPT-4 technical report. CoRR.
  40. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023a. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  41. Michael Park, Erin Leahey, and Russell J Funk. 2023b. Papers and patents are becoming less disruptive over time. Nature, 613(7942):138–144. find
  42. Jill E Perry-Smith and Pier Vittorio Mannucci. 2017. From creativity to innovation: The social network drivers of the four phases of the idea journey. Academy of management review, 42(1):53–79. find
  43. Julia Pitters and Thomas Oberlechner. 2014. The psychology of trading and investing. Investor behavior: The psychology of financial planning and investing, pages 457–476. find
  44. Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. 2024. Large language models as biomedical hypothesis generators: A comprehensive evaluation. arXiv preprint arXiv:2407.08940.
  45. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computat…
  46. Maithra Raghu and Eric Schmidt. 2020. A survey of deep learning for scientific discovery. arXiv preprint arXiv:2003.11755. — [arXiv:2003.11755](https://arxiv.org/abs/2003.11755) find
  47. Andrey Rzhetsky, Jacob G Foster, Ian T Foster, and James A Evans. 2015. Choosing experiments to accelerate collective discovery. Proceedings of the National Academy of Sciences, 112(47):14569– 14574. find
  48. Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. Bert-pli: Modeling paragraph-level interactions for legal case retrieval. In IJCAI, pages 3501–3507. find
  49. Feng Shi and James Evans. 2023. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications, 14:1641. find
  50. Zhengliang Shi, Shen Gao, Zhen Zhang, Xiuying Chen, Zhumin Chen, Pengjie Ren, and Zhaochun Ren. 2023. Towards a unified framework for reference retrieval and related work generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5785–5799. find
  51. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.
  52. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.
  53. Stefan Wuchty, Benjamin F Jones, and Brian Uzzi. 2007. The increasing dominance of teams in production of knowledge. Science, 316(5827):1036–1039. find
  54. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109.
  55. Scott Spangler, Angela D Wilkins, Benjamin J Bachman, Meena Nagarajan, Tajhal Dayaram, Peter Haas, Sam Regenbogen, Curtis R Pickering, Austin Comer, Jeffrey N Myers, et al. 2014. Automated hypothesis generation based on mining scientific literature. In Proceedings of the 20th ACM SIGKDD internationa… find
  56. Cass R Sunstein. 2005. Why societies need dissent. Harvard University Press. find
  57. Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990–998. find
  58. Brian Uzzi, Satyam Mukherjee, Michael Stringer, and Ben Jones. 2013. Atypical combinations and scientific impact. Science, 342(6157):468–472. find
  59. Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. 2022. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734. — [arXiv:2209.14734](https://arxiv.org/abs/2209.14734) find
  60. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60. find
  61. Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. 2024. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.18272. find
  62. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  63. Lingfei Wu, Dashun Wang, and James A Evans. 2019. Large teams develop and small teams disrupt science and technology. Nature, 566:378–382. find
  64. Oskar Wysocki, Magdalena Wysocka, Danilo Carvalho, Alex Teodor Bogatu, Danilo Miranda Gusicuma, Maxime Delmas, Harriet Unsworth, and Andre Freitas. 2024. An llm-based knowledge synthesis and scientific reasoning framework for biomedical discovery. arXiv preprint arXiv:2406.18626. find
  65. Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Kai Shu, Adel Bibi, Ziniu Hu, Philip Torr, Bernard Ghanem, and Guohao Li. 2024. Can large language model agents simulate human trust behaviors? arXiv preprint arXiv:2402.04559. — [arXiv:2402.04559](https://arxiv.org/abs/2402.04559) find
  66. Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and Jiashi Feng. 2023. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. find
  67. Yongjun Xu, Xin Liu, Xin Cao, Changping Huang, Enke Liu, Sen Qian, Xingchen Liu, Yanjun Wu, Fengliang Dong, Cheng-Wei Qiu, et al. 2021. Artificial intelligence: A powerful paradigm for scientific research. The Innovation, 2(4). find
  68. Yang Yang, Tanya Y Tian, Teresa K Woodruff, Benjamin F Jones, and Brian Uzzi. 2022. Gender-diverse teams produce more novel and higher-impact scientific ideas. Proceedings of the National Academy of Sciences, 119(36):e2200841119. find
  69. Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, and Jiaxuan You. 2024. Researchtown: Simulator of human research community. arXiv preprint arXiv:2412.17767.
  70. An Zeng, Ying Fan, Zengru Di, Yougui Wang, and Shlomo Havlin. 2021. Fresh teams are associated with original and multidisciplinary research. Nature human behaviour, 5(10):1314–1322. find
  71. An Zeng, Zhesi Shen, Jianlin Zhou, Ying Fan, Zengru Di, Yougui Wang, H Eugene Stanley, and Shlomo Havlin. 2019. Increasing trend of scientists to switch between topics. Nature communications, 10(1):3439. find
  72. Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Evgeny Kharlamov, Bin Shao, et al. 2022. Oag: Linking entities across large-scale heterogeneous knowledge graphs. IEEE Transactions on Knowledge and Data Engineering, 35(9):9225–9239. find
  73. Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. 2024. Exploring collaboration mechanisms for LLM agents: A social psychology view. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 14544–14607.
  74. Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. 2023. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439. find
  75. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. 2023. Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.07984. — [arXiv:2310.07984](https://arxiv.org/abs/2310.07984) find
  76. Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Jianbing Shen, Guodong Long, Can Xu, and Daxin Jiang. 2024. Fine-grained distillation for long document retrieval. In Thirty-Eighth AAAI Conference on Artificial Intelligence, pages 19732–19740. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2025-many-heads-are-better-than/