Elamur Research⌕
☰
home / papers / 2024 / Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

Paper · 2024 · Kai Ruan, Xuan Wang, Jixiang Hong et al.

Summary

Most benchmarks of LLMs on scientific tasks give models rich context, such as full papers or detailed task descriptions. They therefore reveal little about whether a model can produce research ideas from almost nothing. The authors address this gap with LiveIdeaBench. It prompts models with one scientific keyword and rates the resulting ideas on dimensions drawn from Guilford's divergent-thinking theory: originality, feasibility, clarity, fluency and flexibility. A rotating panel of strong LLM judges, selected from the LiveBench leaderboard, does the scoring. Safeguards prevent models from grading themselves and stop any single vendor from dominating the panel. The authors tested 41 models on 1,180 keywords from 22 fields. Scores on idea generation track general-intelligence rankings only loosely. For instance, the far smaller QwQ-32B-preview performs about as well as claude-3.7-sonnet:thinking. The authors conclude that scientific creativity in LLMs needs dedicated evaluation and perhaps different training strategies than general reasoning. They release the generated ideas, scores, code and a continuously updated leaderboard.

Key points

  • Scale: The benchmark covers 1,180 keywords across 22 scientific domains and evaluates 41 models from Anthropic, OpenAI, Google, Qwen, DeepSeek, Meta, Mistral and other developers, as ranked on LiveBench in March 2025.
  • Judging setup:
    • The panel consists of the top 10 LiveBench models, with no more than 2 per organization.
    • Only one variant of each base model is kept, so o3-mini and o3-mini-high would not both serve.
    • The model being evaluated is always excluded from the panel.
    • Each idea receives originality, feasibility and clarity scores averaged over 3 randomly sampled judges, while fluency is assessed by a single sampled judge per keyword–model pair.
  • Flexibility metric: It is computed by averaging the scores on the other four dimensions and taking the 30th percentile of those averages.
  • Generation protocol:
    • Ideas target about 100 words, and any answer longer than 200 words is discarded.
    • Refused prompts get a second attempt with academic framing.
    • qwq-32b-preview required a special "Final Idea:" delimiter for parsing its output.
  • Resources: The dataset is on Hugging Face as 6cf/liveideabench-v2, the code is on GitHub at x66ccff/liveideabench, and both are also archived on Zenodo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu and Hao Sun. 2024. Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context. arXiv:2412.17596.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.17596 · arXiv 2412.17596

@misc{ruan2024evaluating,
  title = {{Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context}},
  author = {Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu and Hao Sun},
  year = {2024},
  doi = {10.48550/arxiv.2412.17596},
  eprint = {2412.17596},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.17596}
}

References

83 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wang, H. et al. Scientific discovery in the age of artificial intelligence. Nature 620, 47–60 (2023). find
  2. Shi, F. & Evans, J. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications 14, 1641 (2023). find
  3. AI4Science, M. R. & Quantum, M. A. The impact of large language models on scientific discovery: a preliminary study using GPT-4. arXiv preprint arXiv:2311.07361 (2023). — [arXiv:2311.07361](https://arxiv.org/abs/2311.07361) find
  4. Krenn, M. et al. Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network. Nature Machine Intelligence 5, 1326–1335 (2023). find
  5. Rafner, J., Beaty, R. E., Kaufman, J. C., Lubart, T. & Sherson, J. Creativity in the age of generative AI. Nature Human Behaviour 7, 1836–1838 (2023). find
  6. Trinh, T. H., Wu, Y., Le, Q. V., He, H. & Luong, T. Solving olympiad geometry without human demonstrations. Nature 625, 476–482 (2024).
  7. Getzels, J. W. & Jackson, P. W. Creativity and Intelligence: Exploration with Gifted Students (John Wiley & Sons, New York, 1962). find
  8. Guilford, J. P. The nature of human intelligence (McGraw-Hill, 1967). find
  9. Jauk, E., Benedek, M., Dunst, B. & Neubauer, A. C. The relationship between intelligence and creativity: New support for the threshold hypothesis by means of empirical breakpoint detection. Intelligence 41, 212–221 (2013). URL https://www.sciencedirect.com/science/ article/pii/S016028961300024X. ↗
  10. Kim, K. H. Can only intelligent people be creative? A meta-analysis. Journal of Secondary Gifted Education 16, 57–66 (2005). find
  11. Preckel, F., Holling, H. & Wiese, M. Relationship of intelligence and creativity in gifted and non-gifted students: An investigation of threshold theory. Personality and individual differences 40, 159–170 (2006). find
  12. Gralewski, J., Weremczuk, E. & Karwowski, M. Intelligence and creativity of polish middle-school students: Looking for the threshold hypothesis. The New Educational Review 29, 328–338 (2012). find
  13. Park, G., Lubinski, D. & Benbow, C. P. Ability differences among people who have commensurate degrees matter for scientific creativity. Psychological science 19, 957–961 (2008). find
  14. Runco, M. A. & Albert, R. S. The threshold theory regarding creativity and intelligence: An empirical test with gifted and nongifted children. Creative Child and Adult Quarterly 11, 212–218 (1986). find
  15. Benedek, M. On the relationship between creative potential and creative achievement: Challenges and future directions. Learning and Individual Differences 110, 102424 (2024). find
  16. Weiss, S., Steger, D., Schroeders, U. & Wilhelm, O. A reappraisal of the threshold hypothesis of creativity and intelligence. Journal of Intelligence 8, 38 (2020). find
  17. Guilford, J. P. Creativity. American Psychologist 5, 444–454 (1950). find
  18. Boden, M. A. The creative mind: Myths and mechanisms (Routledge, 2004). find
  19. Amabile, T. M. The social psychology of creativity: A componential conceptualization. Journal of Personality and Social Psychology, 45(2), 357 (1983). find
  20. Rhodes, M. An analysis of creativity. The Phi Delta Kappan 42, 305–310 (1961). find
  21. Cortes, R. A., Weinberger, A. B., Daker, R. J. & Green, A. E. Re-examining prominent measures of divergent and convergent creativity. Current Opinion in Behavioral Sciences 27, 90–93 (2019). URL https://www.sciencedirect.com/science/article/pii/S2352154618301694. Creativity. ↗
  22. Rafner, J. et al. Towards game-based assessment of creative thinking. Creativity Research Journal, 35(4), 763–782 (2023). find
  23. Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022).
  24. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2024).
  25. Yao, S. et al. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2024). find
  26. Li, X.-Y., Xue, J.-T., Xie, Z. & Li, M. Think outside the code: Brainstorming boosts large language models in code generation. arXiv preprint arXiv:2305.10679 (2023). — [arXiv:2305.10679](https://arxiv.org/abs/2305.10679) find
  27. Lu, C. et al. The AI Scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292 (2024).
  28. Hu, X. et al. Nova: An iterative planning and search approach to enhance novelty and diversity of LLM generated ideas. arXiv preprint arXiv:2410.14255 (2024). find
  29. Baek, J., Jauhar, S. K., Cucerzan, S. & Hwang, S. J. ResearchAgent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738 (2024).
  30. Radensky, M. et al. Scideator: Human-LLM scientific idea generation grounded in research-paper facet recombination. arXiv preprint arXiv:2409.14634 (2024). — [arXiv:2409.14634](https://arxiv.org/abs/2409.14634) find
  31. Gottweis, J. et al. Towards an AI co-scientist. arXiv preprint arXiv:2502.18864 (2025). URL https://arxiv.org/abs/2502.18864. 2502.18864. ↗
  32. Wang, Q., Downey, D., Ji, H. & Hope, T. Scimon: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 279–299 (2024). find
  33. Gu, X. & Krenn, M. Interesting scientific idea generation using knowledge graphs and LLMs: Evaluations with 100 research group leaders. arXiv preprint arXiv:2405.17044 (2024). — [arXiv:2405.17044](https://arxiv.org/abs/2405.17044) find
  34. Gu, X. & Krenn, M. Forecasting high-impact research topics via machine learning on evolving knowledge graphs. arXiv preprint arXiv:2402.08640 (2024). — [arXiv:2402.08640](https://arxiv.org/abs/2402.08640) find
  35. Wang, W. et al. SciPIP: An LLM-based scientific paper idea proposer. arXiv preprint arXiv:2410.23166 (2024). — [arXiv:2410.23166](https://arxiv.org/abs/2410.23166) find
  36. Pu, K. et al. IdeaSynth: Iterative research idea development through evolving and composing idea facets with literature-grounded feedback. arXiv preprint arXiv:2410.04025 (2024). — [arXiv:2410.04025](https://arxiv.org/abs/2410.04025) find
  37. Meincke, L., Girotra, K., Nave, G., Terwiesch, C. & Ulrich, K. T. Using large language models for idea generation in innovation. The Wharton School Research Paper Forthcoming (2024). URL https://ssrn.com/abstract=4526071. 4526071. ↗
  38. Si, C., Yang, D. & Hashimoto, T. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. arXiv preprint arXiv:2409.04109 (2024). URL https://arxiv.org/abs/2409.04109. 2409.04109. ↗
  39. Lee, B. C. & Chung, J. An empirical investigation of the impact of ChatGPT on creativity. Nature Human Behaviour 8, 1906–1914 (2024). find
  40. Cropley, D. Is artificial intelligence more creative than humans?: ChatGPT and the divergent association task. Learning Letters 2, 13–13 (2023). find
  41. Marrone, R., Cropley, D. & Medeiros, K. How does narrow AI impact human creativity? Creativity Research Journal 1–11 (2024). find
  42. Wenger, E. & Kenett, Y. We’re different, we’re the same: Creative homogeneity across LLMs. arXiv preprint arXiv:2501.19361 (2025). — [arXiv:2501.19361](https://arxiv.org/abs/2501.19361) find
  43. Doshi, A. R. & Hauser, O. P. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances 10 (2023). URL https://api.semanticscholar.org/CorpusID:271119565. ↗
  44. Lu, Y., Wang, D., Li, T., Jiang, D. & Khashabi, D. Benchmarking language model creativity: A case study on code generation. arXiv preprint arXiv:2407.09007 (2024).
  45. Zhao, Y. et al. Assessing and understanding creativity in large language models. arXiv preprint arXiv:2401.12491 (2024). — [arXiv:2401.12491](https://arxiv.org/abs/2401.12491) find
  46. Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 (2024). find
  47. Zheng, L. et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, 46595–46623 (2023).
  48. Li, T. et al. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. arXiv preprint arXiv:2406.11939 (2024).
  49. Chiang, C.-H. & Lee, H.-y. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15607–15631 (Association for Computational Linguistics, Toronto, Canada, 2023). URL https:/… find
  50. Chen, Y., Wang, R., Jiang, H., Shi, S. & Xu, R. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Park, J. C. et al. (eds.) Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), 361–374 (Association for Com…
  51. Huang, F., Kwak, H., Park, K. & An, J. ChatGPT rates natural language explanation quality like humans: But on which scales? In Calzolari, N. et al. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 3111–3… find
  52. Gilardi, F., Alizadeh, M. & Kubli, M. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, e2305016120 (2023). — ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
  53. Badshah, S. & Sajjad, H. Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text. arXiv preprint arXiv:2408.09235 (2024). — [arXiv:2408.09235](https://arxiv.org/abs/2408.09235) find
  54. Verga, P. et al. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796 (2024).
  55. Kim, S. et al. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535 (2024).
  56. Rafner, J. et al. Mapping citizen science through the lens of human-centered AI. Human Computation 9, 66–95 (2022). find
  57. Akata, Z. et al. A research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer 53, 18–28 (2020). find
  58. Dellermann, D., Ebel, P., Söllner, M. & Leimeister, J. M. Hybrid intelligence. Business & Information Systems Engineering 61, 637–643 (2019). find
  59. White, C. et al. LiveBench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations (2025). URL https://openreview.net/forum?id=sKYHBTAxVa. ↗
  60. Jain, N. et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 (2024).
  61. Wei, J., Huang, D., Lu, Y., Zhou, D. & Le, Q. V. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958 (2024). URL https://arxiv.org/ abs/2308.03958. 2308.03958. ↗
  62. Gu, J. et al. A survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2025). URL https://arxiv.org/abs/2411.15594. 2411.15594. ↗
  63. Fanous, A. et al. SycEval: Evaluating LLM sycophancy. arXiv preprint arXiv:2502.08177 (2025). URL https://arxiv.org/abs/2502.08177. 2502.08177. — [arXiv:2502.08177](https://arxiv.org/abs/2502.08177) ↗
  64. Huang, L. et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43, 1–55 (2025).
  65. Falk, J. et al. How do hackathons foster creativity? towards ai collaborative evaluation of creativity at scale. In CHI’25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery (ACM), 2025). find
  66. DiStefano, P. V. et al. Evaluating ai’s ideas: The role of individual creativity and expertise in human-ai co-creativity (2025). find
  67. Sourati, J. & Evans, J. A. Accelerating science with human-aware artificial intelligence. Nature Human Behaviour 7, 1682–1696 (2023). find
  68. Lacoste, A., Luccioni, A., Schmidt, V. & Dandres, T. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019).
  69. Rince, S. & Banse, A. EcoLogits: Evaluate the environmental impacts of generative AI (2025). URL https://doi.org/10.5281/zenodo.15132099. — [doi:10.5281/zenodo.15132099](https://doi.org/10.5281/zenodo.15132099) ↗
  70. AI Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Claude-3 Model Card (2024). find
  71. Achiam, J. et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
  72. Team, G. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024).
  73. Bai, J. et al. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023).
  74. Qwen: et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2025). URL https://arxiv.org/abs/2412.15115. 2412.15115. ↗
  75. Liu, A. et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024). find
  76. DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437 (2025). 2412. 19437. find
  77. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). 2501.12948.
  78. Dubey, A. et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024).
  79. Jiang, A. Q. et al. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). find
  80. Intelligence, A. A. G. The Amazon Nova family of models: Technical report and model card (2024). find
  81. Abdin, M. et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). 2412.08905.
  82. Beltagy, I., Lo, K. & Cohan, A. SciBERT: A pretrained language model for scientific text. In Inui, K., Jiang, J., Ng, V. & Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
  83. Cohen, E. The boundary lens: Theorising academic activity. In The University and its Boundaries: Thriving or Surviving in the 21st Century, 14–41 (Routledge, New York, New York, 2021), 1 edn. Acknowledgements: The work is supported by the National Natural Science Foundation of China (No. 92270118, N… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ruan-2024-evaluating-llms-divergent-thinking-capabilities/