Elamur Research⌕
☰
home / papers / 2026 / Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Paper · 2026 · Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff et al.

Summary

Most current ways of measuring creativity in large language models are built around a single task. They depend on curated answers, hand-made rubrics or human raters, which makes them hard to scale or reuse. This paper presents an automated, task-independent framework that keeps the measurement machinery apart from the creative task, following the cognitive-science split between divergent and convergent thinking.

  • Divergent creativity is scored with Semantic Entropy. This metric was originally built to detect hallucinations; here it is reused to measure how widely a model's sampled steps spread across distinct meanings.
  • Convergent creativity, meaning how well the task is fulfilled, is judged by a multi-agent LLM panel. Instead of passing along the full discussion history, the agents retrieve the most relevant earlier notes.

The authors test the framework on MacGyver (problem-solving), HypoGen (research ideation) and BookMIA (creative writing) with many LLMs. The judge agrees with humans about as well as individual annotators do, at much lower cost. Task fulfilment improves with model scale and recency, while divergent breadth does not. This suggests current training mainly rewards correctness rather than creative exploration.

Key points

  • Three judging agents (Problem, Solution, Criterion) save their insights as "fragments" inside a vector database, and every agent pulls back the k fragments with the highest cosine similarity, which cuts token usage by about 63% compared with ChatEval.
  • Semantic Entropy groups generations through bi-directional entailment. It was checked against human annotations, LLM novelty judgments and baseline diversity metrics, and it remained empirically separable from task fulfilment.
  • The evaluation covers 300 problems per domain and analyses the effects of model size, recency, sampling temperature and reasoning mode.
  • On MacGyver, GPT 4o has the best overall score (0.63), ahead of Qwen3 32B (Thinking) at 0.52 and Llama 3.1 405B Instruct at 0.51. Semantic Entropy, however, stays in a narrow band of roughly 1.96–2.19 across all models.
  • The authors list two limitations: the cost of sampling and clustering many outputs, and the risk that LLM judges undervalue unconventional but viable solutions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Banerjee Mohor, Swaagat Bikash Saikia and Alvin Chan. 2026. Automated Creativity Evaluation of Language Models Across Open-Ended Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1061

@inproceedings{sen2026automated,
  title = {{Automated Creativity Evaluation of Language Models Across Open-Ended Tasks}},
  author = {Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Banerjee Mohor, Swaagat Bikash Saikia and Alvin Chan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1061},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1061}
}

References

63 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Teresa Amabile. 1982. Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43:997–1013. find
  2. Teresa Amabile. 1983. The social psychology of creativity: A componential conceptualization. Journal of Personality and Social Psychology, 45:357–376. find
  3. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  4. Antoine Bellemare-Pepin, François Lespinasse, Philipp Thölke, Yann Harel, Kory Mathewson, Jay A. Olson, Yoshua Bengio, and Karim Jerbi. 2024. Divergent creativity in humans and large language models. Preprint, arXiv:2405.13012. — [arXiv:2405.13012](https://arxiv.org/abs/2405.13012) find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  6. Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Hong-shuai Wang, Yongge Li, Mujie Lin, Shuwen Yang, Jiankun Wang, Mingjun Xu, Jin Huang, Xi Fang, Jiaxi Zhuang, Yuqi Yin, Yaqi Li, Changhong Chen, and 4 others. 2024. Sciassess: Benchmarking llm proficiency in… find
  7. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. Preprint, arXiv:2308.07201.
  8. Honghua Chen and Nai Ding. 2023. Probing the creativity of large language models: Can models produce divergent semantic association? Preprint, arXiv:2310.11158. find
  9. Kedi Chen, Qin Chen, Jie Zhou, Xinqi Tao, Bowen Ding, Jingwen Xie, Mingchen Xie, Peilong Li, Feng Zheng, and Liang He. 2025. Enhancing uncertainty modeling with semantic graph for hallucination detection. Preprint, arXiv:2501.02020. — [arXiv:2501.02020](https://arxiv.org/abs/2501.02020) find
  10. Cyril Chhun, Fabian M. Suchanek, and Chloé Clavel. 2024. Do language models enjoy their own stories? prompting large language models for automatic story evaluation. Transactions of the Association for Computational Linguistics, 12:1122–1142.
  11. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. find
  12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  13. Arthur Cropley. 2006. In praise of convergent thinking. Creativity Research Journal, 18(3):391–404. find
  14. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  15. Matthew DeLorenzo, Vasudev Gohil, and Jeyavijayan Rajendran. 2024. Creativeval: Evaluating creativity of llm-based hardware code generation. 2024 IEEE LLM Aided Design Workshop (LAD), pages 1–5. find
  16. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  17. Carlos Gómez-Rodríguez and Paul Williams. 2023. A confederacy of models: a comprehensive evaluation of LLMs on creative writing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14504–14528, Singapore. Association for Computational Linguistics. find
  18. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, and 15 others. 202… — Accelerating scientific discovery with Co-Scientist
  19. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  20. Tianyang Gu, Jingjin Wang, Zhihao Zhang, and Hao-Hong Li. 2024. Llms can realize combinatorial creativity: generating creative ideas via llms for scientific research. Preprint, arXiv:2412.14141. — [arXiv:2412.14141](https://arxiv.org/abs/2412.14141) find
  21. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, Miami, Florida, USA. Association for Computational Linguistics. find
  22. J P Guilford. 1950. Creativity. Am. Psychol., 5(9):444– 454. find
  23. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  24. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. Just Accepted.
  25. Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Lonneke van der Plas. 2025. Creativity in ai: Progresses and challenges. Preprint, arXiv:2410.17218. — [arXiv:2410.17218](https://arxiv.org/abs/2410.17218) find
  26. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. Preprint, arXiv:2310.08491.
  27. Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ashton Anderson. 2024. Human creativity in the age of llms: Randomized experiments on divergent and convergent thinking. Preprint, arXiv:2410.03703. — [arXiv:2410.03703](https://arxiv.org/abs/2410.03703) find
  28. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Preprint, arXiv:1910.13461.
  29. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
  30. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016b. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
  31. Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023. Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation. Preprint, arXiv:2310.19740. — [arXiv:2310.19740](https://arxiv.org/abs/2310.19740) find
  32. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate.
  33. Zhihan Liu, Yubo Chai, and Jianfeng Li. 2024. Towards fully autonomous research powered by llms: Case study on simulations. Preprint, arXiv:2408.15512. — [arXiv:2408.15512](https://arxiv.org/abs/2408.15512) find
  34. Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung yi Lee, and Shao-Hua Sun. 2024. Llm discussion: Enhancing the creativity of large language models via discussion framework and role-play. Preprint, arXiv:2405.06373. — [arXiv:2405.06373](https://arxiv.org/abs/2405.06373) find
  35. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization. Preprint, arXiv:2303.15621.
  36. Fangrui Lv, Kaixiong Gong, Jian Liang, Xinyu Pang, and Changshui Zhang. 2024. Subjective topic meets LLMs: Unleashing comprehensive, reflective and creative thinking through the negation of negation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 123… find
  37. Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations. find
  38. Christopher D. Manning. 2022. Human language understanding & reasoning. Daedalus, 151(2):127– 138. find
  39. Sarnoff A. Mednick and Martha T. Shuch Mednick. 1967. Remote associates test, college, adult, form 1 and examiner’s manual, remote associates test, college and adult forms 1 and 2. find
  40. Behnam Mohammadi. 2024. Creativity has left the chat: The price of debiasing language models. Preprint, arXiv:2406.05587. — [arXiv:2406.05587](https://arxiv.org/abs/2406.05587) find
  41. Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. 2025. Sparks of science: Hypothesis generation using structured paper data. Preprint, arXiv:2504.12976.
  42. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  43. ArXiv, John D. Patterson Paul V. DiStefano and Roger E. Beaty. 2024. Automatic scoring of metaphor creativity with large language models. Creativity Research Journal, 0(0):1–15. find
  44. Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? Preprint, arXiv:2405.00492. — [arXiv:2405.00492](https://arxiv.org/abs/2405.00492) find
  45. Kai Ruan, Xuan Wang, Jixiang Hong, and Hao Sun. 2024. Liveideabench: Evaluating llms’ scientific creativity and idea generation with minimal context. Preprint, arXiv:2412.17596.
  46. Mark A. Runco and Garrett J. Jaeger. 2012. The standard definition of creativity. Creativity Research Journal, 24(1):92–96. find
  47. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. Preprint, arXiv:2310.16789. find
  48. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. Preprint, arXiv:2409.04109.
  49. Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024. LLM-check: Investigating detection of hallucinations in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  50. Douglas Summers-Stay, Stephanie M. Lukin, and Clare R. Voss. 2023. Brainstorm, then select: a generative language model improves its creativity score. find
  51. Luning Sun, Yuzhuo Yuan, Yuan Yao, Yanyan Li, Hao Zhang, Xing Xie, Xiting Wang, Fang Luo, and David Stillwell. 2024. Large language models show both individual and collective creativity comparable to humans. Preprint, arXiv:2412.03151. — [arXiv:2412.03151](https://arxiv.org/abs/2412.03151) find
  52. Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, and Faeze Brahman. 2024. Macgyver: Are large language models creative problem solvers? Preprint, arXiv:2311.09682. find
  53. Jingyuan Yang, Dapeng Chen, Yajing Sun, Rongjun Li, Zhiyong Feng, and Wei Peng. 2025. Enhancing semantic consistency of large language models through model editing: An interpretability-oriented approach. Preprint, arXiv:2501.11041. find
  54. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. find
  55. Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, and Guiling Wang. 2024. Assessing the creativity of llms in proposing novel solutions to mathematical problems. Preprint, arXiv:2410.18336.
  56. Qiang Yi, Yangfan He, Jianhui Wang, Xinyuan Song, ShiYao Qian, Xinhang Yuan, Yi Xin, Yijin Wang, Jingqun Tang, Yuchen Li, Junjiang Lin, Hongyang He, Zhen Tian, Tianxiang Xu, Keqin Li, Kuan Lu, Menghao Huo, Jiaqi Chen, Miao Zhang, and 2 others. 2025. Score: Story coherence and retrieval enhancement f… — [arXiv:2503.23512](https://arxiv.org/abs/2503.23512) find
  57. Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023. Enhancing uncertainty-based hallucination detection with stronger focus. Preprint, arXiv:2311.13230. find
  58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  59. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. Judgelm: Fine-tuned large language models are scalable judges. Preprint, arXiv:2310.17631.
  60. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. Preprint, arXiv:1802.01886.
  61. E Paul Torrance. Torrance tests of creative thinking. Title of the publication associated with this dataset: PsycTESTS Dataset. find
  62. Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkun. 2024a. Reasoning in token economies: Budget-aware evaluation of llm reasoning strategies. Preprint, arXiv:2406.06461. find
  63. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024b. Helpsteer2-preference: Complementing ratings with preferences. Preprint, arXiv:2410.01257. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sen-2026-automated-creativity-evaluation-language-models/