Elamur Research⌕
☰
home / papers / 2026 / Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations

Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations

Paper · 2026 · Li-Chun Lu, Miri Liu, Pin Chun Lu et al.

Summary

The paper asks whether common automatic creativity metrics agree with human perceptions of creativity across different kinds of creative work. The authors compare four measures, each working at a different granularity: perplexity (token-level surprise), the Creativity Index (CI; phrase-level n-gram overlap with web corpora), syntactic templates (reliance on frequent part-of-speech patterns), and LLM-as-a-Judge (holistic scoring with rubrics and chain-of-thought). They evaluate these measures across three domains: generating research ideas, creative writing, and solving problems through unconventional approaches. For each domain they build a dataset that splits examples into creative and uncreative sets according to human judgment, then check whether each metric separates the two sets. No metric works reliably everywhere. A metric that succeeds in one domain can fail in another, and different metrics sometimes rank the same sets in opposite directions. An exploratory concept-level score based on embeddings did not fix the problem either. The study is diagnostic and does not propose a new metric. It shows that current tools mostly capture surface or stylistic novelty and miss conceptual novelty, so practitioners should treat these scores with caution.

Key points

  • Domains and labels: The three domains use different creative/uncreative splits:
    • creative writing: human-expert vs LLM-generated texts;
    • problem-solving: unconventional vs conventional solutions;
    • research ideation: accepted vs rejected papers.
  • Creativity Index: CI does distinguish the two sets in creative writing but fails in problem-solving. It mainly reflects lexical diversity and depends heavily on implementation choices. Switching the L-uniqueness range from 5–7 to 5–11 can change the gap between the sets roughly twofold.
  • Perplexity and syntactic templates: Perplexity tracks fluency more than novelty, and its creative and uncreative score distributions overlap heavily in every domain. Syntactic templates struggle where new ideas are written in formulaic language, as in research ideation and problem-solving.
  • LLM-as-a-Judge: Judgments shift with small changes in prompt wording, and judges show one-sided label bias. When the same judge rated the same item three times, it agreed with itself only about 40% of the time.
  • Semantic Novelty Score (SNS): This exploratory score averages embedding distances (1 − cosine similarity, using text-embedding-004) between extracted concepts. It was tested on 30 creative and 30 uncreative samples per domain and did not separate the sets consistently:
    • creative writing: 0.447 vs 0.421;
    • problem-solving: 0.402 vs 0.402;
    • research ideation: 0.420 vs 0.435, with rejected papers scoring higher.
  • Released resources: The datasets and evaluation code are available at github.com/lichun-19/creative_eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓National Taiwan UniversityTaipei · Taiwan29 works · 2,116 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Li-Chun Lu, Miri Liu, Pin Chun Lu, Yufei Tian, Shao-Hua Sun and Nanyun Peng. 2026. Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.297

@inproceedings{lu2026rethinking,
  title = {{Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations}},
  author = {Li-Chun Lu, Miri Liu, Pin Chun Lu, Yufei Tian, Shao-Hua Sun and Nanyun Peng},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.297},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.297}
}

References

70 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STO-RIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. In Empirical Methods in Natural Language Processing. find
  2. Akhil Atmakuru, Jatin Nainani, Rithwik Sai Reddy Bheemreddy, Aditi Lakkaraju, Zhen Yao, Hamed Zamani, and H Susan Chang. 2024. Cs4: Measuring the creativity of large language models automatically by controlling the number of story-writing constraints. arXiv preprint arXiv:2410.04197. — [arXiv:2410.04197](https://arxiv.org/abs/2410.04197) find
  3. Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, and 1 others. 2024. Large concept models: Language modeling in a sentence representation space. CoRR. find
  4. Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. 2021. Mirostat: A neural text decoding algorithm that directly controls perplexity. In International Conference on Learning Representations. find
  5. R. E. Beaty and D. R. Johnson. 2021. Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods. find
  6. Stephanie Cabral, Daniel Restrepo, Zahir Kanjee, Philip Wilson, Byron Crowe, Raja-Elie Abdulnour, and Adam Rodman. 2024. Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA Internal Medicine, (5). find
  7. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024. Art or artifice? large language models and the false promise of creativity. In CHI Conference on Human Factors in Computing Systems. find
  8. Sijia Chen, Baochun Li, and Di Niu. 2024. Boosting of thoughts: Trial-and-error problem solving with large language models. In International Conference on Learning Representations. find
  9. David H Cropley. 2006. The role of creativity as a driver of innovation. In IEEE International Conference on Management of Innovation and Technology. find
  10. Scott A Crossley, Kasia Muldner, and Danielle S Mc-Namara. 2016. Idea generation in student writing: Computational assessments and links to successful writing. Written Communication, (3). find
  11. Shuangrui Ding, Zihan Liu, Xiaoyi Dong, Pan Zhang, Rui Qian, Conghui He, Dahua Lin, and Jiaqi Wang. 2024. Songcomposer: A large language model for lyric and melody composition in song generation. arXiv preprint arXiv:2402.17645. — [arXiv:2402.17645](https://arxiv.org/abs/2402.17645) find
  12. Paul Distefano, John Patterson, and Roger Beaty. 2024. Automatic scoring of metaphor creativity with large language models. Creativity Research Journal. find
  13. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. Melbourne, Australia. Association for Computational Linguistics.
  14. Daisy Fancourt, Claire Garnett, Neta Spiro, Robert West, and Daniel Müllensiefen. 2019. How do artistic creative activities regulate our emotions? validation of the emotion regulation strategies for artistic creative activities scale (ers-aca). PLOS ONE, (2). find
  15. Gregory J Feist. 1998. A meta-analysis of personality in scientific and artistic creativity. Personality and social psychology review, 2(4):290–309. find
  16. Emma Felton, Karen Vichie, and Eloise Moore. 2015. Widening participation creatively: creative arts education for social inclusion. Higher Education Research & Development, (3). find
  17. Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. 2023. Llms accelerate annotation for medical information extraction. In Proceedings of the 3rd Machine Learning for Heal… find
  18. Fabrício Góes, Piotr Sawicki, Marek Grzes, Marco Volpe, and Jacob Watson. 2023. Pushing gpt’s creativity to its limits: Alternative uses and torrance tests. In International Conference on Innovative Computing and Cloud Computing. find
  19. Carlos Gómez-Rodríguez and Paul Williams. 2023. A confederacy of models: a comprehensive evaluation of LLMs on creative writing. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. Association for Computational Linguistics. find
  20. Erik E. Guzik, Christian Byrge, and Christian Gilde. 2023. The originality of machines: Ai takes the torrance test. Journal of Creativity, (3). find
  21. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2025. Training large language models to reason in a continuous latent space. In Second Conference on Language Modeling. find
  22. Sakib Haque, Zachary Eberhart, Aakash Bansal, and Collin McMillan. 2022. Semantic similarity metrics for evaluating source code summarization. In IEEE/ACM International Conference on Program Comprehension. find
  23. Mete Ismayilzada, Antonio Laverghetta Jr., Simone A. Luchini, Reet Patel, Antoine Bosselut, Lonneke van der Plas, and Roger Beaty. 2025. Creative preference optimization. Findings of the Association for Computational Linguistics: EMNLP 2025. find
  24. Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Lonneke van der Plas. 2024a. Creativity in ai: Progresses and challenges. arXiv preprint arXiv:2410.17218. — [arXiv:2410.17218](https://arxiv.org/abs/2410.17218) find
  25. Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2024b. Evaluating creative short story generation in humans and large language models. arXiv preprint arXiv:2411.02316. — [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) find
  26. Ducel Jean-Berluche. 2024. Creative expression and mental health. Journal of Creativity, (2). find
  27. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, (S1). find
  28. Girija Kaimal, Hasan Ayaz, Joanna Herres, Rebekka Dieterich-Hartwell, Bindal Makwana, Donna H. Kaiser, and Jennifer A. Nasser. 2017. Functional near-infrared spectroscopy assessment of reward perception based on visual self-expression: Coloring, doodling, and free drawing. The Arts in Psychotherapy. find
  29. H. Kim. 2015. Community and art: creative education fostering resilience through art. Asia Pacific Education Review. find
  30. J. P. Guilford. 1967. The nature of human intelligence. McGraw-Hill. find
  31. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San…
  32. Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2024. The curious decline of linguistic diversity: Training language models on synthetic text. In Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics. find
  33. Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jianyong Wang. 2024. Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 18608–18616. find
  34. Ana Guerberof-Arenas and Antonio Toral. 2022. Creativity in translation: Machine translation as a constraint for literary texts. Translation Spaces, 11(2):184–212. find
  35. Ethan Lin, Zhiyuan Peng, and Yi Fang. 2024. Evaluating and enhancing large language models for novelty assessment in scholarly publications. arXiv preprint arXiv:2409.16605. find
  36. Anirban Mukherjee and Hannah Chang. 2023. The creative frontier of generative ai: Managing the novelty-usefulness tradeoff. arXiv preprint arXiv:2306.03601. — [arXiv:2306.03601](https://arxiv.org/abs/2306.03601) find
  37. J Liu, S Min, L Zettlemoyer, Y Choi, and H Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens, 2024a. URL https://arxiv.org/abs/2401.17377. ↗
  38. Timothy Nguyen. 2024. Understanding transformers via n-gram statistics. Advances in neural information processing systems, 37:98049–98082. find
  39. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024a. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  40. Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung yi Lee, and Shao-Hua Sun. 2024b. LLM discussion: Enhancing the creativity of large language models via discussion framework and role-play. In First Conference on Language Modeling. find
  41. Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and 1 others. 2024c. Ai as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text aga… — [arXiv:2410.04265](https://arxiv.org/abs/2410.04265) find
  42. Yizhong Lu, Deren Wang, Tianyi Li, Dongji Jiang, Sanjeev Khudanpur, Ming Jiang, and Daniel Khashabi. 2024d. Benchmarking language model creativity: A case study on code generation. arXiv preprint arXiv:2407.09007.
  43. S. A. Luchini, N. T. Maliakkal, P. V. DiStefano, Jr. Laverghetta, A., J. D. Patterson, R. E. Beaty, and R. Reiter-Palmon. 2025a. Automated scoring of creative problem solving with large language models: A comparison of originality and quality ratings. Psychology of Aesthetics, Creativity, and the Ar… find
  44. S. A. Luchini, I. M. Moosa, J. D. Patterson, D. Johnson, M. Baas, B. Barbot, I. Bashmakova, M. Benedek, Q. Chen, G. E. Corazza, B. Forthmann, B. Goecke, S. Said-Metwaly, M. Karwowski, Y. N. Kenett, I. Lebuda, T. Lubart, K. G. Miroshnik, F.-K. Obialo, and 1 others. 2025b. Automated assessment of crea… find
  45. Sarnoff A. Mednick. 1962. The associative basis of the creative process. Psychological Review, (3). find
  46. William Merrill, Noah A Smith, and Yanai Elazar. 2024. Evaluating n-gram novelty of language models using rusty-dawg. arXiv preprint arXiv:2406.13069. find
  47. Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In CHI Conference on Human Factors in Computing Systems. find
  48. Jay A. Olson, Johnny Nahas, Denis Chmoulevitch, and Margaret E. Webb. 2021. Naming unrelated words predicts creativity. Proceedings of the National Academy of Sciences, (25). find
  49. Peter Organisciak, Selcuk Acar, Denis Dumas, and Kelly Berthiaume. 2023. Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity. find
  50. Tsung-Min Pai, Jui-I Wang, Li-Chun Lu, Shao-Hua Sun, Hung-Yi Lee, and Kai-Wei Chang. 2026. BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation . In Conference of the European Chapter of the Association for Computational Linguistics.
  51. Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772–68802. find
  52. Julian D. Parker, Janne Spijkervet, Katerina Kosta, Furkan Yesiler, Boris Kuznetsov, Ju-Chiang Wang, Matt Avent, Jitong Chen, and Duc Le. 2024. Stem-gen: A music generation model that listens. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  53. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) find
  54. Kai Ruan, Xuan Wang, Jixiang Hong, and Hao Sun. 2024. Liveideabench: Evaluating llms’ scientific creativity and idea generation with minimal context. arXiv e-prints, pages arXiv–2412. find
  55. Piotr Sawicki, Marek Grzes, Fabricio Goes, Dan Brown, Max Peeperkorn, and Aisha Khatun. 2023a. Bits of grass: Does gpt already know how to write like whitman? arXiv preprint arXiv:2305.11064. — [arXiv:2305.11064](https://arxiv.org/abs/2305.11064) find
  56. Piotr Sawicki, Marek Grze´s, Fabrício Góes, Daniel Brown, Max Peeperkorn, Aisha Khatun, and Simona Paraskevopoulou. 2023b. On the power of special-purpose gpt models to create and evaluate new poetry in old styles. In International Conference on Innovative Computing and Cloud Computing. find
  57. Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. 2024. Detection and measurement of syntactic templates in generated text. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Computational Linguistics.
  58. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic study of position bias in llm-as-a-judge. arXiv preprint arXiv:2406.07791.
  59. Dean Keith Simonton. 2004. Creativity in science: Chance, logic, genius, and zeitgeist. find
  60. Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. 2024a. Are large language models capable of generating human-level narratives? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, U…
  61. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622.
  62. Y. Zhao and C. Zhang. 2025. A review on the novelty measurements of academic papers. Scientometrics. find
  63. Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, and Faeze Brahman. 2024b. Mac-Gyver: Are large language models creative problem solvers? Mexico City, Mexico. Association for Computational Linguistics. find
  64. Ellis Paul Torrance. 1966. Torrance tests of creative thinking: Norms-technical manual: Verbal tests, forms a and b: Figural tests, forms a and b. Personal Press, Incorporated. find
  65. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Neural Information Processing Systems.
  66. Chao-Chung Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, and Hung-yi Lee. 2025. Mitigating forgetting in llm fine-tuning via low-perplexity token learning. Advances in Neural Information Processing Systems. ArXiv:2501.14315, accepted to NeurIPS 2025. — [arXiv:2501.14315](https://arxiv.org/abs/2501.14315) find
  67. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
  68. Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, and Guiling Wang. 2025. Assessing the creativity of llms in proposing novel solutions to mathematical problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25687– 25696.
  69. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, and 1 others. 2024. Chatmusician: Understanding and generating music intrinsically with llm. arXiv preprint arXiv:2402.16153. find
  70. Yuchen Zhang, Harsh Diddee, Samuel Holm, Han Liu, Xingyu Liu, Vighnesh Samuel, Daphne Ippolito, and 1 others. 2025. Noveltybench: Evaluating language models for humanlike diversity. arXiv preprint arXiv:2504.05228. — [arXiv:2504.05228](https://arxiv.org/abs/2504.05228) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2026-rethinking-creativity-evaluation-critical-analysis/