Elamur Research⌕
☰
home / papers / 2025 / Benchmarking Language Model Creativity: A Case Study on Code Generation

Benchmarking Language Model Creativity: A Case Study on Code Generation

Paper · 2025 · Yining Lu, Dixuan Wang, Tianjian Li et al.

Summary

Most studies of LLM creativity focus on open-ended tasks such as story writing. This leaves open how creative models are when they solve problems. Two things have been missing: a way to push models beyond repetitive outputs, and a grounded, quantitative metric. Drawing on cognitive science, the authors treat creativity as two parts: convergent thinking (reaching a valid goal) and divergent thinking (exploring novel approaches).

They introduce Denial Prompting. It detects the techniques a model used in its previous solution (e.g., for-loops, if-else) and then forbids them. This yields a chain of increasingly constrained versions of the same problem. Their NeoGauge metric credits a solution only if it is correct and respects the constraints. It then rewards the share of techniques not found in historical human solutions.

Applied to recent Codeforces problems, the pipeline produces the NeoCoder benchmark. Across proprietary and open-source models, GPT-4 scores highest but still falls short of human-like creativity. Common reasoning strategies add little. The work gives a reproducible, human-grounded way to track creative problem solving in future models.

Key points

  • NeoCoder is built from the 199 most recent Codeforces problems, each paired with 30 human solutions that passed unit tests. Denial Prompting adds sequences of progressively harder constraints, using GPT-4 as the augmentation model.
  • NeoGauge combines convergent checks (correctness and constraint adherence) with divergent novelty. Novelty is measured against human solutions, an H-creativity view following Boden et al.; the iterative denial itself covers P-creativity.
  • Four reasoning strategies were tested: MCTS (on CodeGemma-7B), and self-correction, planning and sampling (on GPT-3.5). Sampling-based evaluation used k = 5 samples per problem.
  • Only MCTS improved divergent creativity. However, its NeoGauge gain was minimal and reached zero after t = 2. Self-correction and planning traded divergence for accuracy, and no strategy improved both dimensions at once.
  • Stated limitations: the method is restricted to Text-to-Code because it needs a historical set of human solutions. There is also a risk of future data leakage, since newer LLMs may see these Codeforces problems during pre-training. Code and data are available at github.com/JHU-CLSP/NeoCoder.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of Notre DameNotre Dame · United States51 works · 36,717 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 129 works

Cite

Cite as Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang and Daniel Khashabi. 2025. Benchmarking Language Model Creativity: A Case Study on Code Generation. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.141

@inproceedings{lu2025benchmarking,
  title = {{Benchmarking Language Model Creativity: A Case Study on Code Generation}},
  author = {Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang and Daniel Khashabi},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.141},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.141}
}

References

50 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Teresa M Amabile. 1982. Social psychology of creativity: A consensual assessment technique. Journal of personality and social psychology, 43(5):997. find
  3. David Henry Feldman. 1998. The Development of Creativity, page 169–186. Cambridge University Press. find
  4. T.M. Amabile. 1996. Creativity In Context: Update To The Social Psychology Of Creativity. Avalon Publishing. find
  5. David Henry Feldman, Mihaly Csikszentmihalyi, and Howard Gardner. 1994. Changing the world: A framework for the study of creativity. Praeger Publishers/Greenwood Publishing Group. find
  6. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. find
  7. Anirudh Atmakuru, Jatin Nainani, Rohith Siddhartha Reddy Bheemreddy, Anirudh Lakkaraju, Zonghai Yao, Hamed Zamani, and Haw-Shiuan Chang. 2024. Cs4: Measuring the creativity of large language models automatically by controlling the number of story-writing constraints. find
  8. John Baer. 1994. Divergent thinking is not a general trait: A multidomain training experiment. Creativity Research Journal, 7(1):35–46. find
  9. Margaret A Boden et al. 1994. Dimensions of creativity. find
  10. Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Alexandre Allauzen. 2024. Exploring precision and recall to assess the quality and diversity of llms. find
  11. Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024a. Art or artifice? large language models and the false Ronald A. Finke, Thomas B. Ward, and Steven M. Smith. 1996. Creative Cognition: Theory, Research, and Applications. The MIT Press. find
  12. Google. 2024. Codegemma: Open code models based on gemma.
  13. J. P. Guilford. 1950. Creativity. American Psychologist, 5(9):444–454. find
  14. Carlos Gómez-Rodríguez and Paul Williams. 2023. A confederacy of models: a comprehensive evaluation of llms on creative writing. find
  15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR).
  16. K.J. Holyoak and R.G. Morrison. 2005. The Cambridge Handbook of Thinking and Reasoning. Cambridge Handbooks in Psychology. Cambridge University Press. find
  17. Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, et al. 2023. Competition-level problems are effective llm evaluators. arXiv preprint arXiv:2312.02143. find
  18. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée La… find
  19. Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2023b. Self-planning code generation with large language models. find
  20. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv, abs/2001.08361.
  21. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS).
  22. Joseph S. Renzulli. 2005. The Three-Ring Conception of Giftedness: A Developmental Model for Promoting Creative Productivity, page 246–279. Cambridge University Press. find
  23. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, J… — Code Llama: Open Foundation Models for Code
  24. Mark A. Runco. 2003. Critical creative processes. Perspectives on creativity. Hampton Press, Cresskill, N.J. find
  25. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations.
  26. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.
  27. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).
  28. Robert J Sternberg. 1982. Natural, unnatural, and super-natural concepts. Cognitive Psychology, 14(4):451– 488. find
  29. Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung yi Lee, and Shao-Hua Sun. 2024a. Llm discussion: Enhancing the creativity of large language models via discussion framework and role-play. find
  30. Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. 2024b. Ai as humanity’s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text a… find
  31. Todd I. Lubart. 2001. Models of the creative process: Past, present and future. Creativity Research Journal, 13(3-4):295–308. find
  32. Michael D. Mumford, Michele I. Mobley, Roni Reiter-Palmon, Charles E. Uhlman, and Lesli M. Doares. 1991. Process analytic models of creative capacities. Creativity Research Journal, 4(2):91–122. find
  33. OpenAI. 2024. Gpt-4 technical report.
  34. Robert J. Sternberg. 1981. Intelligence and nonentrench-ment. In Journal of Educational Psychology. find
  35. Robert J. Sternberg and Joyce Gastel. 1989a. Coping with novelty in human intelligence: An empirical investigation. Intelligence, 13(2):187–197. find
  36. Robert J Sternberg and Joyce Gastel. 1989b. If dancers ate their shoes: Inductive reasoning with factual and counterfactual premises. Memory & Cognition, 17:1– 10. find
  37. Robert J. Sternberg and Todd I. Lubart. 1991. An investment theory of creativity and its development. Human Development, 34(1):1–31. find
  38. Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326–346, Online. Association for Computational Linguistics.
  39. Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, and Faeze Brahman. 2024. Macgyver: Are large language models creative problem solvers? find
  40. E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement. find
  41. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing.
  43. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024a. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
  44. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024b. Hallucination is inevitable: An innate limitation of large language models.
  45. Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. Planning with large language models for code generation. In International Conference on Learning Representations (ICLR). find
  46. Tianhui Zhang, Bei Peng, and Danushka Bollegala. 2024a. Improving diversity of commonsense generation by large language models via in-context learning. find
  47. Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, and Daphne Ippolito. 2024b. Forcing diffuse distributions out of language models. find
  48. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of l…
  49. Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li, and Yunji Chen. 2024. Assessing and understanding creativity in large language models. find
  50. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. Dyval: Dynamic evaluation of large language models for reasoning tasks. In International Conference on Learning Representations (ICLR). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2025-benchmarking-language-model-creativity-case/