Elamur Research⌕
☰
home / papers / 2026 / CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges

Paper · 2026 · Zihan Wang, Lam Nguyen, Zhengyang Zhao et al.

Summary

Evolutionary code-generation systems such as AlphaEvolve aim to keep producing new artifacts. There is still no rigorous, automatic way to measure whether their outputs are actually creative rather than merely correct or hallucinated. The authors address this with CreativeBench, a code benchmark built on Boden's cognitive theory of creativity. It has two subsets. CreativeBench-Combo tests combinatorial creativity, meaning the fusion of ideas from different domains. CreativeBench-Explore tests exploratory creativity, meaning search under accumulating constraints. Tasks are generated automatically: Combo uses reverse engineering and Explore uses a self-play loop. Each solution receives a creativity score that multiplies quality, checked by sandboxed execution and an LLM judge, by novelty, measured as distance from baseline programs.

Evaluating current models yields three findings:

  • Scaling helps combination far more than exploration.
  • Larger models become more accurate but less divergent, which the authors call "convergence-by-scaling."
  • Reasoning modes mostly help constrained exploration.

The authors also introduce EvoRePE, an inference-time steering method that adds a "creativity vector" derived from evolutionary trajectories. The work gives evolutionary systems a grounded creativity metric.

Key points

  • CreativeBench contains 1,859 problems across 14 domains:
    • CreativeBench-Combo has 1,308 problems and 16,404 tests.
    • CreativeBench-Explore has 551 problems and 8,452 tests.
  • Tasks were generated with GPT-4.1, starting from the 196 Python seed problems of AutoCodeBench.
  • The benchmark is hard: Gemini-3-Pro's Pass@1 stays under 60%.
  • Human experts confirmed 89.1% of instances as valid. Automatic and human creativity rankings correlate at Spearman's ρ = 0.78.
  • EvoRePE is plug-and-play and works independently of the evolutionary strategy used. It contrasts hidden states to extract a steering vector, which is injected at inference time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations🎓University of BristolBristol · United Kingdom14 works · 725 citations🎓Southern University of Science and TechnologyShenzhen · China19 works · 1,337 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

Cite

Cite as Zihan Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang and Linyi Yang. 2026. CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1546

@inproceedings{wang2026creativebench,
  title = {{CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges}},
  author = {Zihan Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang and Linyi Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1546},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1546}
}

References

66 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2025. Gepa: Reflective prompt evoluti… find
  2. Teresa M. Amabile. 1982. Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5):997–1013. find
  3. Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, and 1 others. 2025. Amo-bench: Large language models still struggle in high school math competitions. arXiv preprint arXiv:2510.26768. find
  4. Anthropic. 2025. Introducing claude 3.5. find
  5. Anirudh Atmakuru, Jatin Nainani, Rohith Siddhartha Reddy Bheemreddy, Anirudh Lakkaraju, Zonghai Yao, Hamed Zamani, and Haw-Shiuan Chang. 2024. Cs4: Measuring the creativity of large language models automatically by controlling the number of story-writing constraints. Preprint, arXiv:2410.04197. — [arXiv:2410.04197](https://arxiv.org/abs/2410.04197) find
  6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  7. Margaret A. Boden. 2004. The Creative Mind: Myths and Mechanisms, 2 edition. Routledge. find
  8. Zhuang Qiang Bok and Watson Wei Khong Chua. 2025. Reasoning beyond the obvious: Evaluating divergent and convergent thinking in llms for financial scenarios. Preprint, arXiv:2507.18368. — [arXiv:2507.18368](https://arxiv.org/abs/2507.18368) find
  9. James M. Borg, Andrew Buskell, Rohan Kapitany, Simon T. Powers, Eva Reindl, and Claudio Tennie. 2022. Evolved open-endedness in cultural evolution: A new dimension in open-ended evolution research. Preprint, arXiv:2203.13050. — [arXiv:2203.13050](https://arxiv.org/abs/2203.13050) find
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  11. Qian Cao, Yahui Liu, Wei Bi, Yi Zhao, Ruihua Song, Xiting Wang, Ruiming Tang, Guorui Zhou, and Han Li. 2026. Dpwriter: Reinforcement learning with diverse planning branching for creative writing. arXiv preprint arXiv:2601.09609. find
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  13. Jason Chou, Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, Ziyan Xu, Bohui Zhai, Hengyi Liu, Speed Zhu, Wiggin Zhou, and Fengzong Lian. 2025. Autocodebench: Large language models are automatic code benchmark generators. find
  14. Google DeepMind. 2025. Gemini 3 Pro. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf. [Large language model]. ↗
  15. DeepSeek-AI. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  16. Matthew DeLorenzo, Vasudev Gohil, and Jeyavijayan Rajendran. 2024. Creativeval: Evaluating creativity of llm-based hardware code generation. Preprint, arXiv:2404.08806. — [arXiv:2404.08806](https://arxiv.org/abs/2404.08806) find
  17. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. 2024. Language modeling is compression. Preprint, arXiv:2309.10668.
  18. Maxence Faldor and Antoine Cully. 2024. Toward artificial open-ended evolution within lenia using quality-diversity. Preprint, arXiv:2406.04235. — [arXiv:2406.04235](https://arxiv.org/abs/2406.04235) find
  19. Gemini. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261.
  20. J. P. Guilford. 1950. Creativity. American Psychologist, 5(9):444–454. find
  21. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, and 5 others. 2021. Pre-trained models: Past, present and future. Preprint, arXiv:… find
  22. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. Preprint, arXiv:2403.07974.
  23. Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi. 2025. Artificial hivemind: The open-ended homogeneity of language models (and beyond). In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets an… find
  24. Xuhui Jiang, Yuxing Tian, Fengrui Hua, Chengjin Xu, Yuanzhuo Wang, and Jian Guo. 2024. A survey on large language model hallucination via a creativity perspective. Preprint, arXiv:2402.06647. — [arXiv:2402.06647](https://arxiv.org/abs/2402.06647) find
  25. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  26. Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. 2024. Style vectors for steering generative large language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 782–802, St. Julian’s, Malta.… find
  27. Robert Tjarko Lange, Yujin Tang, and Yingtao Tian. 2023. Neuroevobench: Benchmarking evolutionary optimizers for deep learning applications. Preprint, arXiv:2311.02394. — [arXiv:2311.02394](https://arxiv.org/abs/2311.02394) find
  28. NeurIPS 2023 Track on Datasets and Benchmarks. find
  29. Joel Lehman and Kenneth O. Stanley. 2011a. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2):189– 223. Epub 2011-02-14. find
  30. Joel Lehman and Kenneth O. Stanley. 2011b. Novelty Search and the Problem with Objectives, pages 37– 56. Springer New York, New York, NY. find
  31. Shiting Huang, Zhen Fang, Zehui Chen, Siyu Yuan, Junjie Ye, Yu Zeng, Lin Chen, Qi Mao, and Feng Zhao. 2025. Critictool: Evaluating self-critique capabilities of large language models in tool-calling error scenarios. arXiv preprint arXiv:2506.13977.
  32. Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. In Advances in Neural Information Processing Systems, volume 37, pages 57619–57641. Curran Associates, I… find
  33. Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. 2024. Open-endedness is essential for artificial superhuman intelligence. Preprint, arXiv:2406.04268. find
  34. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems.
  35. Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu, and Junyang Lin. 2025. Mtr-bench: A comprehensive benchmark for multi-turn reasoning evaluation. arXiv preprint arXiv:2505.17123.
  36. Ye Liu, Rui Meng, Shafiq Joty, silvio savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2025. CodeXEmbed: A generalist embedding model family for multilingual and multi-task code retrieval. In Second Conference on Language Modeling. find
  37. Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang, and Daniel Khashabi. 2025. Benchmarking language model creativity: A case study on code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin…
  38. Michael D. Mumford, Michele I. Mobley, Roni Reiter-Palmon, Charles E. Uhlman, and Lesli M. Doares. 1991. Process analytic models of creative capacities. Creativity Research Journal, 4(2):91–122. find
  39. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Ba… — AlphaEvolve: A coding agent for scientific and algorithmic discovery
  40. OpenAI. 2024. text-embedding-3-small [text embedding model]. https://platform.openai.com/ docs/models/text-embedding-3-small. ↗
  41. OpenAI. 2025. GPT-4.1 [large language model]. https://platform.openai.com/docs/models/ gpt-4.1. Accessed: 2025-\<month-you-accessed\>. ↗
  42. OpenAI. 2025. Introducing GPT-5.2. https://openai.com/index/introducing-gpt-5-2/. Accessed: 2025-12-21. ↗
  43. Samuel J. Paech. 2024. Eq-bench: An emotional intelligence benchmark for large language models. Preprint, arXiv:2312.06281.
  44. Tsung-Min Pai, Jui-I Wang, Li-Chun Lu, Shao-Hua Sun, Hung-Yi Lee, and Kai-Wei Chang. 2025. Billy: Steering large language models via merging persona vectors for creative generation. Preprint, arXiv:2510.10157.
  45. Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. 2016. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, Volume 3 - 2016. find
  46. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  47. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thaila… find
  48. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. Mathematical discoveries from program search with large language models…
  49. Mark A. Runco and Garrett J. Jaeger. 2012. The standard definition of creativity. Creativity Research Journal, 24(1):92–96. find
  50. Tan Min Sen, Zachary Choy Kit Chun, Swaagat Bikash Saikia, Syed Ali Redha Alsagoff, Banerjee Mohor, Nadya Yuki Wangsajaya, and Alvin Chan. 2025. Think outside the bot: Automating evaluation of creativity in LLMs for physical reasoning with semantic entropy and efficient multi-agent judge. In Worksho… find
  51. Jonathon Shlens. 2014. A tutorial on principal component analysis. Preprint, arXiv:1404.1100. — [arXiv:1404.1100](https://arxiv.org/abs/1404.1100) find
  52. Zara Siddique, Irtaza Khalid, Liam D. Turner, and Luis Espinosa-Anke. 2025. Shifting perspectives: Steering vectors for robust bias mitigation in llms. Preprint, arXiv:2503.05371.
  53. Peiqi Sui, Eamon Duede, Sophie Wu, and Richard So. 2024. Confabulation: The surprising value of large language model hallucinations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14274–14284, Bangkok, Thailand. Association f… find
  54. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  55. Dawei Wang, Difang Huang, Haipeng Shen, and Brian Uzzi. 2025. A large-scale comparison of divergent creativity in humans and large language models. Nature Human Behaviour. find
  56. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. Preprint, arXiv:2305.16291. find
  57. Frank E. Williams. 1980. Creativity Assessment Packet (CAP): Manual. D.O.K. Publishers, Buffalo, NY. find
  58. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. 2025. Writingbench: A comprehensive benchmark for generative writing. Preprint, arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  59. Junhao Xiao, Zhiyu Wu, Hao Lin, Yi Chen, Yahui Liu, Xiaoran Zhao, Zixu Wang, and Zejiang He. 2026. Not just what’s there: Enabling clip to comprehend negated visual descriptions without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 10978–10986. find
  60. Hongshen Xu, Zihan Wang, Zichen Zhu, Lei Pan, Xingyu Chen, Lu Chen, and Kai Yu. 2025a. Alignment for efficient tool calling of large language models. Preprint, arXiv:2503.06708.
  61. Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. 2025b. Reducing tool hallucination via reliability alignment. Preprint, arXiv:2412.04141. — [arXiv:2412.04141](https://arxiv.org/abs/2412.04141) find
  62. Wanghan Xu, Yuhao Zhou, Yifan Zhou, Qinglong Cao, Shuo Li, Jia Bu, Bo Liu, Yixin Chen, Xuming He, Xiangyu Zhao, Xiang Zhuang, Fengxiang Wang, Zhiwang Zhou, Qiantai Feng, Wenxuan Huang, Jiaqi Wei, Hao Wu, Yuejin Yang, Guangshuai Wang, and 88 others. 2025c. Probing scientific general intelligence of l… — [arXiv:2512.16969](https://arxiv.org/abs/2512.16969) find
  63. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  64. Yunpu Zhao, Rui Zhang, Wenyi Li, and Ling Li. 2025. Assessing and understanding creativity in large language models. Machine Intelligence Research, 22(3):417–436. find
  65. Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and Jun Wang. 2025. Memento: Fine-tuning llm agents without fine-tuning llms. Preprint, arXiv:2508.16153. — [arXiv:2508.16153](https://arxiv.org/abs/2508.16153) find
  66. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-creativebench-benchmarking-enhancing-machine-creativity/