Elamur Research⌕
☰
home / papers / 2024 / Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems

Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems

Paper · 2024 · Junyi Ye, Jingyi Gu, Xinyun Zhao et al.

Summary

Benchmarks for LLM mathematics usually check only whether the final answer is right. This paper asks a different question: can a model find a valid solution path that differs from the ones it has already been shown? The authors built CreativeMath from competition problems and the many community-written solutions on the Art of Problem Solving (AoPS) wiki. The problems span middle-school contests to the IMO.

Each model receives a problem plus the first k known solutions, in the order they were posted, and must write a new one. The output then goes through four stages: generation, a correctness check, a coarse-grained novelty check and a fine-grained novelty check. Three LLM judges score correctness and novelty against stated criteria, such as using a different method, different intermediate steps or different assumptions.

Closed-source, open-source and math-specialized models were compared. Gemini-1.5-Pro was the strongest at producing correct solutions that were also new. Smaller and math-tuned models rarely produced novel solutions. The work matters because it offers a reusable way to measure solution diversity, not just accuracy, which is relevant to AI-assisted mathematical discovery.

Key points

  • CreativeMath has 6,469 problems and 14,223 solutions drawn from eight US-related competitions. These are the three AMC levels (8, 10 and 12), the AHSME, the AIME, the USAMO and its junior version the USAJMO, plus the IMO.
  • Problems with images or missing solutions were removed. Difficulty and problem type come from official competition data, and math categories were assigned by Llama-3-70B. Algebra and Geometry are the most common categories.
  • GPT-4, Claude-3.5-Sonnet and Gemini-1.5-Pro act as evaluators, and each gives its own verdict before a final decision is made.
  • Showing more reference solutions raised accuracy, and Gemini-1.5-Pro reached 100% with four of them. It also made novel solutions harder to produce, which suggests a trade-off between using prior knowledge and being original.
  • On harder problems, accuracy dropped but correct answers were more often novel. Llama-3-70B and Yi-1.5-34B produced diverse approaches, while Mixtral-8x22B's outputs were more alike, so combining several different LLMs may increase originality.

Data and code: github.com/JunyiYe/CreativeMath

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations🎓New Jersey Institute of TechnologyNewark · United States15 works · 1,333 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMathematics · 177 worksApplied Mathematics · 46 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin and Guiling Wang. 2024. Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems. arXiv:2410.18336.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.18336 · arXiv 2410.18336

@misc{ye2024assessing,
  title = {{Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems}},
  author = {Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin and Guiling Wang},
  year = {2024},
  doi = {10.48550/arxiv.2410.18336},
  eprint = {2410.18336},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.18336}
}

References

35 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Neele Falk, Sara Papi, and Mike Zhang (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lingui…
  2. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf. Accessed: 2024-08-13. ↗
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  4. Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165, 2020.
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  6. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. find
  7. Giorgio Franceschelli and Mirco Musolesi. On the creativity of large language models. arXiv preprint arXiv:2304.00008, 2023. — [arXiv:2304.00008](https://arxiv.org/abs/2304.00008) find
  8. Carlos Gómez-Rodrı́guez and Paul Williams. A confederacy of models: A comprehensive evaluation of llms on creative writing. arXiv preprint arXiv:2310.08433, 2023. find
  9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021a.
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
  11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  12. Yu-Ju Lan and Nian-Shing Chen. Teachers’ agency in the era of llm and generative ai. Educational Technology & Society, 27(1):I–XVIII, 2024. find
  13. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models, 2022. URL https://arxiv.org/abs/2206.14858, 2022. ↗
  14. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.
  15. Yiren Liu, Si Chen, Haocong Cheng, Mengxia Yu, Xiao Ran, Andrew Mo, Yiliu Tang, and Yun Huang. How ai processing delays foster creativity: Exploring research question co-creation with an llm-based agent. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–25, 2024b. find
  16. Jay Lofstead. Economic, societal, legal, and ethical considerations for large language models. In 2023 Fifth International Conference on Transdisciplinary AI (TransAI), pp. 155–162. find
  17. Pronita Mehrotra, Aishni Parab, and Sumit Gulwani. Enhancing creativity in large language models through associative thinking strategies. arXiv preprint arXiv:2405.06715, 2024. — [arXiv:2405.06715](https://arxiv.org/abs/2405.06715) find
  18. Meta AI. Meta llama 3. https://ai.meta.com/blog/meta-llama-3/, 2024. Accessed: 22-October-2024. ↗
  19. Mistral AI. Mixtral 8x22b: The new frontier in ai models, 2024. URL https://mistral.ai/ news/mixtral-8x22b/. Accessed: 2024-08-13. ↗
  20. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106–26128. PMLR, 2023.
  21. OpenAI. Hello gpt-4o!, 2024. URL https://openai.com/index/hello-gpt-4o/. Accessed: 2024-08-13. ↗
  22. Michael Sheinman Orenstrakh, Oscar Karnalim, Carlos Anibal Suarez, and Michael Liut. Detecting llm-generated text in computing education: A comparative study for chatgpt cases. arXiv preprint arXiv:2307.07411, 2023. — [arXiv:2307.07411](https://arxiv.org/abs/2307.07411) find
  23. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530,…
  24. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nat.…
  25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  26. E Paul Torrance. Torrance tests of creative thinking. Educational and psychological measurement, 1966. find
  27. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nat., 625(7995):476–482, 2024. doi: 10.1038/S41586-023-06747-5. URL https://doi.org/10.1038/s41586-023-06747-5. ↗
  28. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  29. Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internlm-math: Open m… find
  30. Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024b. find
  31. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
  32. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. Chatmusician: Understanding and generating music intrinsically with llm. arXiv preprint arXiv:2402.16153, 2024. find
  33. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024.
  34. Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, et al. Assessing and understanding creativity in large language models. arXiv preprint arXiv:2401.12491, 2024. — [arXiv:2401.12491](https://arxiv.org/abs/2401.12491) find
  35. Jiajun Zhou, Shijie Rao, Liang Gao, Chunjiang Zhang, Hongtao Tang, Yun Li, and Felix TS Chan. Solving many-task optimization problems via online intertask learning. Expert Systems with Applications, 225:120110, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2024-assessing-creativity-llms-proposing-novel/