Elamur Research
☰
home / papers / 2023 / Mathematical Capabilities of ChatGPT

Mathematical Capabilities of ChatGPT

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2301.13867

Summary

Most natural-language benchmarks for language models test only school-level mathematics or contain very few problems. Formal mathematics, by contrast, has large proof libraries. This paper asks whether ChatGPT and GPT-4 handle research-level mathematics well enough to help professional mathematicians. Working mathematicians built GHOSTS, a benchmark of 709 prompts in six subdatasets. Two subdatasets reuse existing sources (MATH and a symbolic-integration set), and four are new and hand-written. Together they cover graduate textbook exercises, proofs with missing steps, olympiad problems and literature-search style queries. Experts graded every answer on a 1–5 scale and added warning and error codes that describe how the answer failed. They evaluated the 9-January-2023 and 30-January-2023 ChatGPT releases on the full set. GPT-4 was tested on miniGHOSTS, a smaller subset with similar statistics. The ChatGPT update did not improve the overall score, and its ratings mostly shifted between prompts. GPT-4 scored clearly higher but still failed on graduate-level material. The study suggests that these models currently work best as a lookup tool for mathematical facts and definitions. They cannot yet be trusted to write correct proofs.

Key points

  • Datasets: GHOSTS (709 prompts) and miniGHOSTS are published at github.com/xyfrieder/science-GHOSTS. The subdatasets are Grad-Text, Holes-in-Proofs, Olympiad-Problem-Solving, Symbolic-Integration, MATH and Search-Engine-Aspects.
  • Grading scale: answers are rated from 1 to 5, and a rating of 3.5 is treated as roughly a passing grade. Failure-mode codes go beyond the reference-graph schema used for NaturalProofs.
  • GPT-4 results: on miniGHOSTS, GPT-4 averaged 4.15 (SD 1.12). Compared with ChatGPT, it got about twice as many perfect scores and about half as many answers rated 1–2. Its answers were longer and sometimes rambled, and it still made simple arithmetic slips.
  • ChatGPT update: the 30-January ChatGPT release was advertised as better at maths. Its overall scores were about the same as the 9-January release, but which prompts it got right changed considerably.
  • Overall finding: both models do best on definition retrieval and other search-style tasks. Their scores drop as problems get harder, and they stay below graduate-student level and below specialised single-task models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.13867 · arXiv 2301.13867

@misc{frieder2023mathematical,
  title = {{Mathematical Capabilities of ChatGPT}},
  author = {Frieder, Simon and Pinchetti, Luca and Chevalier, Alexis and Griffiths, Ryan-Rhys and Salvatori, Tommaso and Lukasiewicz, Thomas and Petersen, Philipp Christian and Berner, Julius},
  year = {2023},
  doi = {10.48550/arxiv.2301.13867},
  eprint = {2301.13867},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.13867}
}

References

54 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sascha Lobo. Das Ende von Google, wie wir es kannten. Der Spiegel, Retrieved 2023-01-10. https: //www.spiegel.de/netzwelt/netzpolitik/bessere-treffer-durch-chatgpt-das-ende-von-google-wie-wir-es-kannten-kolumne-a-77820af6-51d7-4c03-b822-cf93094fd709. find
  2. John Naughton. The ChatGPT bot is causing panic now – but it’ll soon be as mundane a tool as Excel. The Guardian, Retrieved 2023-01-14. https://www.theguardian.com/commentisfree/2023/jan/07/chatgpt-bot-excel-ai-chatbot-tec. ↗
  3. Kevon Roose. The Brilliance and Weirdness of ChatGPT. The New York Times, Retrieved 2023-01-24. https://www.nytimes.com/2022/12/05/technology/chatgpt-ai-twitter.html. ↗
  4. teddy [@teddynpc]. I made ChatGPT take a full SAT test. Here’s how it did: [Image attached] [Tweet]. Twitter. 2023-01-13. https://twitter.com/teddynpc/status/1598767389390573569. ↗
  5. Timothy Gowers [@wtgowers]. It’s amusing when ChatGPT makes ridiculous mathematical mistakes. But of course, it’s more interesting to find out what it can do well. Here’s one example that wasn’t bad: I gave it a very rough outline of a proof and asked it to fill in the details [Tweet]. Twitter. 2023… find
  6. OpenAI (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. find our card
  7. Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, and Lorie De Leon et al. Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. medRxiv, 2022. find
  8. David Rozado. What is the IQ of ChatGPT? Retrieved 2023-01-09. https://davidrozado. substack.com/p/what-is-the-iq-of-chatgpt. find
  9. Christian Terwiesch. Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course. Retrieved 2023-01-04. https://mackinstitute.wharton. upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf. ↗
  10. Natalie. ChatGPT – Release Notes. Retrieved 2023-04-03. articles/6825453-chatgpt-release-notes. find
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, and Steven Basart et al. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021. find our card
  12. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019. find
  13. A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):210–229, 1959. find
  14. Jörg Denzinger, Matthias Fuchs, Christoph Goller, and Stephan Schulz. Learning from previous proof experience: A survey. Technical report, TU München, 1999. find
  15. John Harrison, Josef Urban, and Freek Wiedijk. History of interactive theorem proving. In Computational Logic, volume 9, pages 135–214, 2014. find
  16. Malik Amir, Yang-Hui He, Kyu-Hwan Lee, Thomas Oliver, and Eldar Sultanow. Machine Learning Class Numbers of Real Quadratic Fields. arXiv preprint arXiv:2209.09283, 2022. — [arXiv:2209.09283](https://arxiv.org/abs/2209.09283) find
  17. Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, and Daniel Zheng et al. Advancing mathematics by guiding human intuition with AI. Nature, 600(7887):70–74, 2021. find
  18. Yang-Hui He. Machine-learning the string landscape. Physics Letters B, 774:564–568, 2017. find
  19. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, and Henryk Michalewski et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, 2022. find our card
  20. Francois Charton, Amaury Hayat, and Guillaume Lample. Learning advanced mathematical computations from examples. In International Conference on Learning Representations, 2021. https://help.openai.com/en/ ↗
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017. find
  22. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, and Gaurav Mishra et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  23. Aida Amini, Saadia Gabriel, Shanchuan Lin, and Rik Koncel-Kedziorski et al. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find our card
  24. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, and Heewoo Jun et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  25. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared D Kaplan et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901, 2020. find
  26. Piotr Piękos, Mateusz Malinowski, and Henryk Michalewski. Measuring and improving BERT’s mathematical abilities by predicting the order of reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lang… find
  27. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 158–167. Association for Computational Linguisti… find
  28. Teven Le Scao and Angela Fan et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  29. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, and Apoorv Kulshreshtha et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. find our card
  30. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. arXiv preprint arXiv:2212.10535, 2022. find
  31. Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021. find
  32. Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Naturalprover: Grounded mathematical proof generation with language models. arXiv preprint arXiv:2205.12910, 2022. find
  33. Rick Durrett. Probability: Theory and Examples. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. find
  34. James R. Munkres. Topology. Prentice-Hall, 2000. find
  35. Walter Rudin. Functional analysis. McgGraw-Hill, 1991. find
  36. Sheldon Axler. Linear algebra done right. Springer, 2015. find
  37. W. Rudin. Principles of Mathematical Analysis. International series in pure and applied mathematics. McGraw-Hill, 1976. find
  38. Arthur Engel. Problem-Solving Strategies. Springer, 1998. find
  39. Tranquil Sea Of Math. Does ChatGPT code LaTeX and write proofs? Youtube. Retrieved 2023-01-12. https://www.youtube.com/watch?v=ge2N7VI\_8P0. ↗
  40. Richard Van Noorden @richvn@mastodon.social [@Richvn]. Huh. ChatGPT confidently gives the right kind of reasoning to solve this math problem, but whiffs on the algebra in the middle and gets the answer wrong [Tweet]. Twitter. 2023-01-09. https://twitter.com/Richvn/status/1598714487711756288. ↗
  41. Amos Azaria. ChatGPT Usage and Limitations. Retrieved 2023-01-15. https://hal.science/hal-03913837. ↗
  42. Ernest Davis. Mathematics, word problems, common sense, and artificial intelligence. arXiv preprint arXiv:2301.09723, 2023. find
  43. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. find our card
  44. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. find our card
  45. The mathlib Community. The Lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. ACM, 2020. find
  46. Markus N. Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. arXiv preprint arXiv:2006.04757v3, 2020. find
  47. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, and Carroll L. Wainwright et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. find
  48. Carroll Wainwright and Ryan Lowe. InstructGPT: Training Language Models to Follow Instructions with Human Feedback . GitHub repository, Retrieved 2023-01-09. https://github.com/openai/following-instructions-human-feedback. ↗
  49. Sarah Wiegreffe (sigmoid.social/@sarah) [@sarahwiegreffe]. If text-davinci-001 is a rough approximate to the model reported in the NeurIPS 2020 paper, and text-davinci-002 is InstructGPT in the 2022 preprint, then what is just "davinci"? Trying to reproduce results from a time before this naming exi… find
  50. OpenAI. GPT-4 API waitlist. Retrieved 2023-06-06. https://openai.com/waitlist/gpt-4-api. ↗
  51. OpenAI. Documentation - Models. Retrieved 2023-06-06. https://platform.openai.com/docs/models/gpt-4. ↗
  52. OpenAI. OpenAI API Reference - Chat Completion Endpoint. Retrieved 2023-06-06. https: //platform.openai.com/docs/api-reference/chat. find
  53. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and Henrique Ponde de Oliveira Pinto et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. find our card
  54. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86– 92, 2021. find