Elamur Research⌕
☰
home / papers / 2023 / Mathematics, word problems, common sense, and artificial intelligence

Mathematics, word problems, common sense, and artificial intelligence

Paper · 2023 · Ernest Davis

Summary

In this survey and analysis paper, Ernest Davis (NYU) examines how well current artificial intelligence can solve math word problems that require elementary mathematics and everyday common sense. Examples include counting coins after an exchange between two people or determining the water level in a cylinder after a cube is placed in it. The author proposes a classification of problems and shows that, as of late 2022, no system solved such problems reliably. He describes three approaches based on large language models (LLMs): the model gives the answer directly, writes a program that computes it, or translates the problem into a formal representation for an automated proof checker. He also reviews benchmarks and experiments and criticizes the excessive precision of published metrics. The author believes that these limitations may not be critical for formal proofs in pure mathematics. However, they are significant for applied mathematics and, probably, for machine reading of papers written by humans.

Key points

  • The problems are classified as follows: symbolic problems, word problems, purely mathematical and "real-world" word problems, commonsense word problems (CSW), and their simplest subtype, elementary CSW. Knowledge is further divided into commonly known, encyclopedic, and expert knowledge.
  • The history of the field begins with Bobrow's STUDENT system (1964). Almost all current solutions are built around LLMs. The exception is Wolfram Alpha, which has strong mathematics but weak language understanding and almost no common sense.
  • The scale of LLMs, according to the paper: roughly 10⁹–10¹² parameters and training corpora of 10¹⁰–10¹³ tokens. The context window is only hundreds or thousands of words.
  • On metrics: accuracy results reported to three significant digits are not statistically justified, since that would require a sample of about a million examples. The author considers only about "half a digit" (about 2 bits) to be genuinely meaningful. The fact that OOD results are sometimes higher than IID results also indicates that such numbers are unreliable.
  • Separate open problems: how to choose an appropriate precision for describing a real-world situation and how detailed a model to build. An example is the pendulum: from a harmonic oscillator to accounting for friction, the rotation of the Earth (Foucault pendulum), and the influence of a nearby mountain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Artificial intelligence · 49 works

Cite

Cite as Ernest Davis. 2023. Mathematics, word problems, common sense, and artificial intelligence. arXiv:2301.09723.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.09723 · arXiv 2301.09723

@misc{davis2023mathematics,
  title = {{Mathematics, word problems, common sense, and artificial intelligence}},
  author = {Ernest Davis},
  year = {2023},
  doi = {10.48550/arxiv.2301.09723},
  eprint = {2301.09723},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.09723}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beech, Martin. The Pendulum Paradigm: Variations on a Theme and the Measure of Heaven and Earth. Brown Walker Press, 2014. find
  2. Bobrow, Daniel G. “A question-answering system for high school algebra word problems.” AFIPS, 1964. 591-614. find
  3. Bommasani, Rishi, et al. “On the Opportunities and Risks of Foundation Models.” arXiv preprint arXiv:2018.07258. https://arxiv.org/abs/2108.07258 ↗
  4. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. “Language models are few-shot learners.” Advances in Neural Information Processing Systemsi 33 (2020): 1877-1901. https://proceedings.neurips.cc/ paper/2020/hash/1457c0d6bfcb4967418b… ↗
  5. Chen, Mark et al. (2021). “Evaluating large language models trained on code.” arXiv preprint arXiv:2107.03374 https://arxiv.org/abs/2107.03374 ↗
  6. Davis, Ernest. “Using human skills taxonomies and tests as measures of AI.” In Stuart Elliott (ed.) AI and the Future of Skills, Volume 1: Capabilities and Assessments, OECD Publishing, 2021. find
  7. Davis, Ernest. “Limits of an AI program for solving college math problems.” arXiv preprint arXiv:2208.06906 https://arxiv.org/abs/2107.03374 — [arXiv:2208.06906](https://arxiv.org/abs/2208.06906) ↗
  8. Davis, Ernest. “Benchmarks for Automated Commonsense Reasoning: A Survey”. In preparation. find
  9. Drori, Iddo et al. (2022) “A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.” Proceedings of the National Academy of Sciences (PNAS) 119(32), p.e2123433119. https://www.pnas.org/doi/abs/10. 1073/pnas.2123433119 ↗
  10. Ellenberg, Jordan. Shape: The Hidden Geometry of Information, Biology, Strategy, Democracy, and Everything Else. Penguin Press, 2021. find
  11. Gillioz, Anthony, Jacky Casas, Elena Mugellini, and Omar Abou Khaled. ”Overview of the Transformer-based Models for NLP Tasks.” In 2020 15th Conference on Computer Science and Information Systems (FedCSIS), pp. 179-183. IEEE, 2020. https://ieeexplore.ieee.org/ abstract/document/9222960 ↗
  12. Lenat, Douglas B., Mayank Prakash, and Mary Shepherd. “CYC: Using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks.” AI Magazine 6, no. 4 (1985): 65-65. find
  13. Miao, Shen-yun, Chao-Chun Liang, and Keh-Yih Su. 2020. “A diverse corpus for evaluating and developing English math word problem solvers.” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, https://doi.org/10.18653/ v1/2020.acl-main.92 find
  14. Mishra, Swaroop, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit et al. “Līla: A unified benchmark for mathematical reasoning.” arXiv preprint arXiv:2210.17517 (2022). https://arxiv.org/abs/2210.17517 ↗
  15. Nelsen, Roger B., Proofs Without Words: Exercises in Visual Thinking. Mathematical Association of America, 1993. find
  16. Nguyen, Nhan, and Sarah Nadi. “An empirical evaluation of GitHub copilot’s code suggestions.” In Proceedings of the 19th International Conference on Mining Software Repositories, pp. 1-5. 2022. find
  17. Patel, Arkil, Satwik Bhattamishra, and Navin Goyal. “Are NLP Models really able to Solve Simple Math Word Problems?.” arXiv preprint arXiv:2103.07191 (2021). https://arxiv.org/ abs/2103.07191 ↗
  18. Qian, Jing, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. “Limitations of Language Models in Arithmetic and Symbolic Induction.” arXiv preprint arXiv:2208.05051 (2022). https://arxiv.org/abs/2208.05051 ↗
  19. Wu, Yuhuai, Albert Q. Jiang, Wenda Li, Markus N. Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. “Autoformalization with large language models.” arXiv preprint arXiv:2205.12615 (2022). https://arxiv.org/abs/2205.12615 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/davis-2023-mathematics-word-problems-common-sense/