Elamur Research⌕
☰
home / papers / 2024 / What Makes Math Word Problems Challenging for LLMs?

What Makes Math Word Problems Challenging for LLMs?

Paper · 2024 · Kv Aditya Srivatsa and Ekaterina Kochmar

Summary

The paper asks why some English math word problems (MWPs) defeat large language models while others are solved easily. The authors take GSM8K (7,473 training and 1,319 test problems) and describe each item with three groups of hand-crafted features. Linguistic features cover length, vocabulary, syntax, coreference and readability. Mathematical features cover the operations and steps in the gold solution. A third group measures how much real-world knowledge or NLU a problem needs. Four open-source LLMs attempted every problem five times. Problems that a model always or never solved served as labelled data for interpretable classifiers, whose feature importances were then inspected. The hardest problems combine many distinct operations, unusual numbers, long and syntactically deep phrasing, poor readability, and reliance on unstated facts such as unit conversions. Such simple feature-based predictors come close to a fine-tuned RoBERTa-base. The study gives a transparent, model-agnostic way to diagnose what makes reasoning benchmarks difficult and suggests how problem wording could be adjusted.

Key points

  • Models and success rates: Llama2-13B, Llama2-70B, Mistral-7B and math-tuned MetaMath-13B were sampled at temperature 0.8. Mean test success rates were about 0.37, 0.56, 0.36 and 0.64. MetaMath-13B beat even the 70B Llama2.
  • Classifiers: Logistic Regression, Decision Tree and Random Forest were trained. Random Forest was usually best, with accuracy of roughly 71.7–81.4% across LLM splits. RoBERTa-base matched it or exceeded it by only a few points.
  • Most important features: the top-ranked features were Qx_np_count (number of noun phrases), Qx_mean_numerical_word_rank, Gx_op_unique_count and Gx_op_diversity. Readability (Qx_flesch_kinkaid_grade), Gx_world_knowledge and Qx_constituency_tree_depth also ranked highly.
  • Linguistic effect with math held fixed: KMeans grouped problems into 100 clusters with similar math features. Within these clusters, success rate correlated negatively with linguistic complexity, for example Spearman ρ = −0.82 for mean word rank and −0.64 for tree depth.
  • Ablations and availability: the full L+M+W feature set was not always the best choice, and the optimal feature subset varied by LLM. Code and data are available at github.com/kvadityasrivatsa/analyzing-llms-for-mwps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksMathematics · 233 worksApplied Mathematics · 73 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Kv Aditya Srivatsa and Ekaterina Kochmar. 2024. What Makes Math Word Problems Challenging for LLMs?. Findings of the Association for Computational Linguistics: NAACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-naacl.72 · arXiv 2403.11369

@inproceedings{srivatsa2024what,
  title = {{What Makes Math Word Problems Challenging for LLMs?}},
  author = {Kv Aditya Srivatsa and Ekaterina Kochmar},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.72},
  eprint = {2403.11369},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-naacl.72}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Husni Almoubayyed, Stephen E Fancsali, and Steve Ritter. 2023. Instruction-embedded assessment for reading ability in adaptive mathematics software. In LAK23: 13th International Learning Analytics and Knowledge Conference, pages 366–377. find
  2. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. https://huggingface.co/ spaces/HuggingFaceH4/open_llm_leaderboard. ↗
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. find
  6. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  7. Jiwoo Kim, Youngbin Kim, Ilwoong Baek, JinYeong Bak, and Jongwuk Lee. 2023. It ain’t over: A multi-aspect diverse math word problem dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14984– 15011, Singapore. Association for Computational Linguis… find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  9. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online. Association for Computational Linguistics. find
  10. Kole Norberg, Husni Almoubayyed, Stephen E. Fancsali, Logan De Ley, Kyle Weldon, April Murphy, and Steven Ritter. 2023. Rewriting math word problems with large language models. In Proceedings of the Workshop on Empowering Education with LLMs -the Next-Gen Interface and Content Generation 2023 co-loc… find
  11. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
  12. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  13. Syed Rifat Raiyan, Md Nafis Faiyaz, Shah Md. Jawad Kabir, Mohsinul Kabir, Hasan Mahmud, and Md Kamrul Hasan. 2023. Math word problem solving by generating linguistic variants of problem statements. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4:… find
  14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/srivatsa-2024-what-makes-math-word-problems/