Elamur Research⌕
☰
home / papers / 2022 / ArMATH: a Dataset for Solving Arabic Math Word Problems

ArMATH: a Dataset for Solving Arabic Math Word Problems

Summary

Work on automatically solving math word problems has focused almost entirely on English and Chinese. Arabic had only one earlier statistical study, which used 500 problems translated from English. The authors fill this gap with ArMATH, 6,000 single-unknown primary-school problems in Modern Standard Arabic. Each problem is paired with an equation. Some items were newly written by native-speaker writers to mirror Saudi Ministry of Education textbooks. The rest were adapted from MATH23K by translation agencies, with names, foods and other cultural details localized and sometimes the equation itself changed. On this data they train GTS, a tree-structured decoder solver, and compare learned embeddings against pretrained aravec and fasttext vectors. They also test cross-lingual transfer: a GTS model trained on Chinese MATH23K is used to initialize the Arabic model, either fully or only on the decoder side. Copying only the decoder works best and lifts answer accuracy to 74.15%, about 3 points above the non-transfer baseline. The largest gains come on rare equation templates. The dataset and code are public, giving a first benchmark for Arabic math problem solving.

Key points

  • ArMATH has 6,000 MSA problems. Each is tagged by origin: novel (written from scratch) or inspired (adapted from MATH23K). Each also has a preprocessed segmented version in which Indian digits and spelled-out numbers are converted to Arabic numerals.
  • Quality control combined three checks:
    • a native speaker and a Saudi primary-school math teacher reviewed the items;
    • professionals proofread them;
    • a script flagged any equation number that neither appears in the question nor is a constant, which caught digit typos such as 112 vs 122.
  • Without transfer, GTS with aravec (CBOW n-gram, d=100) reached 58.75% equation accuracy and 71.17% answer accuracy. The fasttext version scored 56.30% and 68.58%.
  • Transferring the Chinese-trained decoder only gave the best results: 61.02% equation and 74.15% answer accuracy. Transferring encoder and decoder together helped little or even hurt, because Arabic and Chinese differ too much on the input side.
  • With transfer, accuracy rose by more than 7 points on low-frequency templates. The model solved problems whose equation form was never seen in training 27% of the time, 11 points above the baseline.

Interesting ideas

  • I1. Check every number in a gold equation: it must either occur in the problem text or be a known constant, and anything else is flagged as a probable annotation typo. — For Mova (answer): Run this provenance test on our maths word-problem items and on the solver's own equations, so that any quantity not grounded in the text or a constant table gets flagged and explained.
  • I2. Mark each item by how it was produced (written fresh vs adapted from another corpus), so results can be broken down by origin. — For Mova (learn): Tag our test items as hand-written, LLM-generated or converted, and report scores per tag to detect when gains come only from one source's style.
  • I3. Adapt borrowed problems to the target culture, swapping names and objects and sometimes altering the arithmetic, instead of translating word for word. — For Mova (answer): Build perturbed copies of school word problems with new names, objects and changed numbers or relations, to confirm that the world model reasons over structure rather than memorized surface forms.
  • I4. Share only the language-independent output side (the equation decoder) across languages, and leave the language-specific encoder local. — For Mova (world): Keep the step from world state to equation free of English-specific features, so it can be reused unchanged if a new language front end is ever added.
  • I5. Report accuracy bucketed by how often each equation template occurs, plus a separate score for templates never seen in training. — For Mova (learn): Add template-frequency and unseen-template slices to our maths evaluation, so we can see whether improvements reach rare problem shapes or only the common ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alghamdi2022armath,
  title = {{ArMATH: a Dataset for Solving Arabic Math Word Problems}},
  author = {Alghamdi, Reem and Liang, Zhenwen and Zhang, Xiangliang},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/4aruuvt7kbnv},
  url = {https://doi.org/10.63317/4aruuvt7kbnv}
}

References

37 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. (2019a). Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In NAACL, pages 2357–2367. find our card
  2. Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. (2019b). MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, Minneapolis, Minnesota, June. Association for Computational Linguistics. find our card
  3. Chiang, T. and Chen, Y. (2018). Semantically-aligned equation generation for solving and reasoning math word problems. In NAACL. find
  4. Harer, J., Reale, C., and Chin, P. (2019). Tree-transformer: A transformer-based method for correction of tree-structured data. find
  5. Hosseini, M. J., Hajishirzi, H., Etzioni, O., and Kushman, N. (2014). Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523–533. find our card
  6. Huang, D., Shi, S., Lin, C.-Y., and Yin, J. (2017). Learning fine-grained expressions to solve math word problems. In EMNLP, pages 805–814. find
  7. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016). Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651. ↗ our card
  8. Kingma, D. P. and Ba, J. (2017). Adam: A method for stochastic optimization. find
  9. Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. (2016). Mawps: A math word problem repository. In NAACL, June. find
  10. Li, J., Wang, L., Zhang, J., Wang, Y., Dai, B. T., and Zhang, D. (2019a). Modeling intra-relation in math word problems with different functional multi-head attentions. In ACL, pages 6162–6167. find
  11. Li, Y., Zhao, L., Wang, J., and Hestness, J. (2019b). Compositional generalization for primitive substitutions. find
  12. Liang, Z. and Zhang, X. (2021). Solving math word problems with teacher supervision. In IJ-CAI, pages 3522–3528. find
  13. Liang, C.-C., Wong, Y.-S., Lin, Y.-C., and Su, K.-Y. (2018). A meaning-based statistical english math word problem solver. In NAACL, pages 652–662. find
  14. Liu, Q., Guan, W., Li, S., and Kawahara, D. (2019). Tree-structured decoding for solving math word problems. In EMNLP, pages 2370–2379. find
  15. Miao, S.-Y., Liang, C.-C., and Su, K.-Y. (2020). A diverse corpus for evaluating and developing english math word problem solvers. In ACL, pages 975–984. find
  16. Mitra, A. and Baral, C. (2016). Learning to use formulas to solve simple arithmetic problems. In ACL, pages 2144–2153. find our card
  17. Nguyen, T. Q. and Chiang, D. (2017). Transfer learning across low-resource, related languages for neural machine translation. find our card
  18. Patel, A., Bhattamishra, S., and Goyal, N. (2021). Are nlp models really able to solve simple math word problems? find our card
  19. Qin, J., Lin, L., Liang, X., Zhang, R., and Lin, L. (2020). Semantically-aligned universal tree-structured solver for math word problems. In EMNLP. find
  20. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. (2019). Analysing mathematical reasoning abilities of neural models. find our card
  21. Shen, Y. and Jin, C. (2020). Solving math word problems with multi-encoders and multi-decoders. In COLING, pages 2924–2934. find
  22. Shi, S., Wang, Y., Lin, C.-Y., Liu, X., and Rui, Y. (2015). Automatically solving number word problems by semantic parsing and reasoning. In EMNLP, pages 1132–1142. find
  23. Siyam, B., Saa, A. A., Alqaryouti, O., and Shaalan, K. (2017). Arabic arithmetic word problems solver. Procedia Computer Science, 117:153– 160. Arabic Computational Linguistics. find
  24. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. Arabic Computational Linguistics. find
  25. Torrey, L. and Shavlik, J. (2009). Transfer Learning. University of Wisconsin, Madison WI, USA. find
  26. Wang, Y., Liu, X., and Shi, S. (2017a). Deep neural solver for math word problems. In EMNLP, pages 845–854. find
  27. Wang, Y., Liu, X., and Shi, S. (2017b). Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  28. Wang, L., Wang, Y., Cai, D., Zhang, D., and Liu, X. (2018). Translating a math word problem to a expression tree. In EMNLP, pages 1064–1069. find
  29. Wang, L., Zhang, D., Zhang, J., Xu, X., Gao, L., Dai, B. T., and Shen, H. (2019a). Template-based math word problem solvers with recursive neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33:7144–7151, 07. find
  30. Wang, L., Zhang, D., Zhang, J., Xu, X., Gao, L., Dai, B. T., and Shen, H. T. (2019b). Template-based math word problem solvers with recursive neural networks. In AAAI, volume 33, pages 7144–7151. find
  31. Wu, Q., Zhang, Q., Fu, J., and Huang, X.-J. (2020). A knowledge-aware sequence-to-tree network for math word problem solving. In EMNLP, pages 7137–7146. find
  32. Xie, Z. and Sun, S. (2019a). A goal-driven tree-structured neural model for math word problems. In IJCAI, pages 5299–5305. find
  33. Xie, Z. and Sun, S. (2019b). A goal-driven tree-structured neural model for math word problems. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5299–5305. International Joint Conferences on Artificial Intelligence Organization, 7. find
  34. Zhang, J., Lee, R. K.-W., Lim, E.-P., Qin, W., Wang, L., Shao, J., and Sun, Q. (2020a). Teacher-student networks with multiple decoders for solving math word problem. In IJCAI, pages 4011– 4017. find
  35. Zhang, J., Wang, L., Lee, R. K.-W., Bin, Y., Wang, Y., Shao, J., and Lim, E.-P. (2020b). Graph-to-tree learning for solving math word problems. In ACL, pages 3928–3937. find
  36. Zoph, B., Yuret, D., May, J., and Knight, K. (2016). Transfer learning for low-resource neural machine translation. find
  37. Zou, Y. and Lu, W. (2019). Text2math: End-to-end parsing text into math expressions. In EMNLP, pages 5330–5340. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alghamdi-2022-armath-dataset-solving-arabic-math/