Elamur Research
☰
home / papers / 2019 / MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms

MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1245.pdf

Summary

The authors take the noisy AQuA (over 100K multiple-choice problems at the GRE/GMAT level) and rewrite its usable portion in a formal language of "operation programs". A program is a chain of steps op(arguments), where an argument can be a number from the problem statement, a category constant (e.g., π), or the result of a previous step (#0, #1…). The language has 58 operations, grouped by domain (geometry, physics, probability, gain-loss, general, other). The annotation was done by crowdworkers on a dynamic platform: choosing an operation, choosing an argument from an allowed list, automatically computing the intermediate value, and checking that the final result is close to the correct answer. The result is MathQA: about 37K problems with programs. The model is Seq2prog (an attention-based LSTM encoder-decoder, OpenNMT), plus a category-aware variant: an n-gram lexicon selects the domain, and the domain hard-switches the decoder parameters. For the subtask tree, the paper is valuable as a ready-made corpus of "operation + references to previous results" steps — essentially a computation DAG that the kernel can execute exactly.

Key points

  • Features: text tokens; the output vocabulary is split into operations and arguments; numbers are represented not by their values but by their source (constants, numbers from the problem statement in order of appearance, intermediate results in order of steps); domain category from n-gram frequencies.
  • Architecture: a two-layer BiRNN/LSTM encoder and an LSTM decoder with attention, d=100, Adam lr 0.001; a categorized decoder LSTM_c; beam 200 (AQuA) and 100 (MathQA); among the executed programs, the closest answer option is chosen.
  • Numbers (Table 3, accuracy): MathQA — Seq2prog 51.9, Seq2prog+cat 54.2, random choice 20.0; AQuA — the Ling et al. model 36.4, Seq2prog 33.0, Seq2prog+cat 37.9.
  • Dataset: 37,259 problems, 5.3 operations per problem on average; 80/12/8 split; program validity according to validators 94.64%, expert check of 500 problems — 92% valid.
  • Limitations: errors — long reasoning chains (45%), operations missing from the language (25%, e.g., factorization), wrong category (12.5%), garbled problem text (17.5%); the categorizer agrees with humans 74.5% of the time.
  • For the subtask tree: explicit references to previous steps provide ready-made dependency edges; the vocabulary of 58 operations is a good starting point for a node inventory.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

393 papers of our library cite it (of the 140673 whose bibliographies we read); 44 of them have a page here, 349 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1245 · arXiv 1905.13319

@inproceedings{amini2019mathqa,
  title = {{MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms}},
  author = {Amini, Aida and Gabriel, Saadia and Lin, Shanchuan and Koncel-Kedziorski, Rik and Choi, Yejin and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1245},
  eprint = {1905.13319},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1245}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Conclusion In this work, we introduced a representation language and annotation system for large-scale math word problem-solving datasets that addresses unwanted noise in these datasets and lack of formal operation-based representations. We demonstrated the effectiveness of our representation lan… find
  2. Yefim Bakman. 2007. Robust understanding of word problems with extraneous information. In arXiv preprint math/0701393. find
  3. Daniel G Bobrow. 1964. Natural language input for a computer problem solving system. find
  4. Diane J Briars and Jill H Larkin. 1984. An integrated model of skill in solving elementary word problems. In Cognition and instruction 1(3), pages 245–296. find
  5. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In NAACL. find
  6. Eugene Charniak. 1968. Calculus word problems. Ph.D. thesis, Massachusetts Institute of Technology. find
  7. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL). find
  8. Eugene Charniak. 1969. Computer solution of calculus word problems. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 303–316. Morgan Kaufmann Publishers Inc. find
  9. Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata. 2017. Learning structured natural language representations for semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 44–55. Association for Computatio… find
  10. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoderdecoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111. find
  11. Denise Dellarosa. 1986. A computer simulation of childrens arithmetic word-problem solving. In Behavior Research Methods, Instruments, Computers 18(2), pages 147–154. find
  12. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2017. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, and Keh-Yih Su. 2018. A meaning-based statistical english math word problem solver. In Proceedings of NAACL-HLT 2018. find
  14. Christian Liguda and Thies Pfeiffer. 2012. Modeling math word problems with augmented semantic networks. In International Conference on Application of Natural Language to Information Systems, pages 247–252. Springer. find
  15. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of ACL. find
  16. Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay. 2016. Neural generation of regular expressions from natural language with minimal domain knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 247–2… find
  17. Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. In Proceedings of NAACL-HLT 2018. find our card
  18. Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. 2018a. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 213–223. find
  19. Shyam Upadhyay and Ming-Wei Chang. 2017. Annotating derivations: A new evaluation strategy and dataset for algebra word problems. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find
  20. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). find
  21. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  22. Danqing Huang, Jin-Ge Yao, Chin-Yew Lin, Qingyu Zhou, and Jian Yin. 2018b. Using intermediate representations to solve math word problems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In the 3rd International Conference for Learning Representations (ICLR). find
  24. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. Opennmt: Open-source toolkit for neural machine translation. ArXiv e-prints. find our card
  25. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, pages 1050–1055. AAAI Press/MIT Press. find
  26. Luke Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI 05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, pages 658–666. find
  27. Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed ccg grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLPCoNLL), pages 678–687. Association for Co… find