Elamur Research
☰
home / papers / 2020 / Injecting Numerical Reasoning Skills into Language Models

Injecting Numerical Reasoning Skills into Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.89.pdf

Summary

Pre-trained language models capture a great deal of linguistic knowledge, but they struggle with arithmetic over text. Earlier systems for benchmarks like DROP compensated by bolting on special-purpose modules, such as counters limited to 0–9 or heads that add or subtract only a few numbers. The authors argue that numerical reasoning lends itself to synthetic data creation, so the skill can instead be taught through extra pre-training. They build GenBERT, a BERT-initialized Transformer encoder-decoder that can either extract answer spans or generate numbers digit by digit. It is trained in two additional stages:

  • Numerical data (ND): automatically produced arithmetic problems.
  • Textual data (TD): passage–question pairs built from a small grammar.

A masked-LM loss is kept alongside these stages so the model retains its language knowledge. The extra pre-training lifts DROP F1 from 49.3 to 72.3, on par with specialized models of similar size. GenBERT also transfers to math word problems and keeps its SQuAD accuracy. The broader takeaway is a generic recipe: any skill whose training examples can be generated automatically can be added to a pre-trained LM without redesigning its architecture.

Key points

  • DROP: GenBERT with ND+TD pre-training reaches 72.3 F1, versus 49.3 without it, using one general-purpose generative architecture instead of task-specific heads.
  • SQuAD v1: GenBERT+ND+TD scores 81.3 EM / 88.6 F1, essentially identical to BERT's 81.1 / 88.6. Removing the MLM objective (GenBERT+ND-LM) costs about 3 EM, which shows that the LM loss prevents forgetting.
  • Drop-in initialization: GenBERT weights also help other models. NABERT+ rises from 63.0 to 65.1 EM on DROP, and MS-TAG from 67.3 to 69.3 EM.
  • Math word problems (AddSub, SoP, SeQ, from MAWPS): ND+TD pre-training generalizes better than either data type alone. However, every model breaks down on expressions with more than 3 terms, and MTMSN still scores higher.
  • Error analysis (100 DROP failures): 43% required skills that are uncovered or non-numerical, such as sorting, and 23% were imprecise predictions. Code and data are available at github.com/ag1988/injecting_numeracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

106 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 99 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.89 · arXiv 2004.04487

@inproceedings{geva2020injecting,
  title = {{Injecting Numerical Reasoning Skills into Language Models}},
  author = {Geva, Mor and Gupta, Ankit and Berant, Jonathan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.89},
  eprint = {2004.04487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.89}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In North American Association for Computational Linguistics (NAACL), pages 2357–2367, Minneapolis, Min… find our card
  2. Daniel Andor, Emily Pitler, Kenton Lee, and Luheng He. 2019. Giving bert a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  4. Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations (ICLR). find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL), pages 4171–4186, Minneapolis, Minnesota. find
  6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL). find our card
  7. Avia Efrat, Elad Segal, and Mor Shoham. 2019. Tag-based multi-span extraction in reading comprehension. arXiv preprint arXiv:1909.13375. — [arXiv:1909.13375](https://arxiv.org/abs/1909.13375) find
  8. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. find
  9. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In North American Association for Computational Linguistics (NAACL), pages 4129–4138. find our card
  10. M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Empirical Methods in Natural Language Processing (EMNLP), pages 523–533. find our card
  11. Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  12. Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. Aishwarya Kamath and Rajarshi Das. 2019. A survey on semantic parsing. Automated Knowledge Base Construction (AKBC). find
  14. Jambay Kinley and Raymond Lin. 2019. NABERT+: Improving Numerical Reasoning in Reading Comprehension. find
  15. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… find
  16. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. In Transactions of the Association for Computational Linguistics (TACL), volume 3, pages 585– 597. MIT Press. find
  17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Association for Computational Linguistics (ACL), pages 1152–1157, San Diego, California. find
  18. Guillaume Lample and FranÃğois Charton. 2020. Deep learning for symbolic mathematics. In International Conference on Learning Representations (ICLR). find
  19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  20. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Computational Linguistics (ACL). find our card
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). find
  23. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  24. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  25. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. As… find our card
  26. S. Roy, T. Vieira, and D. Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics (TACL), 1. find
  27. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Empirical Methods in Natural Language Processing (EMNLP), pages 1743– 1752. find our card
  28. Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196–205, Hong Kong, China. Association for… find
  29. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations (ICLR). find our card
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998–6008. find
  31. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card