Elamur Research⌕
☰
home / papers / 2021 / Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Paper · 2021 · Rodrigo Nogueira, Zhiying Jiang and Jimmy Lin

Summary

Sequence-to-sequence transformers often fail at basic arithmetic. This study asks how much of that failure comes from the way numbers are written in the input. The authors generate synthetic addition and subtraction questions phrased in English and fine-tune pretrained T5 models of several sizes, plus non-pretrained transformers. Outputs are scored by exact-match accuracy. They compare six number formats: DECIMAL, CHARACTER (space-separated digits), FIXED-CHARACTER (zero-padded), UNDERSCORE, WORDS, and two formats that place explicit position markers between digits, 10-BASED and 10E-BASED. The format largely determines success. With subword-tokenized decimals, the model cannot learn to add even five-digit numbers. With position markers, it adds and subtracts numbers of up to 60 digits after seeing few examples. However, no setup generalizes across lengths: models fail on numbers whose lengths differ from the training distribution, regardless of model size or amount of data. The results point to subword tokenizers and positional encodings as weak spots in current transformers. They also show that domain knowledge can be supplied through input formatting alone, without re-pretraining.

Key points

  • Main setup: T5-220M was trained for 100 epochs on 1,000 balanced-sampled examples. Results were averaged over five runs, with sizes T5-60M, T5-220M, T5-770M and T5-3B also tested. Code is released at castorini/transformers-arithmetic.
  • Accuracy by format:
    • DECIMAL drops to zero accuracy at 5+ digits.
    • CHARACTER and UNDERSCORE reach zero at 15 digits.
    • FIXED-CHARACTER gets about 20% at 15 digits.
    • WORDS stays at 40–60% for 5–15 digits but fails at 20 or more.
  • Effect of more data: With 10M training examples, nearly every format exceeds 99.9% accuracy on 30-digit addition. DECIMAL is the exception, still showing a 2.1% error rate.
  • Effect of pretraining: Pretrained T5-220M and T5-3B need roughly ten times fewer examples than from-scratch models to reach 100% on 60-digit addition. On 15-digit addition in bases 2, 3, 10 and 19, pretrained models score 0.976–0.999, while from-scratch models score 0.
  • Length mismatch: A model trained on random sampling (mostly 60-digit numbers) scores only 0.014 on a balanced test set, so it cannot handle shorter numbers. Training on balanced data gives 1.000 on both test sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Rodrigo Nogueira, Zhiying Jiang and Jimmy Lin. 2021. Investigating the Limitations of Transformers with Simple Arithmetic Tasks. arXiv:2102.13019.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.13019 · arXiv 2102.13019

@misc{nogueira2021investigating,
  title = {{Investigating the Limitations of Transformers with Simple Arithmetic Tasks}},
  author = {Rodrigo Nogueira, Zhiying Jiang and Jimmy Lin},
  year = {2021},
  doi = {10.48550/arxiv.2102.13019},
  eprint = {2102.13019},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.13019}
}

References

47 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. Giving BERT a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… — Language Models are Few-Shot Learners
  3. Jui Chu, Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. Learning to generate correct numeric values in news headlines. In Companion Proceedings of the Web Conference 2020, pp. 17–18, 2020. find
  4. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2018. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  6. David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatiotemporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508, 2020. — [arXiv:2012.08508](https://arxiv.org/abs/2012.08508) find
  7. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  8. Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 946–958, July 2020.
  9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
  10. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for auto… — Scaling Laws for Autoregressive Generative Modeling
  11. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
  12. Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3327–3335, 2020. find
  13. Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, Haofen Wang, and Kewei Tu. Learning numeral embeddings. arXiv preprint arXiv:2001.00003, 2019. — [arXiv:2001.00003](https://arxiv.org/abs/2001.00003) find
  14. Devin Johnson, Denise Mak, Andrew Barker, and Lexi Loessberg-Zahl. Probing for multilingual numerical understanding in transformer-based language models. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 184–192, 2020. find
  15. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in Neural Information Processing Systems, 28:190–198, 2015. find
  16. Łukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. arXiv preprint arXiv:1511.08228, 2015. find
  17. Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint arXiv:1507.01526, 2015. find
  18. Guolin Ke, Di He, and Tie-Yan Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020. find
  19. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2019.
  20. Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. Modeling intra-relation in math word problems with different functional multi-head attentions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6162–6167, 2019. find
  21. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models. arXiv preprint arXiv:2005.00683, 2020.
  22. Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. Tree-structured decoding for solving math word problems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.… find
  23. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. find
  24. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. Towards question format independent numerical reasoning: A set of prerequisite tasks. arXiv preprint arXiv:2005.08516, 2020. — [arXiv:2005.08516](https://arxiv.org/abs/2005.08516) find
  25. Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3374–3380, 2019. find
  26. Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. The EOS decision and length extrapolation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 276–291, 2020. find
  27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V…
  28. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020.
  29. Eric Price, Wojciech Zaremba, and Ilya Sutskever. Extensions and limitations of the neural GPU. arXiv preprint arXiv:1611.00736, 2016. find
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  31. Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. NumNet: Machine reading comprehension with numerical reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… find
  32. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 349–361, 2019.
  33. Yuanhang Ren and Ye Du. Enhancing the numeracy of word embeddings: A linear algebraic perspective. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 170–178. Springer, 2020. find
  34. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. Technical report, Institute for Cognitive Science, University of California, San Diego, 1985. find
  35. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2018.
  36. Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv:1910.06611, 2019. — [arXiv:1910.06611](https://arxiv.org/abs/1910.06611) find
  37. Hongjie Shi. A sequence-to-sequence approach for numerical slot-filling dialog systems. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 272–277, 2020. find
  38. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. oLMpics—on what language model pre-training captures. arXiv preprint arXiv:1912.13283, 2019.
  39. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601, 2019.
  40. Avijit Thawani, Jay Pujara, Pedro A. Szekely, and Filip Ilievski. Representing numbers in NLP: a survey and a vision. arXiv preprint arXiv:2103.13136, 2021.
  41. Andrew Trask, Felix Hill, Scott E. Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pp. 8035–8044, 2018. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.
  43. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know numbers? Probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL…
  44. Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. In International Conference on Learning Representations, 2019. find
  45. Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. Do language embeddings capture scales? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 292–299, 2020.
  46. Yanyan Zou and Wei Lu. Quantity tagger: A latent-variable sequence labeling approach to solving addition-subtraction word problems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5246–5251, 2019a. find
  47. Yanyan Zou and Wei Lu. Text2Math: End-to-end parsing text into math expressions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5330–5340, 2019b. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nogueira-2021-investigating-limitations-transformers-simple-arithmetic/