Elamur Research
☰
home / papers / 2019 / Do NLP Models Know Numbers? Probing Numeracy in Embeddings

Do NLP Models Know Numbers? Probing Numeracy in Embeddings

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1534.pdf

Summary

The paper asks whether standard neural NLP models, which embed numbers like ordinary tokens, actually represent numerical value (numeracy). The authors first analyze NAQANet, the leading reading comprehension model on DROP. Using question templates, they isolate the comparative and superlative questions from the validation set. NAQANet scores far better on these questions than on the dataset overall. However, it degrades when passage numbers are enlarged or written as words.

To find where this ability comes from, they train probing networks on synthetic tasks: list maximum, number decoding, and addition. They apply these probes to Word2Vec, GloVe, ELMo, BERT, and Char-CNN/Char-LSTM models trained directly on the tasks. Pre-trained embeddings turn out to encode magnitude surprisingly well, including for floats and negative numbers. Character-level representations are the most precise, with ELMo leading among pre-trained models and sub-word BERT trailing behind. All models extrapolate poorly beyond the training range. The work matters because it shows numeracy emerging without supervision and locates its limits, pointing to data augmentation as a practical fix.

Key points

  • NAQANet, which uses GloVe plus a character-level CNN, gets only 49.2 F1 on the full DROP validation set. On binary comparative questions it reaches 89.0 F1, and it scores 69.2 F1 on superlatives with number answers and 66.3 F1 on argmax-style span answers.
  • In the stress tests, multiplying passage numbers by a value in [11, 100] lowers superlative F1 from 67.7 to 32.0. Converting digits to words also hurts the model.
  • GloVe and word2vec reliably encode the magnitude of numbers up to about 1,000.
  • On list maximum with negative integers in [-50, 50], ELMo scores 0.96, BERT 0.94, and Char-LSTM 0.97, against 0.23 for random vectors.
  • For extrapolation, probes were trained on [0, 150] and tested on [151, 200]. Accuracy was 0.82 for the learned Char-LSTM, 0.38 for ELMo, and 0.21 for GloVe. Augmenting DROP with Add/Multiply-modified examples can roughly double superlative accuracy on passages with large numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1534 · arXiv 1909.07940

@inproceedings{wallace2019do,
  title = {{Do NLP Models Know Numbers? Probing Numeracy in Embeddings}},
  author = {Wallace, Eric and Wang, Yizhong and Li, Sujian and Singh, Sameer and Gardner, Matt},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1534},
  eprint = {1909.07940},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1534}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Finegrained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR. find
  2. Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the origins of bias in word embeddings. In ICML. find
  3. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL. find our card
  4. Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word embeddings. In ACL. find
  5. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. find
  6. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics. find
  7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  9. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. arXiv preprint arXiv:1901.03735. find our card
  10. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. find our card
  11. Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. 2011. On random weights and unsupervised feature learning. In ICML. find
  12. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL. find
  13. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  14. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In ACL. find
  15. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In ICML. find our card
  16. Bhushan Kotnis and Alberto Garcı́a-Durán. 2019. Learning numerical attributes in knowledge bases. In AKBC. find
  17. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP. find
  18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In TACL. find our card
  19. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. In NAACL. find our card
  20. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In LREC. find our card
  21. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. find our card
  22. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. find our card
  23. Georgios Spithourakis and Sebastian Riedel. 2018. Numeracy for language models: Evaluating and improving their ability to predict numbers. In ACL. find
  24. Andrew Trask, Felix Hill, Scott Reed, Jack W. Rae, Chris Dyer, and Phil Blunsom. 2018. Neural arithmetic logic units. In NeurIPS. find
  25. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In ICLR. find
  26. Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In ICLR. find
  27. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision rnns for language recognition. In ACL. find
  28. Yiben Yang, Larry Birnbaum, Ji-Ping Wang, and Doug Downey. 2018. Extracting commonsense properties from embeddings with limited human guidance. In ACL. find
  29. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In ICLR. find our card