Elamur Research⌕
☰
home / papers / 2019 / How Pre-trained Word Representations Capture Commonsense Physical Comparisons

How Pre-trained Word Representations Capture Commonsense Physical Comparisons

Paper · 2019 · Pranav Goel, Shi Feng and Jordan Boyd-Graber

Summary

The paper asks whether frozen pre-trained word representations already contain commonsense knowledge about physical comparisons, such as whether one object is larger, heavier or faster than another. For evaluation, the authors rely on Verb Physics, a resource released by Forbes and Choi (2017) that spans five attributes: size, weight, strength, rigidness and speed. They build a deliberately minimal probe. GloVe, ELMo or BERT embeddings of two words are concatenated or subtracted and passed to a linear classifier or a network with one hidden layer. The classifier predicts <, > or ≈. Feeding the reversed word pair during both training and testing improves results.

Using only the two words, this probe matches or beats earlier systems that relied on verbs or semantic "poles". It also generalizes to objects it never saw in training. Further analysis suggests the models arrange all objects along a consistent scale and compare them through that ranking. The work shows that static and contextual embeddings encode usable physical common sense, and that strong scores are not just a result of memorizing individual words.

Key points

  • On Verb Physics, overall accuracy is 0.77 for GloVe, 0.78 for ELMo and 0.75 for BERT-base. Baselines score lower: PCE (Yang et al., 2018) reaches 0.76, F&C 0.71 and the majority-class baseline 0.64. GloVe nearly equals ELMo and outperforms BERT.
  • Generalization was tested on the EB evaluation set of Bagherinezhad et al. (2016), with 486 size comparisons. Only about 33% of its objects appear in training. Results were 0.879 for GloVe, 0.905 for ELMo and 0.893 for BERT, compared with 0.835 for an earlier visual+textual model.
  • Baselines that see only one word, or that use cosine similarity, perform clearly worse. This rules out simple lexical memorization as the explanation.
  • Local orderings built from logit differences are fully consistent for GloVe and BERT, and over 90% consistent for ELMo. A linear model's weights produce a global ranking of objects that classifies pairs about as well as the original models.
  • The ≈ label is handled poorly, with per-label accuracy of only 0.12–0.33. The likely causes are that this label makes up only about 10% of the training data and that an ordering cannot easily express near-equality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States151 works · 29,733 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Pranav Goel, Shi Feng and Jordan Boyd-Graber. 2019. How Pre-trained Word Representations Capture Commonsense Physical Comparisons. Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-6016

@inproceedings{goel2019how,
  title = {{How Pre-trained Word Representations Capture Commonsense Physical Comparisons}},
  author = {Pranav Goel, Shi Feng and Jordan Boyd-Graber},
  booktitle = {Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-6016},
  url = {https://doi.org/10.18653/v1/d19-6016}
}

References

21 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207. \#\# Conclusion A linear or a small fully connected neural network probing model can compare two words on comm…
  2. Hessam Bagherinezhad, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. 2016. Are elephants bigger than butterflies? reasoning about sizes of objects. In Thirtieth AAAI Conference on Artificial Intelligence. find
  3. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology?
  4. Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930–2939. find
  5. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings f…
  6. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  8. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo…
  9. Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. arXiv preprint arXiv:1906.01327. find
  10. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509.
  11. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 266–276. find
  12. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lear…
  13. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899.
  14. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25–30. ACM. find
  15. Benjamin Van Durme. 2010. knowledge from text. find
  16. Extracting implicit Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  17. Yiben Yang, Larry Birnbaum, Ji-Ping Wang, and Doug Downey. 2018. Extracting commonsense properties from embeddings with limited human guidance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 644–649.
  18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  19. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395.
  20. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goel-2019-how-pre-trained-word-representations/