Elamur Research⌕
☰
home / papers / 2019 / Do Neural Language Representations Learn Physical Commonsense?

Do Neural Language Representations Learn Physical Commonsense?

Paper · 2019 · Maxwell Forbes, Ari Holtzman and Yejin Choi

Summary

This paper asks how much physical commonsense large text-trained language representations actually acquire. The authors define physical knowledge as links among objects, their static properties (e.g., edible, rigid), and their affordances, meaning the actions people typically perform with them. They build two resources with over 200k new crowd annotations. The abstract dataset is a cleaned and fully re-annotated version of the McRae norms. The situated dataset labels properties and affordances for objects pictured in MS COCO images. GloVe, ELMo and BERT are tested on three binary compatibility tasks: object–property (O↔P), object–affordance (O↔A) and affordance–property (A↔P). The models guess object properties and affordances fairly well but struggle to connect affordances with properties. These are implications people rarely state outright, such as "something you plug in uses electricity." Performance also does not track how frequently words appear in text. The authors conclude that new model designs, rather than more data, are needed for this kind of reasoning.

Key points

  • Abstract dataset: 514 objects × 50 properties, giving 25,700 pairs. Each pair was annotated three times (77,100 labels), and only labels with ≥2/3 agreement were kept. The split is 411 train / 103 test objects.
  • Situated dataset: 1,024 objects from 220 MS COCO images, each containing 3–7 objects. It covers 80 categories (64 train / 16 test), with 3 affordances per object and 156,672 annotations.
  • Task setup: Each task is a series of pairwise yes/no compatibility decisions. This worked better than predicting a full property vector from the object alone.
  • Results by property type: Perceptual properties showed the weakest and least consistent results. Encyclopedic and commonsense properties fell in between.
  • Frequency analysis: Using Google Web 1T n-gram counts, the authors found no clear correlation between word frequency and F1. Data and code are available at mbforbes.github.io/physical-commonsense.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Maxwell Forbes, Ari Holtzman and Yejin Choi. 2019. Do Neural Language Representations Learn Physical Commonsense?. arXiv:1908.02899.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.02899 · arXiv 1908.02899

@misc{forbes2019do,
  title = {{Do Neural Language Representations Learn Physical Commonsense?}},
  author = {Maxwell Forbes, Ari Holtzman and Yejin Choi},
  year = {2019},
  doi = {10.48550/arxiv.1908.02899},
  eprint = {1908.02899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.02899}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 18327–18332. find
  2. Brants, T., & Franz, A. (2006). Web 1t 5-gram version 1. find
  3. Devereux, B. J., Tyler, L. K., Geertzen, J., & Randall, B. (2014). The centre for speech, language and the brain (cslb) concept property norms. Behavior research methods, 46(4), 1119–1127. find
  4. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  5. Gaver, W. W. (1991). Technology affordances. In Proceedings of the sigchi conference on human factors in computing systems (pp. 79–84). find
  6. Gerstenberg, T., Zhou, L., Smith, K. A., & Tenenbaum, J. B. (2017). Faulty towers: A hypothetical simulation model of physical support. Proceedings of the 39th Annual Confer-on natural language processing (volume 2: Short papers) ence of the Cognitive Science Society. (Vol. 2, pp. 726–730). find
  7. Van Durme, B. D. (2010). Extracting implicit knowledge Gibson, J. J. (1966). The senses considered as perceptual from text. University of Rochester. systems. find
  8. Wilson, M. (2002). Six views of embodied cognition. find
  9. Psy-Gover, M. R. (1996). The embodied mind: Cognitive science chonomic bulletin & review, 9(4), 625–636. and human experience (book). Mind, Culture, and Activity, Wu, J., Yildirim, I., Lim, J. J., Freeman, B., & Tenenbaum, 3(4), 295–299. find
  10. J. (2015). Galileo: Perceiving physical object properties Herbelot, A., & Vecchi, E. M. (2015). From concepts to by integrating a physics engine with deep learning. In Ad-models: some issues in quantifying feature norms. In Lilt vances in neural information processing systems (pp. 127– (Vol. 2). 135… find
  11. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochas-Yatskar, M., Zettlemoyer, L., & Farhadi, A. (2016). Situa-tic optimization. arXiv preprint arXiv:1412.6980. tion recognition: Visual semantic role labeling for image Lake, B. M., Linzen, T., & Baroni, M. (2019). Human few-understanding. In P…
  12. Levesque, H. J., Davis, E., & Morgenstern, L. (2012). The winograd schema challenge. In Proceedings of the thirteenth international conference on principles of knowledge representation and reasoning (pp. 552–561). AAAI Press. Retrieved from http://dl.acm.org/citation.cfm?id=3031843.3031909 ↗
  13. Levy, O., & Goldberg, Y. (2014). Dependency-based word embeddings. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers) (Vol. 2, pp. 302–308). find
  14. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., . . . Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740–755).
  15. Lucy, L., & Gauthier, J. (2017). Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning. arXiv preprint arXiv:1705.11168. find
  16. McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4), 547–559. find
  17. Norman, D. (1988). The design of everyday things: Revised and expanded edition. Constellation. find
  18. Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) (pp. 1532–1543).
  19. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long p…
  20. Rubinstein, D., Levi, E., Schwartz, R., & Rappoport, A. (2015). How well do distributional models capture different types of semantic knowledge? In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/forbes-2019-do-neural-language-representations-learn/