Elamur Research⌕
☰
home / papers / 2019 / Evaluating Commonsense in Pre-trained Language Models

Evaluating Commonsense in Pre-trained Language Models

Paper · 2019 · Xuhui Zhou, Yue Zhang, Leyang Cui et al.

Summary

Pre-trained contextualized encoders are known to capture syntax, semantics and word senses, but nobody had systematically measured how much everyday commonsense they hold. The authors probe off-the-shelf GPT, GPT2, BERT, XLNet and RoBERTa without any task-specific training. They recast seven benchmarks into a single format: Sense Making, Sense Making with Reasoning, Winograd Schema Challenge, SWAG, HellaSwag, Argument Reasoning Comprehension, and a newly built Conjunction Acceptability task. In each instance, one sentence is sensible and the others are not, and a model succeeds if its LM score favors the sensible one. Some tests change a single key word, while others replace a whole clause. Every model outperforms chance. Bidirectional models and those trained on larger corpora score higher, but accuracy falls as the required number of reasoning steps grows. A robustness check uses minimally edited "dual" instances, and the models seldom answer such pairs consistently. This points to shallow rather than deep commonsense. The suite is released as CATs, giving a cheap zero-shot probe for comparing language models.

Key points

  • Five model families are tested: GPT, GPT2, BERT, XLNet and RoBERTa, in base/large or base/medium sizes where available. They are evaluated on seven tasks reframed as sentence scoring.
  • Conjunction Acceptability is new. It takes 189 positive WSC sentences and manually substitutes the conjunction, using "because", "before", "when", "but" and "and" to probe logical relations such as cause, precondition and contradiction.
  • WSC contributes 273 pronoun-resolution instances. Each candidate noun is inserted in place of the pronoun, which yields positive and negative sentences.
  • The dual-instance robustness test has four edit methods: Add, Del, Swap and Sub. Each method has 75 instances drawn from WSC, SM and ARCT. Consistency mostly stays below the 0.50 random baseline; the best Sub result is RoBERTa-large at 0.44, and the best Swap result is 0.55.
  • The data and test script (CATs) are available at github.com/XuhuiZhou/CATS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,763 citations🎓Westlake UniversityHangzhou · China51 works · 8,572 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Xuhui Zhou, Yue Zhang, Leyang Cui and Dandan Huang. 2019. Evaluating Commonsense in Pre-trained Language Models. arXiv:1911.11931.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.11931 · arXiv 1911.11931

@misc{zhou2019evaluating,
  title = {{Evaluating Commonsense in Pre-trained Language Models}},
  author = {Xuhui Zhou, Yue Zhang, Leyang Cui and Dandan Huang},
  year = {2019},
  doi = {10.48550/arxiv.1911.11931},
  eprint = {1911.11931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.11931}
}

References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
  2. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo…
  3. Habernal, I.; Wachsmuth, H.; Gurevych, I.; and Stein, B. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  4. Kim, N.; Patel, R.; Poliak, A.; Xia, P.; Wang, A.; McCoy, T.; Tenney, I.; Ross, A.; Linzen, T.; Van Durme, B.; Bowman, S. R.; and Pavlick, E. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computat…
  5. Lai, G.; Xie, Q.; Liu, H.; Yang, Y.; and Hovy, E. H. 2017. Race: Large-scale reading comprehension dataset from examinations. In EMNLP.
  6. Levesque, H. J.; Davis, E.; and Morgenstern, L. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, 552–561. AAAI Press. find
  7. Liu, H., and Singh, P. 2004. Conceptnet — a practical commonsense reasoning tool-kit. BT Technology Journal 22:211–226. find
  8. Liu, N. F.; Gardner, M.; Belinkov, Y.; Peters, M. E.; and Smith, N. A. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
  9. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019b. Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692.
  10. LoBue, P., and Yates, A. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 329–334.
  11. Niven, T., and Kao, H.-Y. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4658–4664. find
  12. Peters, M.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P…
  13. Radford, A., and Sutskever, I. 2018. Improving language understanding by generative pre-training. find
  14. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. find
  15. Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 784–789.
  16. Tenney, I.; Xia, P.; Chen, B.; Wang, A.; Poliak, A.; McCoy, R. T.; Kim, N.; Durme, B. V.; Bowman, S. R.; Das, D.; and Pavlick, E. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. ArXiv abs/1905.06316.
  17. Trinh, T. H., and Le, Q. V. 2018. A simple method for commonsense reasoning. CoRR abs/1806.02847. find
  18. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353– 355.
  19. Wang, C.; Liang, S.; Zhang, Y.; Li, X.; and Gao, T. 2019. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4020–4026.
  20. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J. G.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR abs/1906.08237.
  21. Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 93–104. find
  22. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4791–4800. find
  23. Zhu, Y.; Kiros, R.; Zemel, R. S.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV) 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2019-evaluating-commonsense-pre-trained-language/