Elamur Research
☰
home / papers / 2019 / Explain Yourself! Leveraging Language Models for Commonsense Reasoning

Explain Yourself! Leveraging Language Models for Commonsense Reasoning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1487.pdf

Summary

Neural models still struggle with commonsense questions, because answering them often needs world knowledge that is not in the input itself. The authors crowdsourced human justifications on Amazon Mechanical Turk for the random split of CommonsenseQA (CQA) and released them as the Common Sense Explanations (CoS-E) dataset. Each entry has two parts: words highlighted in the question, and a short free-text rationale. They then built CAGE (Commonsense Auto-Generated Explanations), a two-stage pipeline. First, a pretrained language model is fine-tuned to write a CoS-E-style explanation from a question and its answer options. Second, the generated explanation is appended to the input of a separate commonsense reasoning classifier, which picks the answer. CAGE reaches about 65% accuracy on CQA v1.0, roughly 10% better than the strongest earlier baseline. This shows that language models are more helpful on this task when they produce explanations than when they predict answers directly. The paper also tests transfer to other datasets and notes gender bias carried over from CQA.

Key points

  • CoS-E covers 7610 train and 950 dev examples for CQA v1.0 (3 choices), and 9741 train and 1221 dev examples for v1.11 (5 choices). It is available at github.com/nazneenrajani/CoS-E.
  • Quality control included in-browser checks: annotators had to highlight words, write at least 4 words, and not just copy the question or the answers. Template-like explanations were removed and annotated again.
  • About 58% of the explanations contain the correct answer. Even explanations that share no words with any answer option still beat baselines trained without CoS-E.
  • Unlike e-SNLI, where explaining lowered accuracy, CQA performance went up in the explain-then-predict setting. The gains were weaker on the harder 5-choice v1.11.
  • The explanation generator was applied to SWAG and Story Cloze without further training. The explanations looked fluent and relevant, but accuracy dropped slightly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

305 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 280 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1487 · arXiv 1906.02361

@inproceedings{rajani2019explain,
  title = {{Explain Yourself! Leveraging Language Models for Commonsense Reasoning}},
  author = {Rajani, Nazneen Fatema and McCann, Bryan and Xiong, Caiming and Socher, Richard},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1487},
  eprint = {1906.02361},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1487}
}

References

29 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). find
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP2015), pages 632–642. find
  3. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural Language Inference with Natural Language Explanations. In Advances in Neural Information Processing Systems (NeurIPS2018), pages 9560– 9572. find our card
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP2017), pages 670–680. find our card
  5. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS2015), pages 3079–3087. MIT Press. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL2018), volume 1, pages 1884–1895. find
  8. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  9. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages… find
  10. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  11. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. find our card
  12. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  13. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual meeting on Association for Computational Linguistics (ACL2002), pages 311–318. Association for Computational Linguistics. find our card
  15. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP2014), pages 1532–1543. find
  16. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL2018), pages 328–339. find
  17. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  18. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP2016), pages 107–117. find
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/ research-covers/language-unsupervised/ language understanding paper.pdf. ↗
  20. Nazneen Fatema Rajani and Raymond Mooney. 2018. Stacking with auxiliary features for visual question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2… find
  21. Nazneen Fatema Rajani and Raymond J. Mooney. 2017. Ensembling visual explanations for vqa. In Proceedings of the NIPS 2017 workshop on Visually-Grounded Interaction and Language (ViGIL). find
  22. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. CoRR, abs/1611.01603. find our card
  23. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  24. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS2017), pages 5998–6008. find
  26. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  27. Terry Winograd. 1972. Understanding natural language. Cognitive psychology, 3(1):1–191. find
  28. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP2018), pages 93–104. find
  29. Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018. Improving question answering by commonsense-based pre-training. arXiv preprint arXiv:1809.03568. — [arXiv:1809.03568](https://arxiv.org/abs/1809.03568) find