Elamur Research⌕
☰
home / papers / 2021 / Explanations for CommonsenseQA: New Dataset and Models

Explanations for CommonsenseQA: New Dataset and Models

Summary

Most work on CommonsenseQA (CQA) aims only at picking the right answer. This paper instead asks how to explain a known gold answer, independent of any model. The authors define four desired qualities of a good explanation: comprehensive, refutation complete, minimal, and coherent. In practice, an explanation becomes a set of atomic commonsense facts. Positive properties support the correct choice, and negative properties rule out each distractor. A free-flow text explanation is added on top.

Using crowdsourcing, they annotated about 11K CQA question–answer pairs in this format, creating the ECQA dataset. They also built two kinds of systems. XR retrieves properties from a fact corpus using latent representations. XG generates explanations with GPT-2: XGP produces properties after two-stage fine-tuning, and XGF writes free-flow text.

XR roughly doubles the F1 of a BM25 baseline. Generation quality is moderate, judged by a similarity metric chosen to agree with human ratings. The resource matters because earlier CoS-E explanations were short and never addressed the wrong options.

Key points

  • ECQA dataset: about 11K CQA pairs annotated with positive properties, negative properties, and free-flow explanations. It is publicly released (github.com/dair-iitd/ECQA-Dataset), along with code and models.
  • XR retrieval: relative F1 gain over BM25 of about 100% with the gold fact corpus and about 70% with a silver corpus.
  • XGP property generation: GPT-2 with two-step fine-tuning reaches F1 36.4 under STS-BERT (32.2 under SPICE). This beats the single-step variant XGP-W (33.0) and the best silver-corpus XR setup (27.6).
  • XGF free-flow generation:
    • XGF-I generates text directly and scores 62.5 STS-BERT.
    • XGF-II first generates properties with XGP and scores 61.9.
    • Bare pre-trained GPT-2 produced unusable output.
  • Comparison with prior datasets: unlike COS-E, WorldTree V2, QASC and OpenBookQA, ECQA combines refutation of incorrect choices, a knowledge corpus of facts, and free-flow explanations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.238

@inproceedings{aggarwal2021explanations,
  title = {{Explanations for CommonsenseQA: New Dataset and Models}},
  author = {Aggarwal, Shourya and Mandowara, Divyanshu and Agrawal, Vishwajeet and Khandelwal, Dinesh and Singla, Parag and Garg, Dinesh},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.238},
  url = {https://doi.org/10.18653/v1/2021.acl-long.238}
}

References

48 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Daniel M. Cer, Mona T. Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2012, pages 385–393. find our card
  2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of NAACL-HLT, pages 2357–2367. find our card
  3. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic propositional image caption evaluation. In Proceedings of ECCV, pages 382–398. find our card
  4. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and ex-Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-a… find our card
  5. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2019. Abductive Commonsense Reasoning. In Proceedings of ICLR. find our card
  6. G. P. Shrivatsa Bhargav, Michael R. Glass, Dinesh Garg, Shirish K. Shevade, Saswati Dana, Dinesh Khandelwal, L. Venkata Subramaniam, and Alfio Gliozzo. 2020. Translucent Answer Predictions in Multi-Hop Reading Comprehension. In Proceedings of AAAI, pages 7700–7707. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal andf Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Wint… find
  8. Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020. Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension. In Proceedings of ICLR. find
  9. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as Soft Reasoners over Language. In Proceedings of IJCAI, pages 3882–3890. find our card
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pages 4171–4186. find
  11. Alvan R Feinstein and Domenic V Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of clinical epidemiology, 43(6):543–549. find
  12. Miguel Angel Rı́os Gaona. 2014. Methods for measuring semantic similarity of texts. Ph.D. thesis, University of Wolverhampton, UK. find
  13. Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. 2018. Generating Natural Language Explanations for Visual Question Answering Using Scene Graphs and Visual Attention. In Proceedings of XAI@ICML. find
  14. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652. — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) find
  15. Helmut Horacek. 2017. Requirements for Conceptual Representations of Explanations and How Reasoning Systems Can Serve Them. In Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017). find
  16. Peter Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton Morrison. 2018. WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference. In Proceedings of LREC. find our card
  17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of ACL, pages 1601–1611. find our card
  18. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing Format Boundaries with a Single QA System. In Findings of ACL: EMNLP 2020, pages 1896–1907. find our card
  19. Generalizable Neuro-Symbolic Systems for Commonsense Question Answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 22–32. find
  20. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. In Proceedings of ACL, pages 975–984. find
  21. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of EMNLP, pages 2381–2391. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. find
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
  24. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In Proceedings of ACL, pages 4932–4942. find our card
  25. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A Dataset for Question Answering via Sentence Composition. In Proceedings of AAAI, pages 8082–8090. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP, pages 2383–2392. find our card
  27. Neema Kotonya and Francesca Toni. 2020. Explainable Automated Fact-Checking for Public Health Claims. In Proceedings of EMNLP, pages 7740– 7754. find our card
  28. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP, pages 3973– 3983. find
  29. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97. find
  30. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc. find
  31. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu… find our card
  32. P SINGH. 2002. The public acquisition of common sense knowledge. In Proceedings of AAAI Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for Information Access, 2002. AAAI. find
  33. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find
  34. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In Proceedings of ACL, pages 158–167. find
  35. Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. 2019. find
  36. Towards Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of NAACL-HLT, pages 4149–4158. find
  37. Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. LC-QuAD: A corpus for complex question answering over knowledge graphs. In Proceedings of ISWC, pages 210–218. find
  38. Christina Unger, Corina Forascu, Vanessa López, Axel Cyrille Ngonga Ngomo, Elena Cabrio, Philipp Cimiano, and Sebastian Walter. 2014. Question Answering over Linked Data (QALD-4). In Proceedings of CLEF. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of NeurIPS, pages 5998– 6008. find
  40. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of CVPR, pages 4566–4575. find our card
  41. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation. In Proceedings of ACL, pages 4020– 4026. find
  42. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing Multiple Choice Science Questions. In Proceedings of NUT@EMNLP, pages 94–106. find our card
  43. Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter Jansen. 2020. WorldTree V2: A Corpus of Science-Domain Structured Explanations and Inference Patterns supporting Multi-Hop Inference. In Proceedings of LREC, pages 5456–5473. find
  44. Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020. Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering. In Proceedings of ACL, pages 4514–4525. find
  45. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative Data Augmentation for Commonsense Reasoning. In Findings of ACL: EMNLP 2020, pages 1008–1025. find our card
  46. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of EMNLP, pages 2369–2380. find our card
  47. Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2020. Retrospective Reader for Machine Reading Comprehension. CoRR, abs/2001.09694. — [arXiv:2001.09694](https://arxiv.org/abs/2001.09694) find
  48. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. FreeLB: Enhanced Adversarial Training for Natural Language Understanding. In Proceedings of ICLR. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/