Elamur Research⌕
☰
home / papers / 2021 / QED: A Framework and Dataset for Explanations in Question Answering

QED: A Framework and Dataset for Explanations in Question Answering

Paper · 2021 · Matthew Lamm, Jennimaria Palomaki, Chris Alberti et al.

Summary

Neural QA systems reach high accuracy but rarely show how they arrive at an answer, which limits trust and makes them harder to debug. The authors define QED, a linguistically motivated format for explaining extractive answers. An explanation consists of three parts. First, it picks one passage sentence that implies the answer. Second, it links noun phrases in the question and that sentence that denote the same entity or concept, including coreference and bridging cases. Third, it confirms that, once those shared referents are abstracted out, the sentence's remaining predicate entails the question's predicate. Experts applied this scheme to a subset of Google Natural Questions, and the resulting corpus is public. The paper reports baselines for two tasks: explaining an answer that is already given, and producing an answer and explanation together. In the joint setting, a modest amount of QED training data appears to improve answer accuracy. A large user study found that showing QED explanations helps non-expert raters catch mistakes made by a strong neural QA model.

Key points

  • The dataset is built on Natural Questions and contains 7,638 training and 1,353 dev examples. Three annotators labelled them without replication, spending about 2 minutes per item.
  • About 10% of the examined NQ short answers turned out to be wrong. Among the valid ones, 77% could receive a single-sentence QED explanation.
  • Explanations rest on referential equality, sentencehood, and predicate entailment. Bridging anaphora are handled through an implicit-modifier notation.
  • Two baseline tasks are evaluated: post-hoc explanation generation and joint QA plus explanation.
  • The resource is released at github.com/google-research-datasets/QED. The authors outline extensions to multi-hop, yes/no, and set-valued questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Stanford UniversityStanford · United States310 works · 284,480 citations🎓Columbia UniversityNew York · United States149 works · 48,852 citations🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 242 works

Cite

Cite as Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares and Michael Collins. 2021. QED: A Framework and Dataset for Explanations in Question Answering. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00398

@article{lamm2021qed,
  title = {{QED: A Framework and Dataset for Explanations in Question Answering}},
  author = {Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares and Michael Collins},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00398},
  url = {https://doi.org/10.1162/tacl_a_00398}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Herbert H. Clark and Catherine R. Marshall. 1981. Definite reference and mutual knowledge. Elements of Discourse Understanding. find
  2. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa…
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10 .1162/tacl a 00300
  5. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
  6. Manfred Krifka. 2003. Bare NPS: Kind-referring, indefinites, both, or neither? Semantics and Linguistic Theory, 13:180–203. find
  7. Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. 2019. Automated rationale generation: A technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263–274. ACM. https://d… — [doi:10.1145/3301275](https://doi.org/10.1145/3301275) find
  8. Sawan Kumar and Partha Talukdar. 2020. Nile: Natural language inference with faithful natural language explanations. https://doi.org/10.3765/salt.v13i0.2880 ↗
  9. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  10. Andrew Gelman and Jennifer Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge University Press. https://doi.org/10.1017/CBO9780 511790942 — [doi:10.1017/cbo9780](https://doi.org/10.1017/cbo9780) ↗
  11. Ben Goodrich, Jonah Gabry, Imad Ali, and Sam Brilleman. 2020. rstanarm: Bayesian Applied Regression Modeling via Stan. R package version 2.19.3. find
  12. Matthew Lamm, Arun Chaganty, Christopher D. Manning, Dan Jurafsky, and Percy Liang. 2018. Textual analogy parsing: What’s shared and what’s compared among analogous facts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 82–92, Brussels, Belgium. Assoc… find
  13. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics. https://doi.org/10.3115/992133.992154 ↗
  14. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics. https://d…
  15. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end ution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. find
  16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/D16-1264 ↗
  17. Peter Lipton. 2001, What good is an explanation?, Giora Hon and Sam S. Rakover, editors, Explanation: Theoretical Approaches and Applications, Springer Netherlands, Dordrecht, pages 43–59. https://doi.org/10 .1007/978-94-015-9731-9 2 find
  18. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. https://doi.org/10.1162/tacl a 00266 ↗
  19. Line Mikkelsen. 2011. Copular clauses. In Claudia Maienborn, Klaus von Heusinger, and Paul Portner, editors, Semantics: An International Handbook of Natural Language Meaning, 2, pages 1805–1829, Berlin. Mouton De Gruyter. find
  20. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to AI complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8722–8731. https://doi.org/10.1609/aaai.v34i05.6398 — [doi:10.1609/aaai.v34i05.6398](https://doi.org/10.1609/aaai.v34i05.6398) ↗
  21. Eleni Miltsakaki, Rashmi Prasad, Aravind K. Joshi, and Bonnie L. Webber. 2004. The Penn Discourse Treebank. In LREC.
  22. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206–4216, Online… find
  23. Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2662–2670. find
  24. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! Training text-to-text models to explain their predictions. find
  25. Bertrand Russell. 1905. On denoting. Mind, 14(56):479–493. https://doi.org/10 .1093/mind/XIV.4.479 find
  26. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for ‘‘What if...’’ reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  27. Judea Pearl. 2019. The limitations of opaque learning machines. Possible Minds: Twenty-Five Ways of Looking at AI13–19. find
  28. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in Onto-Notes. In EMNLP-CoNLL Shared Task. find
  29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  30. Michael Tomasello, Malinda Carpenter, and Ulf Liszkowski. 2007. A new look at infant pointing. Child Development, 78(3):705–722. https://doi.org/10.1111/j.1467 -8624.2007.01025.x, PubMed: 17516997 — [doi:10.1111/j.1467](https://doi.org/10.1111/j.1467) ↗
  31. We now describe the form of the e and a annotations, making reference to the following example. Subscripts indicate token positions: Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/