Elamur Research⌕
☰
home / papers / 2022 / Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?

Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?

Paper · 2022 · Danish Pruthi, Rachit Bansal, Bhuwan Dhingra et al.

Summary

Feature-attribution methods that mark certain input tokens as important are widely used to explain NLP models. Yet these methods frequently disagree with one another, and they are usually judged by inspecting a few examples or by metrics that trivial strategies can exploit. This paper treats an explanation as a message from a teacher model to a student model. An explanation's worth is measured by how much it improves the student's ability to reproduce the teacher's predictions on new inputs.

Students learn only from teacher outputs, never from gold labels. Explanations serve as extra training signal but are withheld at test time, which prevents label leakage. Two further constraints stop the student from simply copying the teacher's weights: explanations must be human-readable spans, and students differ architecturally from the teacher.

The authors apply this protocol to text classification, comparing seven attribution techniques, and to question answering, testing both human-written and SpanBERT-generated explanations. Attention-based explanations and integrated gradients help students most, while plain gradient saliency and LIME help least. These rankings largely hold across student designs. The result is a scalable, automatic, model-agnostic way to compare explanation methods.

Key points

  • Core metric: simulation accuracy, the fraction of test inputs where student and teacher predictions agree. An explanation's value is the gain of an explanation-trained student over a student trained on teacher labels alone.
  • Methods compared for classification: seven attribution techniques, spanning gradient-based saliency, integrated gradients, LIME, attention, and methods from Shrikumar et al. (2017) and Dhamdhere et al. (2019).
  • Main findings: attention and integrated gradients rank highest, while vanilla gradients and LIME rank lowest. Rankings show moderate-to-high agreement across student architectures and learning strategies.
  • Question answering: student learners are validated with human explanations (referential equalities) from Lamm et al. (2021) and with automatic explanations from SpanBERT.
  • Comparison with prior evaluation approaches:
    • Human studies by Hase and Bansal (2020) trained participants on only 16 or 32 examples, and the differences they found were not significant.
    • The LAS metric (Hase et al., 2020) and ERASER's comprehensiveness and sufficiency scores are criticized for leakage or distribution-shift problems.
    • Code is available at github.com/danishpruthi/evaluating-explanations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Delhi Technological UniversityNew Delhi · India2 works · 188 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 685 works

Cite

Cite as Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig and William W. Cohen. 2022. Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00465

@article{pruthi2022evaluating,
  title = {{Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?}},
  author = {Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig and William W. Cohen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00465},
  url = {https://doi.org/10.1162/tacl_a_00465}
}

References

36 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Several papers have suggested simulatability as an approach to measure interpretability (Lipton, 2016; Doshi-Velez and Kim, 2017). In a survey on interpretability, Doshi-Velez and Kim (2017) propose the task of forward simulation: Given an input and an explanation, people must predict what a model w… find
  2. Chandrasekaran et al. (2018) conduct human-studies to evaluate if explanations from Visual Question Answering (VQA) models help users predict the output. find
  3. Recently, Hase and Bansal (2020) perform a similar human-study across text and tabular classification tasks. Due to the nature of these two studies, the observed differences with and without explanation, and among different explanation types, were not significant. Conducting large-scale human studie… find
  4. Lastly, Poerner et al. (2018) present a hybrid document classification task, where the sentences are sampled from different documents with different class labels. The evaluation metric validates if the important tokens (as per a given interpretation technique) point to the tokens from the ‘‘right’’… find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR.
  6. Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1903–1913. https://doi.org/10 .18653/v1/D18-1216 find
  7. Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1036–1042. https://doi.org/10 .18653/v1/D1… find
  8. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems, volume 22.
  9. Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. 2020. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1448–1458. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and ByronC.Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458. https://doi.or… ↗
  12. Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. 2019. How important is a neuron. In 7th International Conference on Learning Representations, ICLR. tional Linguistics, 7:453–466. https://doi.org/10.1162/tacl_a_00276 ↗
  13. John Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 18th International Conference on Machine Learning 2001 (ICML 2001), pages 282–289. find
  14. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
  15. Peter Hase and Mohit Bansal. 2020. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540–5552. https://doi.org/10 .18653/v1/2020.acl-main.491
  16. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. QED: A framework and dataset for explanations in question answering. Transactions of the Association for Computational Linguistics, 9:790–806. https://doi.org/10.1162 /tacl_a…
  17. Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4351–4367.
  18. Zachary C. Lipton. 2016. The mythos of model interpretability. ACM Queue, 16(3):31–57. https://doi.org/10.1145/3236386 .3241340 — [doi:10.1145/3236386](https://doi.org/10.1145/3236386) ↗
  19. Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, pages 9737–9748. https://doi.org/10.18653/v1/2020.findings -emnlp.390 — [doi:10.18653/v1/2020.findings](https://doi.org/10.18653/v1/2020.findings) ↗
  20. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150.
  21. Alon Jacovi and Yoav Goldberg. 2021. Aligning Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics, 9:294–310. https://doi.org/10.1162/tacl a 00367 ↗
  22. Hugo Mercier and Dan Sperber. 2017. The Enigma of Reason, Harvard University Press. find
  23. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10 .1162/tacl_a_00300
  24. Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 340–350. https://doi… find
  25. Danish Pruthi, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Weakly-and semi-supervised evidence extraction. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3965–3970. https://doi.org/10.18653/v1/2020 .findings-emnlp.353 ↗
  26. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural que…
  27. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ‘‘why should I trust you?’’: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144.
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  29. Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André F. T. Martins, and Graham Neubig. 2021. Do context-aware translation models pay the right attention? In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co… find
  30. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, volume 70 of Proceedings of Machine Lear…
  31. Omar Zaidan and Jason Eisner. 2008. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 31–40. https://doi.org/10.3115/1613715 .1613721 ↗
  32. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR.
  33. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. find
  34. Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using ‘‘annotator rationales’’ to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Confe…
  35. Marcos V. Treviso and André F. T. Martins. 2020. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2020, Online, November 2020, pages 107–118… find
  36. Ruiqi Zhong, Steven Shao, and Kathleen McKeown. 2019. Fine-grained sentiment analysis with faithful attention. arXiv preprint arXiv:1908.06870. — [arXiv:1908.06870](https://arxiv.org/abs/1908.06870) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pruthi-2022-evaluating-explanations-how-much-do/