Elamur Research⌕
☰
home / papers / 2020 / The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

Paper · 2020 · Jasmijn Bastings and Katja Filippova

Summary

This position paper asks why the NLP community keeps arguing over whether attention weights count as explanations, when an established family of tools already exists for the job attention is usually asked to do. The authors survey the attention-as-explanation debate, which mostly concerns BiLSTM text classifiers with a single MLP-based attention layer. They observe that the unstated aim is almost always to find which input tokens mattered most for a prediction. They also note that the implied audience is a model developer, for whom faithfulness matters more than plausibility. Against this, they review input saliency methods built for faithful per-token attribution: gradient-based methods (plain gradients, gradient×input, Integrated Gradients), propagation-based LRP, and occlusion-based erasure or omission. The paper offers no new data or experiments; its contribution is an argument. For this goal and user, they find no strong reason to prefer attention, so research effort should shift toward saliency. They also urge authors to state explicitly who an explanation is for and what it should achieve. Saliency's own weaknesses are acknowledged as well.

Key points

  • The debate centres on single-sequence text classification with BiLSTMs. Yet attention matters far less there: random weights barely hurt classification, while uniform weights substantially degrade NMT. Conclusions may therefore not transfer to tasks like NLI or MT.
  • Even for MT, where attention was first visualised as word alignment, Ding et al. (2019) report that saliency methods give better word alignments.
  • Saliency methods are grouped into three families, following Arras et al. (2019):
    • Gradient-based: gradients, gradient×input, and Integrated Gradients, which uses a baseline such as zero or [MASK] vectors.
    • Propagation-based: LRP, reported to beat gradient methods on text classification.
    • Occlusion-based: erasure, or omission, which needs n+1 forward passes.
  • The authors rebut practical arguments for attention. Computing input gradients takes one line in TensorFlow, and most saliency methods need just one forward and one backward pass.
  • Known saliency limitations include:
    • Erasure-style corruption can push inputs off the training manifold, so a drop in probability may reflect an out-of-distribution input rather than a missing important feature.
    • Some methods behave unreliably or violate axioms.
    • A flat set of per-token scores is a very narrow kind of explanation.

Interesting ideas

  • I1. Name the user and the goal before picking an explanation method. Making the audience (developer vs. end user) and purpose explicit determines whether faithfulness or plausibility should drive the choice. This is a useful checklist item for any interpretability work, including Ukrainian NLP models.
  • I2. Separate sensitivity from saliency. A raw gradient norm shows how much the output would react to changes at a position. Gradient×input or IG instead estimate how much the actual token contributed. Reporting which of the two you measure avoids misleading heatmaps.
  • I3. Check whether a component matters before interpreting it. Swapping attention for uniform or random weights reveals whether the mechanism is actually used for the task. This is a cheap sanity check before reading meaning into any internal signal.
  • I4. Treat erasure-based evaluation with caution. Removing or zeroing tokens may produce unnatural inputs, so a drop in output can be an artefact. Any evaluation that deletes words, such as for treebank-based or classification models, should consider this confound.
  • I5. Use attention-like signals in alignment tasks only after comparison. Since saliency gave better MT alignments than attention, benchmark gradient-based alignments against attention when building word-alignment or lexicon resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 58 works

Cite

Cite as Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.blackboxnlp-1.14

@inproceedings{bastings2020elephant,
  title = {{The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?}},
  author = {Jasmijn Bastings and Katja Filippova},
  booktitle = {Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP},
  year = {2020},
  doi = {10.18653/v1/2020.blackboxnlp-1.14},
  url = {https://doi.org/10.18653/v1/2020.blackboxnlp-1.14}
}

References

41 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197, Online. Association for Computational Linguistics.
  2. Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2019. Gradient-Based Attribution Methods, pages 169–191. Springer International Publishing, Cham. find
  3. Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017. Explaining recurrent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 159–168, Copenhagen, Denm… find
  4. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Asso…
  5. Shuoyang Ding, Hainan Xu, and Philipp Koehn. 2019. Saliency-driven word alignment interpretation for neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 1–12, Florence, Italy. Association for Computational Linguistics. find
  6. Christopher Grimsley, Elijah Mayfield, and Julia R.S. Bursten. 2020. Why attention is not explanation: Surgical intervention and causal reasoning about neural models. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1780–1790, Marseille, France. European Language Resour…
  7. Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9737– 97… find
  8. Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. 2020. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 187–196, Online. Associat… find
  9. Alon Jacovi and Yoav Goldberg. 2020a. Aligning faithful interpretations with their social attribution.
  10. Alon Jacovi and Yoav Goldberg. 2020b. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics.
  11. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa…
  12. Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2020. Explaining explanations: Axiomatic feature interactions for deep networks. arXiv preprint arXiv:2002.04138. — [arXiv:2002.04138](https://arxiv.org/abs/2002.04138) find
  13. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761–780.
  14. Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2017. The (un)reliability of saliency methods. find
  15. W. Landecker, M. D. Thomure, L. M. A. Bettencourt, M. Mitchell, G. T. Kenyon, and S. P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pages 32–38. find
  16. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. Association for Computational Linguistics.
  17. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016a. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681–691, San Diego, California. As…
  18. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. find
  19. Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36–43. find
  20. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  21. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 14014– 14024. Curran Associates, Inc.
  22. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267:1–38.
  23. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206– 4216, Onlin… find
  24. Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. 2019. Layer-wise relevance propagation: An overview. In Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors, Explainable AI: Interpreting, Explaini… find
  25. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As… find
  26. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas. Association for Computational Linguistic…
  27. Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 340–350, Melbourne, A… find
  28. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782– 4793, Online. Association for Computational Linguisti…
  29. Martin Tutek and Jan Snajder. 2020. Staying true to your word: (how) can attention become explanation? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 131–142, Online. Association for Computational Linguistics. find
  30. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process…
  32. Jesse Vig. 2019. A multiscale visualization of attention in the transformer model.
  33. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ”why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144.
  34. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
  35. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. find
  36. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for…
  37. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics.
  38. Julia Strout, Ye Zhang, and Raymond Mooney. 2019. Do human rationales improve machine explanations? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 56–62, Florence, Italy. Association for Computational Linguistics. find
  39. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038, Brussels, Belgium. Association for Comput…
  40. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR. find
  41. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Computer Vision – ECCV 2014, pages 818–833, Cham. Springer International Publishing.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bastings-2020-elephant-interpretability-room-why-use/