Elamur Research
☰
home / papers / 2019 / Attention is not Explanation

Attention is not Explanation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1357.pdf

Summary

Attention heatmaps are often shown as evidence of which input tokens drive a neural model's decision. Jain and Wallace test whether this interpretation holds. They study text classification, question answering and natural language inference, using mainly BiRNN (BiLSTM) encoders with additive or scaled dot-product attention. Convolutional and averaged-embedding encoders serve as points of comparison.

They ask two questions:

  • Do attention weights agree with gradient-based and leave-one-out measures of token importance?
  • Would different attention distributions actually change the prediction?

To answer the second, they randomly permute the learned weights and also search for adversarial distributions. These adversarial distributions differ as much as possible from the original while keeping the output nearly fixed.

With recurrent encoders, agreement with importance measures is weak and inconsistent. Very different attention patterns, whether shuffled or adversarially constructed, often leave predictions essentially unchanged.

The authors conclude that standard attention should not be presented as an explanation of model behaviour. They call for more principled, interpretability-oriented attention variants.

Key points

  • Benchmarks:
    • Classification: SST, IMDB, Twitter ADR, 20 Newsgroups (Hockey vs Baseball), AG News (Business vs World), and MIMIC ICD9 tasks (Diabetes; Chronic vs Acute Anemia).
    • QA: CNN, bAbI tasks 1–3.
    • NLI: SNLI.
  • Correlation with gradients: Kendall τ between attention and gradient importance is modest for BiLSTM models (about 0.34–0.36 on SST). It is noticeably higher for the non-recurrent average encoder (about 0.60–0.61 on SST).
  • Example review (Figure 1): For a negative movie review predicted at 0.01, an adversarial attention map over entirely different words gives the same output. Correlation with gradient importance is only τ_g = 0.29, and shuffling the weights shifts the output by a median of just 0.006.
  • Limitations noted by the authors:
    • Gradient-based measures are not treated as ground truth.
    • Adversarial weights may be unlikely under the trained model.
    • Only simple attention variants and non-seq2seq tasks were examined.
  • Code: available at github.com/successar/AttentionExplanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

399 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 383 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1357

@inproceedings{jain2019attention,
  title = {{Attention is not Explanation}},
  author = {Jain, Sarthak and Wallace, Byron C.},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1357},
  url = {https://doi.org/10.18653/v1/n19-1357}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Alvarez-Melis and Tommi S Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. arXiv preprint arXiv:1707.01943. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  3. Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. arXiv preprint arXiv:1808.09367. find
  4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. find
  5. Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504–3512. find
  6. Shi Feng, Eric Wallace, Alvin Grissom II, Pedro Rodriguez, Mohit Iyyer, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Empirical Methods in Natural Language Processing. find
  7. Reza Ghaeini, Xiaoli Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4952–4957. find
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  9. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035. find our card
  10. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. arXiv preprint arXiv:1702.00887. — [arXiv:1702.00887](https://arxiv.org/abs/1702.00887) find
  11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  12. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155. find
  13. Tao Lei et al. 2017. Interpretable neural models for natural language processing. Ph.D. thesis, Massachusetts Institute of Technology. find
  14. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  15. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490. — [arXiv:1606.03490](https://arxiv.org/abs/1606.03490) find
  16. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon… find our card
  17. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623. find
  18. Azadeh Nikfarjam, Abeed Sarker, Karen OâĂŹConnor, Rachel Ginn, and Graciela Gonzalez. 2015. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association, 22(3):671–68… find
  19. Nikolaos Pappas and Andrei Popescu-Belis. 2016. Human versus machine attention in document classification: A dataset with crowdsourced annotations. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 94–100. find
  20. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933. find our card
  21. Ben Peters, Vlad Niculae, and André FT Martins. 2018. Interpretable structure induction via sparse attention. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 365–367. find
  22. Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717. — [arXiv:1703.03717](https://arxiv.org/abs/1703.03717) find
  23. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. find our card
  24. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  25. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  27. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. find our card
  28. Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. arXiv preprint arXiv:1704.05908. find
  29. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604. find our card
  30. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. find our card
  31. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. find our card
  32. Ye Zhang, Iain Marshall, and Byron C Wallace. 2016. Rationale-augmented convolutional neural networks for text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 2016, pages 795–804. find