Elamur Research
☰
home / papers / 2019 / Is Attention Interpretable?

Is Attention Interpretable?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1282.pdf

Summary

The paper asks whether attention weights in trained text classifiers show which inputs actually drive a prediction. The authors deliberately test a low bar. They ask only whether attention correctly ranks the importance of the attention layer's own inputs, which are contextualized representations, rather than the original words. Their method is erasure. They zero out selected attention weights, renormalize the remaining distribution, rerun the classifier, and measure the Jensen-Shannon divergence of the output and whether the predicted label flips. Experiments cover Yahoo Answers, IMDB, Yelp 2017 and Amazon. The models are variants of the hierarchical attention network (HAN) and a single-layer "flat" version (FLAN), with recurrent, convolutional or no encoders.

Attention correlates with importance only loosely. Removing the top-weighted item usually matters more than removing a random one, but the effect is often tiny. Alternative orderings, such as gradient-based rankings, frequently find smaller sets of items whose removal flips the decision. The authors conclude that attention is a noisy, unreliable explanation of model decisions.

Key points

  • Data: four classification datasets were used.
    • Yahoo Answers: 10 classes, about 1.4M train+dev documents.
    • IMDB: 10 classes, about 122K documents.
    • Amazon: 5 classes, 3M documents.
    • Yelp 2017: 5 classes, 650K documents.
  • Models: HAN and FLAN architectures with biGRU ("rnn"), convolutional ("conv") or no ("noenc") encoders were tested. All use additive (Bahdanau-style) softmax attention, and only the final attention layer is probed.
  • Single-item test: erasing just the highest-weighted item and erasing one random item usually leave the label unchanged. Neither change flips the decision in about 84–90% of test instances for the HANrnn models.
  • Effect of context: the scope of contextualization before attention strongly shapes the results. Attention magnitudes are more informative in models without contextualization. RNN-based models keep their original decision even after many attended items are erased.
  • Limitations and code: the study covers only text classification, the final attention layer, and argmax-based importance. Multi-head and self-attention are not tested. Code is available at github.com/serrano-s/attn-tests.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

237 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 229 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1282

@inproceedings{serrano2019is,
  title = {{Is Attention Interpretable?}},
  author = {Serrano, Sofia and Smith, Noah A.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1282},
  url = {https://doi.org/10.18653/v1/p19-1282}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems. find
  2. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. find our card
  3. Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. Explaining predictions of non-linear classifiers in NLP. arXiv preprint arXiv:1606.07298. find
  4. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). find our card
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  6. Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  7. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  8. Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J. Smola, Jing Jiang, and Chong Wang. 2014. Jointly modeling aspects, ratings and sentiments for movie recommendation (JMARS). In Proceedings of the ACM SIGKDD International Conference on Knowledge Ciscovery and Data mining. find
  9. Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  10. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  11. Alona Fyshe, Leila Wehbe, Partha P. Talukdar, Brian Murphy, and Tom M. Mitchell. 2015. A compositional and interpretable semantic space. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  14. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. find
  15. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730. find our card
  16. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive Visualization and Manipulation of Attention-based Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121–126. find
  17. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  18. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2015. Visualizing and understanding neural models in NLP. arXiv preprint arXiv:1506.01066. find our card
  19. Reza Ghaeini, Xiaoli Z. Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. arXiv preprint arXiv:1808.03894. find
  20. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  21. Amirata Ghorbani, Abubakar Abid, and James Zou. 2017. Interpretation of Neural Networks is Fragile. arXiv preprint arXiv:1710.10547. — [arXiv:1710.10547](https://arxiv.org/abs/1710.10547) find
  22. Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. find
  23. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation. arXiv preprint arXiv:1802.06613. find
  24. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130. find our card
  25. Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association of Computational Linguistics, 6:63–75. find
  26. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  27. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. find
  28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems. find our card
  29. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490. — [arXiv:1606.03490](https://arxiv.org/abs/1606.03490) find
  30. André Martins and Ramón Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning. find
  31. David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems. find
  32. Dong Nguyen. 2018. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). find
  33. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data mining. find our card
  34. Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the International Joint Conference on Artificial Intelligence. find
  35. Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. 2017. SPINE: Sparse interpretable neural embeddings. arXiv preprint arXiv:1711.08792. — [arXiv:1711.08792](https://arxiv.org/abs/1711.08792) find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  37. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems. find
  38. Yequan Wang, Minlie Huang, Li Zhao, et al. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606–615. find
  39. Fan Yang, Arjun Mukherjee, and Eduard Dragut. 2017. Satirical news detection and analysis using attention mechanism and linguistic features. arXiv preprint arXiv:1709.01189. find