Elamur Research
☰
home / papers / 2019 / Attention is not not Explanation

Attention is not not Explanation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1002.pdf

Summary

This paper responds to Jain and Wallace (2019), who argued that attention weights in RNN classifiers do not explain predictions because very different attention distributions can produce nearly the same output. Wiegreffe and Pinter argue that this conclusion depends on how "explanation" is defined. They also note that the earlier experiments altered attention separately for each instance and detached it from the parameters that computed it.

Using the same English classification datasets and a BiLSTM with additive attention, the authors propose four tests:

  • a frozen uniform-weights baseline,
  • a measure of attention variance across random seeds,
  • a diagnostic MLP that uses fixed attention weights from pretrained models,
  • an adversarial training procedure that keeps the model consistent.

They find that some tasks barely need learned attention. Consistent adversarial distributions can be trained, but they are less extreme than per-instance ones and perform poorly in the diagnostic MLP. The study therefore does not rule out attention as explanation. It also gives practitioners practical ways to judge when attention weights are informative.

Key points

  • The datasets, with the train/test splits released by Jain and Wallace, are SST, IMDb, 20 Newsgroups (hockey vs. baseball), AG News (world vs. business), and the MIMIC-III ICD9 tasks Diabetes and Anemia. Average lengths range from 19 tokens (SST) to 2188 (Anemia).
  • The model is a single-layer bidirectional LSTM with tanh activation, additive (Bahdanau) attention, and a softmax output, matching the earlier work's setup and hyperparameters.
  • On some datasets, attention frozen to a uniform distribution performs about as well as learned attention. On those tasks, swapping in other distributions says little about explainability.
  • The adversary is trained end to end with a loss that rewards distance from the base model's attention. Its distributions are less extreme than Jain and Wallace's and do worse than LSTM-trained weights when plugged into the non-contextual MLP diagnostic.
  • The authors separate plausible from faithful explanation and argue that non-exclusivity alone does not invalidate attention-based rationales. Code is available at github.com/sarahwie/attention.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

285 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 276 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1002

@inproceedings{wiegreffe2019attention,
  title = {{Attention is not not Explanation}},
  author = {Wiegreffe, Sarah and Pinter, Yuval},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1002},
  url = {https://doi.org/10.18653/v1/d19-1002}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. arXiv preprint Mark O Riedl. 2019. Human-centered artificial intelligence and machine learning. Human Behavior and Emerging Technologies, 1(1):33–36. find our card
  2. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. find our card
  3. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664. find our card
  4. Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Automated rationale generation: a technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263–274. ACM. find
  5. Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: training differentiable models by constraining their explanations. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 2662–2670. AAAI Press. find
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  7. Cynthia Rudin. 2018. Please stop explaining black box models for high stakes decisions. arXiv preprint arXiv:1811.10154. find our card
  8. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Minneapolis, Minnesota. Association for Computational Ling… find our card
  9. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics. find our card
  10. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035. find our card
  11. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  12. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. Association for Computational Linguistics. find
  13. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V… find
  14. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490. — [arXiv:1606.03490](https://arxiv.org/abs/1606.03490) find
  15. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142–150. Association… find our card
  16. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048–2057. find our card
  17. James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
  18. Azadeh Nikfarjam, Abeed Sarker, Karen O’connor, Rachel Ginn, and Graciela Gonzalez. 2015. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association, 22(3):671–681. find