Elamur Research
☰
home / papers / 2020 / Quantifying Attention Flow in Transformers

Quantifying Attention Flow in Transformers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.385.pdf

Summary

Raw attention maps are often used to explain Transformer decisions. In deeper layers, however, each embedding already blends information from many tokens, so attention between hidden states says little about the role of the original input words. Abnar and Zuidema treat the network as a directed acyclic graph whose edges carry attention weights, adjusted for residual connections. They propose two post hoc ways to trace attention back to input tokens. Attention rollout multiplies the layer attention matrices recursively. Attention flow treats the weights as edge capacities and solves a maximum-flow problem. The authors test both on a small Transformer trained on the Linzen et al. (2016) subject-verb agreement data. They measure Spearman correlation with blank-out ablation scores and with input gradients. Both methods match these importance estimates far better than raw attention in the higher layers. Flow is generally the more reliable of the two, while rollout yields sharper, more focused patterns. The techniques need only attention weights and work with any architecture, which makes them cheap diagnostic and visualization tools.

Key points

  • Setup: The model is a Transformer encoder built from GPT-2 blocks without causal masking, with 6 layers, 8 heads and hidden size 128. A CLS token feeds the classifier, which predicts verb number with 0.96 accuracy. Heads are averaged per layer.
  • Residual connections: These are modelled by mixing each attention matrix with the identity, A = 0.5·W_att + 0.5·I, before rollout or flow is computed.
  • Blank-out results (2000 test samples, final layer): Raw attention reaches ρ ≈ 0.29, versus 0.71 for rollout and 0.70 for flow. At layer 3, raw attention is negative (−0.11), while flow already reaches 0.70.
  • Input-gradient results (final layer): Raw attention scores 0.46, rollout 0.54 and flow 0.61. On DistilBERT fine-tuned on SST-2 (100 samples), all correlations are very low (≈0.09–0.14), but rollout and flow still slightly beat raw attention. In a BERT pronoun-resolution example, flow agreed best with the model's predictions.
  • Code and limitations: Code is released at github.com/samiraabnar/attention_flow. The methods may be biased toward early tokens in masked decoders, so they would need normalization by receptive field there.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.385

@inproceedings{abnar2020quantifying,
  title = {{Quantifying Attention Flow in Transformers}},
  author = {Abnar, Samira and Zuidema, Willem},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.385},
  url = {https://doi.org/10.18653/v1/2020.acl-main.385}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2019. Gradient-Based Attribution Methods, pages 169–191. Springer International Publishing. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In proceedings of the 2015 International Conference on Learning Representations. find our card
  3. Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identifiability in transformers. In International Conference on Learning Representations. find
  4. Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classifiers via data augmentation. arXiv preprint arXiv:1909.04225. — [arXiv:1909.04225](https://arxiv.org/abs/1909.04225) find
  5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… find
  6. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry of bert. arXiv preprint arXiv:1906.02715. find our card
  7. Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms, Third Edition, 3rd edition. The MIT Press. find
  8. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2019. Universal transformers. In proceedings of the 2019 International Conference on Learning Representations. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.… find
  10. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3543–3556. Association for Computational Linguistics. find our card
  11. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121–126. find
  12. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  13. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2019. Learning to deceive with attention-based explanations. arXiv preprint arXiv:1909.07913. find
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  15. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In International Conference on Learning Representations (ICLR). find our card
  16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  17. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find our card
  18. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  19. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. arXiv preprint arXiv:1909.11218. — [arXiv:1909.11218](https://arxiv.org/abs/1909.11218) find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  21. Jesse Vig. 2019. Visualizing attention in transformer-based language models. arXiv preprint arXiv:1904.02679. — [arXiv:1904.02679](https://arxiv.org/abs/1904.02679) find
  22. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  23. Yequan Wang, Minlie Huang, Li Zhao, et al. 2016. Attention-based lstm for aspect-level sentiment classification. In proceedings of the 2016 conference on empirical methods in natural language processing, pages 606–615. find
  24. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics. find our card
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In proceedings of International Conference on Machine Learning, pages 2048–2057. find our card