Elamur Research
☰
home / papers / 2020 / Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.574.pdf

Summary

Research that probes Transformers often treats attention weights as a direct measure of how much each input token shapes a representation. Kobayashi et al. argue this view is incomplete. If you rewrite the attention equation, each output becomes a sum of input vectors that were first linearly transformed and then scaled by their weights. So the size of those transformed vectors matters alongside the weights. The authors propose norm-based analysis. It measures the Euclidean length of each weighted transformed vector, ‖αf(x)‖, and also examines ‖f(x)‖ and α separately.

Applied to BERT, the method overturns weight-based findings by Clark et al. (2019). Tokens such as [SEP], [CLS] and punctuation receive large weights, but they actually contribute little, because their transformed vectors are small. BERT also reduces the influence of frequent, uninformative words by shrinking their vector norms. For a Transformer-based NMT system, source-target attention measured by norms produces sensible word alignments, whereas earlier weight-based studies found these alignments noisy. The work shows that reading weights alone can mislead and offers a simple, more faithful alternative.

Key points

  • The transformed vector is defined as f(x) = (xW^V + b^V)W^O. A token's contribution to an attention output is scored by ‖αf(x)‖ rather than α alone. The approach extends to full multi-head attention (Appendix A).
  • In BERT, the coefficient of variation of ‖f(x)‖ averages 0.22 across heads. It ranges from 0.12 (Layer 2–Head 7) to 0.37 (Layer 2–Head 4), and norms typically fall between 0.78 and 1.22 times the mean. This spread is why weights and norms diverge.
  • Norm-based analysis shows that BERT gathers far less information from special tokens than weight-based studies suggested. It also discounts frequent words.
  • NMT word alignments were evaluated with AER in two settings (AWI and AWO) across five random seeds, with per-head AER compared against average ‖αf(x)‖. Norm-based extraction gave more accurate alignments than weight-based extraction.
  • The method is contrasted with "effective attention" (Brunner et al., 2020) and gradient-based analysis. Code is available at github.com/gorokoba560/norm-analysis-of-transformer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

125 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.574

@inproceedings{kobayashi2020attention,
  title = {{Attention is Not Only a Weight: Analyzing Transformers with Vector Norms}},
  author = {Kobayashi, Goro and Kuribayashi, Tatsuki and Yokoi, Sho and Inui, Kentaro},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.574},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.574}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. arXiv preprint arXiv:1607.06450. find our card
  2. Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On Identifiability in Transformers. In 8th International Conference on Learning Representations (ICLR). find
  3. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What Does BERT Look At? An Analysis of BERT’s Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  5. Shuoyang Ding, Hainan Xu, and Philipp Koehn. 2019. Saliency-driven Word Alignment Interpretation for Neural Machine Translation. In Proceedings of the 4th Conference on Machine Translation (WMT), pages 1–12. find
  6. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACCL-HLT), pages 644– 648. find
  7. Yoav Goldberg. 2019. Assessing BERT’s Syntactic Abilities. arXiv preprint arXiv:1901.05287. find our card
  8. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do Attention Heads in BERT Track Syntactic Dependencies? arXiv preprint arXiv:1911.12246. — [arXiv:1911.12246](https://arxiv.org/abs/1911.12246) find
  9. Sarthak Jain and Byron C Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 3543–3556. find our card
  10. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3651–3657. find our card
  11. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the Dark Secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4364–437… find our card
  12. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In 8th International Conference on Learning Representations (ICLR). find
  13. Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the Word Alignment from Neural Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1293–1303. find
  14. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open Sesame: Getting Inside BERT’s Linguistic Knowledge. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253. find our card
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. find
  16. David Mareček and Rudolf Rosa. 2019. From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 263– 275. find
  17. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One? In Advances in Neural Information Processing Systems 32 (NIPS), pages 14014–14024. find our card
  18. Franz Josef Och and Hermann Ney. 2000. Improved Statistical Alignment Models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL), pages 440–447. find
  19. Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–51. find
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  21. Damian Pascual, Gino Brunner, and Roger Wattenhofer. 2020. Telling BERT’s full story: from Local Attention to Global Aggregation. arXiv preprint arXiv:2004.05916. find
  22. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2020. Learning to Deceive with Attention-Based Explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  23. Alessandro Raganato and Jörg Tiedemann. 2018. An Analysis of Encoder Representations in Transformer-Based Machine Translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297. find
  24. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and Measuring the Geometry of BERT. Advances in Neural Information Processing Systems 32 (NIPS), pages 8594–8603. find our card
  25. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention Interpretability Across NLP Tasks. arXiv preprint arXiv:1909.11218. — [arXiv:1909.11218](https://arxiv.org/abs/1909.11218) find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30 (NIPS), pages 5998–6008. find
  27. David Vilar, Maja Popović, and Hermann Ney. 2006. AER: Do we need to “improve” our alignments? In International Workshop on Spoken Language Translation (IWSLT) 2006, pages 205–212. find
  28. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP Models Know Numbers? Probing Numeracy in Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find our card
  29. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not Explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20. find our card
  30. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What we know about how BERT works. arXiv preprint arXiv:2002.12327. find our card
  31. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems 32 (NIPS), pages 1–18. find our card
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1715–1725. find
  33. Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding Interpretable Attention to Neural Translation Models Improves Word Alignment. arXiv preprint arXiv:1901.11359. — [arXiv:1901.11359](https://arxiv.org/abs/1901.11359) find
  34. Sofia Serrano and Noah A Smith. 2019. Is Attention Interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2931–2951. find our card
  35. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation. In Proceedings of the 3rd Conference on Machine Translation (WMT): Research Papers, pages 26–35. find
  36. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conferenc… find our card