Elamur Research
☰
home / papers / 2019 / Analyzing the Structure of Attention in a Transformer Language Model

Analyzing the Structure of Attention in a Transformer Language Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.04284

Summary

Attention weights in Transformers are often treated as a window into model behavior, but their link to linguistic structure is not well understood. Vig and Belinkov study this for the pretrained GPT-2 small model (12 layers × 12 heads). They first build an open-source visualization tool (bertviz) that displays attention at three granularities: single heads, the full layer-by-head grid, and individual query/key neurons. It adapts earlier encoder-decoder visualizations to decoder-only GPT-2 and encoder-only BERT. They then aggregate attention over a corpus of Wikipedia sentences. For every head, they measure how much attention lands on each part-of-speech tag, how much matches dependency arcs, how far it reaches, and how much it varies with input content. Part-of-speech preferences shift with layer depth. Alignment with dependency relations peaks in the middle layers, where attention spans are shortest. The deepest layers cover the longest distances, though heads differ widely. Exemplar sentences reveal heads specialized for acronyms, place names, list items and noun phrases, mostly serving next-token prediction. The work offers a lightweight, human-readable complement to probing classifiers.

Key points

  • Tool: The bertviz tool (github.com/jessevig/bertviz) provides three perspectives: one for individual attention heads, one for the whole model, and one for neurons. For example, layer 4, head 11 attends mainly to the previous token.
  • Metrics: Four corpus-level metrics are defined per head:
    • attention share per POS tag;
    • dependency alignment, under three parent/child formulations;
    • attention variability, which separates content-dependent from position-based heads;
    • attention-weighted mean token distance.
  • Distance vs. dependency: Attention distance and dependency alignment are negatively correlated across heads (r = −0.73, p < 0.001). About 99.8% of dependency relations occur within 18 tokens.
  • Head examples:
    • layer 8, head 6 targets list items;
    • layer 10, head 10 links acronyms to their expanded phrase;
    • layer 11, head 2 links commas to the preceding place name;
    • layer 11, head 10 links the end of a noun phrase to its head noun.
  • Limitations and future work: The analysis covers only encyclopedic, single-sentence Wikipedia text. The authors propose extending it to BERT, Transformer-XL and Sparse Transformer, and to other genres and longer contexts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Part-of-speech tagging

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.04284 · arXiv 1906.04284

@misc{vig2019analyzing,
  title = {{Analyzing the Structure of Attention in a Transformer Language Model}},
  author = {Vig, Jesse and Belinkov, Yonatan},
  year = {2019},
  doi = {10.48550/arxiv.1906.04284},
  eprint = {1906.04284},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.04284}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). find our card
  2. Yonatan Belinkov. 2018. On Internal Language Representations in Deep Learning: An Analysis of Machine Translation and Speech Recognition. Ph.D. thesis, Massachusetts Institute of Technology. find
  3. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. find our card
  4. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. arXiv preprint arXiv:1805.04218. find
  5. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. find
  6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. Hamidreza Ghader and Christof Monz. 2017. What does attention in neural machine translation pay attention to? In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 30–39. find
  9. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287. find our card
  10. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  11. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  12. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. CoRR, abs/1902.10186. find our card
  13. Llion Jones. 2017. Tensor2tensor transformer visualization. https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization. ↗
  14. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. find
  15. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find our card
  16. Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demo… find
  17. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650. find our card
  18. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report. find
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find
  21. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in Transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297. Association for Computational Linguistics. find
  22. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In International Conference on Learning Representations (ICLR). find our card
  23. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  24. H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush. 2018. Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models. ArXiv e-prints. find
  25. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the Association for Computational Linguistics. find our card
  26. Edward Tufte. 1990. Envisioning Information. Graphics Press, Cheshire, CT, USA. find
  27. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. find our card
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  29. Jesse Vig. 2019. A multiscale visualization of attention in the Transformer model. In Proceedings of the Association for Computational Linguistics: System Demonstrations. find our card
  30. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274. Association for Computational L… find our card
  31. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. find our card
  32. Thomas Wolf. 2019. Some additional experiments extending the tech report ”Assessing BERTs syntactic abilities” by Yoav Goldberg. Technical report. find our card