Elamur Research
☰
home / papers / 2019 / A Multiscale Visualization of Attention in the Transformer Model

A Multiscale Visualization of Attention in the Transformer Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.05714

Summary

Transformer models such as BERT and GPT-2 rely entirely on attention, which can make them easier to inspect. But the number of layer/head combinations (384 per input for BERT-Large) makes raw attention hard to read. Vig extends the multi-head attention visualizer of Jones (2017) into an open-source tool, bertviz, that shows attention at three levels of granularity. The attention-head view draws token-to-token links for selected heads in one layer. The model view arranges thumbnails of every layer and head in a grid. The neuron view breaks down how query and key vector components combine into attention scores. The tool was adapted from the original encoder-decoder setting to encoder-only BERT and decoder-only GPT-2. Three case studies show what it can do: revealing gender-dependent coreference in GPT-2, finding BERT heads that link two sentences (useful for paraphrase detection), and tracing a distance-decaying attention pattern to a few neurons. The work offers a practical interpretability resource and a starting point for targeted model interventions.

Key points

  • The tool is available at github.com/jessevig/bertviz. The demos use the GPT-2 small model and BERT-base (uncased).
  • The neuron view shows several columns for a selected token:
    • its query vector,
    • the key vectors of the other tokens,
    • their element-wise product,
    • the dot product,
    • the resulting softmax attention.
  • Bias case: with greedy decoding, GPT-2 resolves "She" to the nurse and "He" to the doctor in otherwise identical prompts. One attention head mirrors this gendered pattern.
  • In the BERT model view, heads that connect Sentence A and Sentence B have a recognizable cross-hatch shape (e.g., layer 3, head 0).
  • In GPT-2 layer 1, head 10, attention fades smoothly with distance, largely independent of content. A small set of neuron positions drives this effect, suggesting that the decay rate could be adjusted to control context length.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

153 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.05714 · arXiv 1906.05714

@misc{vig2019multiscale,
  title = {{A Multiscale Visualization of Attention in the Transformer Model}},
  author = {Vig, Jesse},
  year = {2019},
  doi = {10.48550/arxiv.1906.05714},
  eprint = {1906.05714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.05714}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  2. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In Proc. ICLR. find
  3. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. TACL. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv Computation and Language. find
  5. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. CoRR, abs/1902.10186. find our card
  6. Llion Jones. 2017. Tensor2tensor transformer visualization. https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization. ↗
  7. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In EMNLP: System Demonstrations. find
  8. Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. In EMNLP: System Demonstrations. find
  9. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. CoRR, abs/1807.11714. find our card
  10. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find
  11. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In EMNLP Workshop: BlackboxNLP. find
  12. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proc. ICLR. find our card
  13. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proc. EMNLP. find
  14. H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush. 2018. Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models. ArXiv e-prints. find
  15. Edward Tufte. 1990. Envisioning Information. Graphics Press, Cheshire, CT, USA. find
  16. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. find our card
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  18. Jesse Vig. 2019. BertViz: A tool for visualizing multi-head self-attention in the BERT model. In ICLR Workshop: Debugging Machine Learning Models. find
  19. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In ACL Workshop: BlackboxNLP. find our card
  20. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. find our card
  21. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT. find our card