Elamur Research⌕
☰
home / papers / 2022 / How to Dissect a Muppet: The Structure of Transformer Embedding Spaces

How to Dissect a Muppet: The Structure of Transformer Embedding Spaces

Paper · 2022 · Timothee Mickus, Denis Paperno and Mathieu Constant

Summary

Studies of pretrained Transformer embeddings usually either inspect attention weights or treat the whole network as an opaque box. Mickus, Paperno and Constant propose a third route: an exact algebraic rewriting of an encoder's output vector for any token as a sum of four parts. The first part is an input term: static word, position and segment embeddings, rescaled by every layer normalization. The second collects all multi-head attention outputs, and the third collects all feed-forward outputs. The fourth gathers the biases and normalization mean shifts. The derivation relies on residual connections: each sublayer adds a new vector to a running total that is then rescaled. Applying this to BERT, the authors measure how much each part weighs in the final embedding and how useful each part is for downstream tasks. They also examine how finetuning reshapes the embedding space. The analysis connects to earlier findings on anisotropy and attention-based explanations. It shows that attention and feed-forward contributions help unequally across tasks, offering a component-level interpretability tool.

Key points

  • Core equation: e_t = i_t + h_t + f_t + c_t. Vectors rebuilt from the four terms matched real BERT embeddings to within ±10⁻⁷ per component.
  • Only f_t is genuinely non-linear because of the ReLU/GELU activation in feed-forward blocks. Elsewhere, non-linearity reduces to scalar factors (LN scaling, attention weights). The terms are not independent, so the decomposition does not fully disentangle computations.
  • A model with L layers has Λ = 2L sublayers, and each one acts as "add a vector, then rescale." A submodule whose output stays consistently small has little influence on later computations.
  • Probing setup:
    • Classifiers and shallow logistic regressions were tuned via Bayes Optimization, using 50 or 100 samples.
    • The tuning objective was accuracy or macro-f1.
    • Everything ran on a single 4GB NVIDIA GPU.
  • Ethics note: individual sub-terms can outperform the full embedding on some tasks. However, they may encode stronger social stereotypes, such as those found with the offset method of Mikolov et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations🎓Université de LorraineNancy · France20 works · 945 citations🔬Analyse et Traitement Informatique de la Langue FrançaiseNancy · France6 works · 390 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Timothee Mickus, Denis Paperno and Mathieu Constant. 2022. How to Dissect a Muppet: The Structure of Transformer Embedding Spaces. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00501

@article{mickus2022how,
  title = {{How to Dissect a Muppet: The Structure of Transformer Embedding Spaces}},
  author = {Timothee Mickus, Denis Paperno and Mathieu Constant},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00501},
  url = {https://doi.org/10.1162/tacl_a_00501}
}

References

42 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. g λ  λ  =λ+1
  2. Using the above, Eq. (5) is equivalent to: ⎛ Λ c t = λ ⎞ ⎜ 1 ⎜ ⎜ Λ  ⎝ λ  = λ +1 ⎟ ⎟ · p λ ⎟ + ⎠ λ  s t ⎛ ⎞ ⎜ − m ⎟ ⎜ ⎟ λ,t · q λ ⎟ ⎜ Λ  ⎝ ⎠ λ s λ  ,t Λ λ  = λ +1 Note that p λ and q λ are constant across all inputs. Assuming their linear independence puts an upper bound of 2Λ vectors necessary… find
  3. Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015 ; Conference date: 07-05-2015 Through 09-05-2015.
  4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. find
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  6. C Computational Details In Section 2.3, we use the default hyperparameters of scikit-learn (Pedregosa et al., 2011). In Section 4, we learn categorical regressions using an AdamW optimizer (Loshchilov and Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of d…
  7. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, and Douglas Eck. 2018. An improved relative self-attention mechanism for transformer with application to music generation. CoRR, abs/1809.04281. find
  8. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa…
  9. Jiaju Du, Fanchao Qi, and Maosong Sun. 2019. Using BERT for word sense disambiguation. CoRR, abs/1909.08358. — [arXiv:1909.08358](https://arxiv.org/abs/1909.08358) find
  10. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  11. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18…
  12. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational…
  13. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.
  14. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… find
  15. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86. Phuket, Thailand. find
  16. Shari Landes, Claudia Leacock, and Randee I. Tengi. 1998. Building semantic concordances. WordNet: An Electronic Lexical Database, chapter 8, pages 199–216. Bradford Books. find
  17. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (GELU). arXiv preprint arXiv:1606.08415. find
  18. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  19. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
  20. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics. https…
  21. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium. Association for Computational L…
  22. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, BERT? In Proceedings of the Society for Computation in Linguistics 2020, pages 279–290, New York, New York. Association for Computational Linguistics. find
  23. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia.…
  24. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
  25. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. https://doi.org/10.1111/j.1551-6709 .2010.01106.x Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transaction… — [doi:10.1111/j.1551-6709](https://doi.org/10.1111/j.1551-6709) ↗
  26. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  27. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P19-1282
  28. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems…
  29. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. CoRR, abs/1803.02155.
  30. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc. find
  31. Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016. Results of the WNUT16 named entity recognition shared task. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 138–144, Osaka, Japan. find
  32. The COLING 2016 Organizing Committee. find
  33. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782–4793, Online. Association for Computational Linguistic…
  34. William Timkey and Marten van Schijndel. 2021. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4527–4546, Online and Punta Cana, Dominican Republic…
  35. Hong Kong, China. Association for Computational Linguistics. https://doi.org/10 .18653/v1/D19-1448 Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical M…
  36. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. https://doi.org/10.3115/1119176.1119195 ↗
  37. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. ArXiv, abs/1909.10430. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  39. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for…
  40. Raúl Vázquez, Hande Celikkanat, Mathias Creutz, and Jörg Tiedemann. 2021. On the differences between BERT and MT encoder spaces and how to address them in translation tasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  41. Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. 2021. Transformer visualization via dictionary learning: Contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for… find
  42. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mickus-2022-how-dissect-muppet-structure-transformer/