Elamur Research⌕
☰
home / papers / 2023 / Analyzing Transformers in Embedding Space

Analyzing Transformers in Embedding Space

Summary

Most Transformer interpretability work runs inputs through a model and inspects the activations. That is costly, and the conclusions depend on the inputs chosen. Earlier static approaches covered only feed-forward value vectors or two-layer attention-only networks. Dar et al. generalize these into one input-independent framework that requires no forward or backward pass. Every trained parameter is multiplied by the embedding matrix, which casts it into vocabulary space. Attention matrices (W_QK, W_VO) then read as interactions between pairs of vocabulary items, and FF keys and values as distributions over tokens.

The experiments center on GPT-2 medium, because the framework needs a linear LM head. Projected parameters turn out to be human-interpretable. After fine-tuning on movie-review sentiment, the parameter changes surface movie-sentiment words. Because the embedding space depends only on the tokenizer, it also serves as a common reference for models sharing a vocabulary. This lets the authors align independently trained BERT models and transfer a classifier into another model without training.

Key points

  • The framework interprets FF keys, FF values, W_QK and W_VO by projecting them through the embedding matrix E. Biases and layer normalization are excluded.
  • BERT instances pretrained from different random seeds align layer by layer in embedding space: layer i of one matches layer i of the other. Doing the same comparison in feature space produces no meaningful pattern.
  • The zero-shot "stitching" experiment works as follows:
    • Hidden states of a GPT-2 medium (24-layer) IMDB classifier are mapped through the kernel E_A·E_B⁺, which uses a pseudo-inverse.
    • The mapped states feed the last three fine-tuned layers of a 12-layer GPT-2 base model.
    • In 3 of 11 runs, stitching at top layers reached about 70% IMDB accuracy, against a 50% baseline.
  • The main limitation is that architectures with non-linear LM heads (BERT, RoBERTa, T5) are not formally covered. Even so, alignment worked in practice for BERT.
  • Code is released at github.com/guyd1995/embedding-space.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.893

@inproceedings{dar2023analyzing,
  title = {{Analyzing Transformers in Embedding Space}},
  author = {Dar, Guy and Geva, Mor and Gupta, Ankit and Berant, Jonathan},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.893},
  url = {https://doi.org/10.18653/v1/2023.acl-long.893}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Adi, E. Kermany, Y. Belinkov, O. Lavi, and Y. Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks, 2016. URL https://arxiv.org/abs/1608.04207. ↗ our card
  2. A. Baevski, H. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020. URL https://arxiv.org/abs/2006.11477. ↗ our card
  3. Y. Bansal, P. Nakkiran, and B. Barak. Revisiting model stitching to compare neural representations. In NeurIPS, 2021. find
  4. A. Bjerhammar. Application of calculus of matrices to method of least squares : with special reference to geodetic calculations. In Trans. Roy. Inst. Tech. Stockholm, 1951. find
  5. N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/ distill.00024. https://distill.pub/2020/circuits. ↗
  6. M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1691–1703. PMLR, 13–18… find
  7. K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does BERT look at? an analysis of bert’s attention. CoRR, abs/1906.04341, 2019. URL http://arxiv.org/abs/1906.04341. ↗
  8. A. Csiszárik, P. Korösi-Szabó, Á. K. Matszangosz, G. Papp, and D. Varga. Similarity and matching of neural network representations. In NeurIPS, 2021. find
  9. D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers, 2021. URL https://arxiv.org/abs/ 2104.08696. ↗
  10. F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, A. Bau, and J. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models, 2018. URL https://arxiv.org/ abs/1812.09355. — [arXiv:1812.09355](https://arxiv.org/abs/1812.09355) ↗
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL https://arxiv.org/abs/1810.04805. ↗
  12. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL https://arxiv.org/abs/2010.11929. ↗
  13. N. Durrani, H. Sajjad, F. Dalvi, and Y. Belinkov. Analyzing individual neurons in pre-trained language models. CoRR, abs/2010.02695, 2020. URL https://arxiv.org/abs/2010.02695. ↗
  14. N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematic… find
  15. N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E.… find
  16. K. Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings, 2019. URL https://arxiv.org/abs/1909.00512. ↗
  17. J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T. Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SkEYojRqtm. ↗ our card
  18. M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories, 2020. URL https://arxiv.org/abs/ 2012.14913. ↗
  19. M. Geva, A. Caciularu, G. Dar, P. Roit, S. Sadde, M. Shlain, B. Tamir, and Y. Goldberg. Lm-debugger: An interactive tool for inspection and intervention in transformer-based language models. arXiv preprint arXiv:2204.12130, 2022a. find
  20. M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space, 2022b. URL https://arxiv.org/ abs/2203.14680. ↗ our card
  21. P. Jaccard. The distribution of the flora in the alpine zone. The New Phytologist, 11(2):37–50, 1912. ISSN 0028646X, 14698137. URL http://www.jstor.org/stable/2427226. ↗
  22. H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2): 83–97, 1955. find
  23. K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 991– 999, 2015. find
  24. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https://arxiv.org/ abs/1907.11692. ↗
  25. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142– 150, Portland, Oregon, USA, June 2011. Association f… find our card
  26. T. Mickus, D. Paperno, and M. Constant. How to dissect a muppet: The structure of transformer embedding spaces. arXiv preprint arXiv:2206.03529, 2022. find
  27. E. H. Moore. On the reciprocal of the general algebraic matrix. Bull. Am. Math. Soc., 26:394–395, 1920. nostalgebraist. interpreting gpt: the logit lens, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/… ↗
  28. R. Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, volume 51, pages 406–413. Cambridge University Press, 1955. find
  29. O. Press and L. Wolf. Using the output embedding to improve language models, 2016. URL https://arxiv.org/abs/1608.05859. ↗ our card
  30. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. In OpenAI blog, 2019. find
  31. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. URL https://arxiv.org/abs/1910.10683. ↗
  32. A. Rogers, O. Kovaleva, and A. Rumshisky. A primer in bertology: What we know about how bert works, 2020. URL https://arxiv.org/abs/ 2002.12327. ↗ our card
  33. W. Rudman, N. Gillman, T. Rayne, and C. Eickhoff. Isoscore: Measuring the uniformity of vector space utilization. CoRR, abs/2108.07344, 2021. URL https://arxiv.org/abs/2108.07344. — [arXiv:2108.07344](https://arxiv.org/abs/2108.07344) ↗
  34. T. Sellam, S. Yadlowsky, I. Tenney, J. Wei, N. Saphra, A. D’Amour, T. Linzen, J. Bastings, I. R. Turc, J. Eisenstein, D. Das, and E. Pavlick. The multi-BERTs: BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?… find
  35. X. Shi, I. Padhi, and K. Knight. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1159. URL https://ac… find our card
  36. S. Sukhbaatar, E. Grave, G. Lample, H. Jegou, and A. Joulin. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470, 2019. — [arXiv:1907.01470](https://arxiv.org/abs/1907.01470) find
  37. I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL https://a… find our card
  38. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2017. URL https://arxiv.org/abs/1706.03762. ↗
  39. E. Voita, R. Sennrich, and I. Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives, 2019a. URL https://arxiv.org/abs/1909.01380. ↗ our card
  40. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, Italy, July 2019b. A… find our card
  41. L. Wang, J. Huang, K. Huang, Z. Hu, G. Wang, and Q. Gu. Improving neural language generation with spectrum control. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ByxY8CNtvr. ↗
  42. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing.… find our card
  43. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/ 2205.010… ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/