Elamur Research
☰
home / papers / 2023 / The Linear Representation Hypothesis and the Geometry of Large Language Models

The Linear Representation Hypothesis and the Geometry of Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2311.03658

Summary

The paper asks what it precisely means for an LLM to encode a high-level concept (such as language, gender, or tense) "linearly," and which inner product makes geometric tools like cosine similarity and projection meaningful. Three informal readings circulate in the literature: a shared subspace, linear probing (measurement), and steering vectors (intervention). The authors frame concepts as latent causal variables and use counterfactual word or context pairs to define linear representations in two spaces: the unembedding (output word) space and the embedding (input context) space. They prove that the unembedding version corresponds to linear probing and the embedding version to steering. They then define a "causal inner product," under which concepts that can vary independently are orthogonal. This inner product makes the two representation spaces coincide and can be estimated from the unembedding matrix. Experiments on LLaMA-2 show that such representations exist, that the causal inner product reflects semantic structure, and that concept directions built from word pairs work for both probing and steering. The work gives interpretability and control methods a principled geometric basis.

Key points

  • The paper formalizes three notions of linear representation (subspace, measurement, intervention) and proves how they relate. Unembedding representations act as an "ideal probe" that ignores correlated off-target concepts.
  • Two concepts are "causally separable" when they can be varied freely and independently. Examples are English⇒French and male⇒female, but not English⇒French and English⇒Russian.
  • The causal inner product maps the embedding and unembedding representations onto a single direction. It is non-Euclidean and is estimated from the LLM's unembedding matrix.
  • In a steering demo with the context "Long live the ", the authors add the male⇒female embedding direction with scale α from 0 to 0.4. As α grows, "queen" becomes the top-1 prediction and "king" falls out of the top 5.
  • The experiments use LLaMA-2, and the code is released at github.com/KihoPark/linear_rep_geometry. The analysis covers only the final embedding and unembedding spaces, not intermediate layers or model parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

148 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.03658 · arXiv 2311.03658

@misc{park2023linear,
  title = {{The Linear Representation Hypothesis and the Geometry of Large Language Models}},
  author = {Park, Kiho and Choe, Yo Joong and Veitch, Victor},
  year = {2023},
  doi = {10.48550/arxiv.2311.03658},
  eprint = {2311.03658},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.03658}
}

References

59 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=ryF7rTqgl. ↗ our card
  2. Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385–399, 2016. find
  3. Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022. find our card
  4. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10–21, Berlin, Germany, August 2016. Association for Computational Linguistics. d… find our card
  5. Chang, T., Tu, Z., and Bergen, B. The geometry of multilingual language model representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 119–136, 2022. find
  6. Chen, B., Fu, Y., Xu, G., Xie, P., Tan, C., Chen, M., and Jing, L. Probing BERT in hyperbolic spaces. In International Conference on Learning Representations, 2021. find
  7. Chiang, H.-Y., Camacho-Collados, J., and Pardos, Z. Understanding the source of semantic regularities in word embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 119–131, 2020. find
  8. Choe, Y. J., Park, K., and Kim, D. word2word: A collection of bilingual lexicons for 3,564 language pairs. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 3036–3045, 2020. find
  9. Drozd, A., Gladkova, A., and Matsuoka, S. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical papers, pp. 3519–3530, 2016. find
  10. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021. find
  11. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. find
  12. Ethayarajh, K. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL… find
  13. Fournier, L., Dupoux, E., and Dunbar, E. Analogies minus analogy test: measuring regularities in word embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 365–375, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.conll-1.29.… find
  14. Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 30–45, 2022. find our card
  15. Gladkova, A., Drozd, A., and Matsuoka, S. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL Student Research Workshop, pp. 8–15, 2016. find
  16. Goldberg, Y. and Levy, O. word2vec explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014. — [arXiv:1402.3722](https://arxiv.org/abs/1402.3722) find
  17. Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, art. arXiv:2310.02207, October 2023. doi: 10.48550/arXiv.2310.02207. find
  18. Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916, 2023. find
  19. Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas, J., Belinkov, Y., and Bau, D. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124, 2023. find
  20. Hewitt, J. and Manning, C. D. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129–4138, 2019. find our card
  21. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016. find
  22. Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018. — [arXiv:1812.02230](https://arxiv.org/abs/1812.02230) find
  23. Hyvarinen, A. and Morioka, H. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, 29, 2016. find
  24. Jiang, Y., Aragam, B., and Veitch, V. Uncovering meanings of embeddings via partial orthogonality. arXiv preprint arXiv:2310.17611, 2023. — [arXiv:2310.17611](https://arxiv.org/abs/2310.17611) find
  25. Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207–2217. PMLR, 2020. find
  26. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learning, pp. 2668–2677. PMLR, 2018. find
  27. Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71, 2018. find our card
  28. Lample, G., Conneau, A., Ranzato, M., Denoyer, L., and Jégou, H. Word translation without parallel data. In International Conference on Learning Representations, 2018. find
  29. Levy, O. and Goldberg, Y. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pp. 171–180, 2014. find
  30. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130, 2020. find our card
  31. Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent world representations: Exploring a sequence model trained on a synthetic task. In International Conference on Learning Representations, 2022. find
  32. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  33. Merullo, J., Eickhoff, C., and Pavlick, E. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.16130, 2023. find
  34. Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. find
  35. Mikolov, T., Le, Q. V., and Sutskever, I. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013a. find
  36. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26, 2013b. find
  37. Mikolov, T., Yih, W.-T., and Zweig, G. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746–751, 2013c. find
  38. Mimno, D. and Thompson, L. The strange geometry of skip-gram with negative sampling. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2873–2878, Copenhagen, Denmark, 2017. Association for Computational Linguist… find
  39. Moran, G. E., Sridhar, D., Wang, Y., and Blei, D. M. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:2110.10804, art. arXiv:2110.10804, October 2021. doi: 10.48550/arXiv.2110.10804. — [doi:10.48550/arxiv.2110.10804](https://doi.org/10.48550/arxiv.2110.10804) find
  40. Nanda, N., Lee, A., and Wattenberg, M. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023. nostalgebraist. Interpreting GPT: the logit lens, 2020. URL https://www.alignmentforum. org/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-… ↗
  41. OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023. arXiv preprint Peng, X., Stevenson, M., Lin, C., and Li, C. Understanding linearity of cross-lingual word embedding mappings. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview. net/forum?id=8HuyXvbvqX. find our card
  42. Pennington, J., Socher, R., and Manning, C. D. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, 2014. find
  43. Perera, P., Trager, M., Zancato, L., Achille, A., and Soatto, S. Prompt algebra for task composition. arXiv preprint arXiv:2306.00310, 2023. — [arXiv:2306.00310](https://arxiv.org/abs/2306.00310) find
  44. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018. find
  45. Reif, E., Yuan, A., Wattenberg, M., Viegas, F. B., Coenen, A., Pearce, A., and Kim, B. Visualizing and measuring the geometry of BERT. Advances in Neural Information Processing Systems, 32, 2019. find our card
  46. Rogers, A., Kovaleva, O., and Rumshisky, A. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866, 2021. find our card
  47. Ruder, S., Vulić, I., and Søgaard, A. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631, 2019. find
  48. Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612–634, 2021. find
  49. Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023. find
  50. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… find
  51. Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., and Soatto, S. Linear spaces of meanings: Compositional structures in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15395–15404, 2023. find
  52. Turner, A. M., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, art. arXiv:2308.10248, August 2023. doi: 10.48550/arXiv.2308.10248. find our card
  53. Ushio, A., Anke, L. E., Schockaert, S., and Camacho-Collados, J. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  54. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. find
  55. Vylomova, E., Rimell, L., Cohn, T., and Baldwin, T. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1671… find
  56. Wang, Z., Gui, L., Negrea, J., and Veitch, V. Concept algebra for score-based conditional models. arXiv preprint arXiv:2302.03693, 2023. — [arXiv:2302.03693](https://arxiv.org/abs/2302.03693) find
  57. Zhu, X. and de Melo, G. Sentence analogies: Linguistic regularities in sentence embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3389–3400, 2020. find
  58. Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979–12990. PMLR, 2021. find
  59. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, Z., and Hendrycks, D. Representation engineering: A top-down approach to AI tr… find our card