The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2310.06824

Summary

LLMs sometimes state things that are false even when they seem to "know" better. Earlier attempts to detect truthfulness by probing internal activations generalized poorly, which raised doubts about whether those probes found truth or only features that correlate with it. Marks and Tegmark study this question in LLaMA-2 models. They use curated sets of short, unambiguous factual statements (city locations, Spanish–English word translations, number comparisons), along with negated, conjunctive and disjunctive versions of them. They add noisier uncurated sets from earlier work and a likely control set, which separates truth from how probable the text is. Patching experiments first show which hidden states matter most: those over the statement's last token and its closing punctuation. In PCA projections of these states, true and false statements separate along a clear linear axis. Probes trained on one dataset carry over to others, and shifting activations along a probe direction flips the model's TRUE/FALSE judgments. The authors conclude that sufficiently large models encode factual truth linearly. They propose mass-mean (difference-in-means) probing, a simple method that finds the most causally relevant directions.

Key points

Datasets: the curated sets include cities (1,496 statements), sp_en_trans (354), larger_than and smaller_than (1,980 each), and neg_*, conj and disj variants. The uncurated sets are companies_true_false (1,200), common_claim_true_false (4,450) and counterfact_true_false (31,960). The likely control set has 10,000 nonfactual texts whose final token is either LLaMA-13B's top completion or its 100th most likely one.
Truth is not the same as probability: in LLaMA-2-70B, a statement's truth and its log-probability are negatively correlated for neg_cities (r = −.63) and neg_sp_en_trans (r = −.89). For the non-negated sets the correlation is positive (.85 and .95). Because of this split, the linear structure cannot simply be a "likely text" signal.
Where truth is stored: patching on LLaMA-2-13B finds three groups of causally important states: the subject entity, the summary over the end of the statement, and the final prediction. The truth representation sits in the middle group.
Causal tests: interventions were scored on sp_en_trans with normalized indirect effects (NIE). Mass-mean directions beat logistic regression (LR) and CCS in 7 of 8 settings. For LLaMA-2-70B they reach NIE ≈ 1, meaning the swap was essentially complete.
Resources and limits: the code, datasets and an interactive data explorer are at github.com/saprmarks/geometry-of-truth. Only LLaMA-2 models were tested, and simple statements cannot separate "true" from related ideas such as "commonly believed".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.06824 · arXiv 2310.06824

@misc{marks2023geometry,
  title = {{The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets}},
  author = {Marks, Samuel and Tegmark, Max},
  year = {2023},
  doi = {10.48550/arxiv.2310.06824},
  eprint = {2310.06824},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.06824}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color, 2021. find
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. find our card
Amos Azaria and Tom Mitchell. The internal state of an llm knows when its lying, 2023. find our card
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907375117. URL https://www.pnas.org/content/earl… — [doi:10.1073/pnas.1907375117](https://doi.org/10.1073/pnas.1907375117) ↗
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=awIpKpwTwF. ↗
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs. ↗
Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch, 2023. find
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, 2021. URL https://docs.google.com/document/d/1WwsnJQstPq91 Yh-Ch2XRL8H EpsnjrC1dwZXR37PC8/edit\#heading=h.jrzi4atzacns. ↗
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. find
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In AAAI Conference on Artificial Intelligence, 2018. URL https: //api.semanticscholar.org/CorpusID:56895415. find
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circu… find
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Associ… find
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179–188, 1936. doi: https://doi.org/10.1111/j.1469-1809.1936.tb02137.x. URL https: //onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809.1936.tb02137.x. — [doi:10.1111/j.1469-1809.1936.tb02137.x](https://doi.org/10.1111/j.1469-1809.1936.tb02137.x) ↗
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third Blackb… find our card
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Inf… find
Geonames. All cities with a population \> 1000, 2023. URL https://download.geonames. org/export/dump/. ↗
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023. find
Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. — [doi:10.23915/distill.00030](https://doi.org/10.23915/distill.00030) ↗
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023. find
B. A. Levinstein and Daniel A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks, 2023. find
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),… find
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id… ↗
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023b. find our card
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguisti… find our card
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022. find
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: //openreview.net/forum?id=mbW GT3ZN-. find
OpenAI. Gpt-4 technical report, 2023. find our card
Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions, 2023. find
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/ forum?id=gJcEM8sxHK. ↗
Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain,… find our card
Plotly Technologies Inc. Collaborative data science, 2015. URL https://plot.ly. ↗
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024. find
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285–1303, 2022. doi: 10.1162/tacl a 00519. URL https://aclanthology.org/2022.tacl-1.74. ↗
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018. find
Jacob Steinhardt. Emergent deception and emergent optimization, 2023. URL https: //bounded-regret.ghost.io/emergent-deception-optimization/. find
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023. find
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processi… find
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132–11152, Abu Dhabi, United Arab Emirates, December… find
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendry… find our card