Elamur Research
☰
home / papers / 2023 / The Internal State of an LLM Knows When It's Lying

The Internal State of an LLM Knows When It's Lying

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.findings-emnlp.68.pdf

Summary

Large language models often produce false claims in an assured tone. This paper asks whether a model's hidden activations carry a detectable signal of whether a statement is true. The authors propose SAPLMA, a small feedforward classifier whose name refers to judging whether a statement is accurate by reading the internal activations of an LLM. It takes the hidden-layer values recorded while an LLM reads or generates a sentence and outputs the probability that the sentence is true.

To train and evaluate it, they built a true-false dataset covering six topics. They always tested on a topic left out of training, so the probe must capture the model's general sense of truth rather than topic-specific patterns. On balanced test sets, SAPLMA reached 71–83% average accuracy depending on the base model. It clearly outperformed few-shot prompting of the same LLM. The raw sentence probability assigned by the LLM proved a weak indicator of truth because it depends on sentence length and word frequency.

The method is cheap at inference and leaves model weights untouched. It could be used to flag or regenerate dubious LLM output.

Key points

  • Base models were OPT-6.7b and LLAMA2-7b. Per-topic accuracy was roughly 60–80% for OPT-6.7b and 70–90% for LLAMA2-7b. Few-shot prompting peaked at about 56% for OPT-6.7b.
  • Evaluation is out-of-distribution: the classifier trains on five topics and is tested on the held-out sixth.
  • The authors contrast their method with Contrast-Consistent Search (CCS), which gains about 4% over 0-shot querying. SAPLMA gains close to 20%, and unlike CCS it needs no in-topic training data or question rephrasing.
  • In a qualitative comparison using OPT-6.7b's 28th-layer activations, SAPLMA scores tracked truth labels far better than LLM-assigned sentence probabilities.
  • The authors released the true-false dataset and a recipe for generating such data. Experiments were run only in English, and the authors note that the model's biases may carry over to the classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

261 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 249 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.68 · arXiv 2304.13734

@inproceedings{azaria2023internal,
  title = {{The Internal State of an LLM Knows When It's Lying}},
  author = {Azaria, Amos and Mitchell, Tom},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.68},
  eprint = {2304.13734},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.68}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Proc… find
  2. Antonio Bella, Cèsar Ferri, José Hernández-Orallo, and María José Ramírez-Quintana. 2010. Calibration of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, pages 128–146. IGI Global. find
  3. Michael Bommarito II and Daniel Martin Katz. 2022. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402. — [arXiv:2212.14402](https://arxiv.org/abs/2212.14402) find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. find our card
  6. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… find
  7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  8. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. arXiv preprint arXiv:2104.13346. find our card
  9. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813. find our card
  10. Konstantinos I Roumeliotis, Nikolaos D Tselikas, and Dimitrios K Nasiopoulos. 2023. Llama 2: Early adopters’ utilization of meta’s new open-source pre-trained model. find
  11. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827. find
  12. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355. find our card
  13. Yuanyuan Chen and Zhang Yi. 2021. Adaptive sparse dropout: Learning the certainty and uncertainty in deep neural networks. Neurocomputing, 450:354– 361. find
  14. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The fever2. 0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 1–6. find
  15. David Dale, Elena Voita, Loïc Barrault, and Marta R Costa-jussà. 2022. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better. arXiv preprint arXiv:2212.08597. find
  16. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find
  17. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241. find
  18. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. find
  19. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. find our card