Elamur Research⌕
☰
home / papers / 2024 / To Believe or Not to Believe Your LLM

To Believe or Not to Believe Your LLM

Paper · 2024 · Yasin Abbasi Yadkori, Ilja Kuzborskij, Andras Gyorgy et al.

Summary

Large language models sometimes produce confident-sounding but false answers. Common detectors, such as likelihood or entropy thresholds, treat any spread in the output as a warning sign. This fails for questions with several legitimate answers, where spread reflects real-world variety (aleatoric uncertainty) rather than missing knowledge (epistemic uncertainty).

The Google DeepMind authors separate the two. Their key idea is that independent answers drawn from the true distribution should not influence each other. They repeatedly prompt the model with the query plus its own earlier answers, building a joint distribution over several responses. If new answers depend strongly on the ones already in context, the model's knowledge is shaky. They formalize this as an information-theoretic gap, lower-bound it by a mutual information (MI) score, and provide a finite-sample estimator. Hallucination detection then thresholds this score, with the threshold set by calibration.

On TriviaQA, AmbigQA and a WordNet-derived multi-answer set, the MI method matches semantic entropy. It clearly beats semantic entropy on mixed single/multi-answer data. This matters for any QA system that must decide when to abstain.

Key points

  • Probability amplification: repeating a wrong answer in the prompt can push the model toward it, but only when the model's knowledge is weak. Confidently known facts (e.g., the capital of the UK) resist this.
  • Mechanistic explanation: analysis of a single attention head suggests the outcome depends on how much the query overlaps with the dominant principal components of the key-query product. Strong overlap means the model answers from stored knowledge; weak overlap means it copies the repeated text.
  • Results on standard benchmarks: on TriviaQA and AmbigQA, the MI detector beats a likelihood baseline and roughly ties with semantic entropy (S.E.). These datasets consist mostly of single-answer or low-entropy queries.
  • Mixed datasets: 842 high-entropy WordNet queries (entropy > 0.7) were combined with an equal number of TriviaQA or AmbigQA items. On these mixed sets, S.E. recall collapses toward zero for high-entropy queries, while the MI method keeps recall high with few errors.
  • Calibration protocol: thresholds were calibrated on 500+500 item splits at target loss rates of 0.05 (TriviaQA) and 0.15 (AmbigQA), averaged over 10 runs. The method requires no retraining, only iterative prompting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Yasin Abbasi Yadkori, Ilja Kuzborskij, Andras Gyorgy and Csaba Szepesvari. 2024. To Believe or Not to Believe Your LLM. arXiv:2406.02543.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.02543 · arXiv 2406.02543

@misc{yadkori2024believe,
  title = {{To Believe or Not to Believe Your LLM}},
  author = {Yasin Abbasi Yadkori, Ilja Kuzborskij, Andras Gyorgy and Csaba Szepesvari},
  year = {2024},
  doi = {10.48550/arxiv.2406.02543},
  eprint = {2406.02543},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.02543}
}

References

62 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. Distinguishing the knowable from the unknowable with language models, 2024. URL https://arxiv.org/abs/2402. 03563. ↗
  2. Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning, 16(4):494–591, 2023. find
  3. Javier Antorán, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2020. find
  4. Amos Azaria and Tom Mitchell. The internal state of an LLM knows when its lying. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. — The Internal State of an LLM Knows When It's Lying
  5. Daniel Berend and Aryeh Kontorovich. The missing mass problem. Statistics & Probability Letters, 82(6): 1102–1110, 2012. find
  6. Daniel Berend and Aryeh Kontorovich. On the concentration of the missing mass. Electronic Communications in Probability, 2013. find
  7. Daniel Berend, Aryeh Kontorovich, and Gil Zagdanski. The expected missing mass under an entropy constraint. Entropy, 19(7):315, 2017. find
  8. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learing (ICML), 2015. find
  9. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint ar…
  10. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR), 2023. find
  11. Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge University Press, 2006. find
  12. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. In International Conference on Learning Representations (ICLR), 2024a. — INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
  13. Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id= s3e8poX… ↗
  14. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation, 2023.
  15. Jeremy R. Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2019.
  17. Vikranth Dwaracherla, Zheng Wen, Ian Osband, Xiuyuan Lu, Seyed Mohammad Asghari, and Benjamin Van Roy. Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835-8856. find
  18. S. Farquhar, J. Kossen, L. Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. 2024. find
  19. Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998. find
  20. Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016. find
  21. William A Gale and Geoffrey Sampson. Good-turing frequency estimation without tears. Journal of quantitative linguistics, 2(3):217–237, 1995. find
  22. Gemini Team, Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [Online; accessed 01-February-2024].
  23. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensembling. In International Conference on Machine Learing (ICML), 2024. find
  24. Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. In Workshop on Challenges in Deployable Generative AI at International Conference on Machine Learning, 2024. find
  25. Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison. Experts don’t cheat: Learning what you don’t know by predicting pairs. arXiv preprint arXiv:2402.08733, 2024. — [arXiv:2402.08733](https://arxiv.org/abs/2402.08733) find
  26. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Transactions of the Association for Computational Linguistics (ACL), pages 1601–1611, 2017.
  27. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, and et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. find
  28. Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Transactions of the Association for Computational Linguistics (ACL), 2020.
  29. Satyapriya Krishna. On the intersection of self-correction and trust in language models, 2023. URL https://arxiv.org/abs/2311.02801. — [arXiv:2311.02801](https://arxiv.org/abs/2311.02801) ↗
  30. Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Understanding the effects of iterative prompting on truthfulness. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=KjazcKPMME. ↗
  31. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations (ICLR), 2023.
  32. Philippe Laban, Lidiya Murakhovs’ka, Caiming Xiong, and Chien-Sheng Wu. Are you sure? challenging llms leads to performance drops in the flipflop experiment, 2024. URL https://arxiv.org/abs/2311. 08596. ↗
  33. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017a. find
  34. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Conference on Neural Information Processing Systems (NeurIPS), 2017b. find
  35. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Transactions of the Association for Computational Linguistics (ACL), 2023.
  36. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection, 2024. URL https://arxiv.org/abs/2403.09972. ↗
  37. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187, 2023. find
  38. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  39. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR), 2020. find
  40. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. find
  41. David A McAllester and Robert E Schapire. On the convergence rate of good-turing estimators. In Conference on Computational Learning Theory (COLT), 2000. find
  42. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents’ overconfidence through linguistic calibration. In Transactions of the Association for Computational Linguistics (ACL), 2022. find
  43. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find
  44. R. M. Neal. Bayesian learning for neural networks. Springer Science & Business Media, 2012. find
  45. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint arXiv:2211.05655, 2022.
  46. Mesrob I Ohannessian and Munther A Dahleh. Distribution-dependent performance of the good-turing estimator for the missing mass. In 19th International Symposium on Mathematical Theory of Networks and Systems, MTNS, 2010. find
  47. Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Conference on Neural Information Processing Systems (NeurIPS), volume 29, 2016.
  48. Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2023. find
  49. Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112–1130, 2014. find
  50. Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning. Cambridge University Press, 2024. find
  51. Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, and Nicolas Papernot. Selective classification via neural network training dynamics. arXiv preprint arXiv:2205.13532, 2022. — [arXiv:2205.13532](https://arxiv.org/abs/2205.13532) find
  52. Shauli Ravfogel, Yoav Goldberg, and Jacob Goldberger. Conformal nucleus sampling. In Transactions of the Association for Computational Linguistics (ACL), pages 27–34. Association for Computational Linguistics, 2023. find
  53. Robert J Tibshirani and Bradley Efron. An introduction to the bootstrap. Monographs on statistics and applied probability, 57(1), 1993. find
  54. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2022.
  55. Ziyu Wang and Chris Holmes. On subjective uncertainty quantification and calibration in natural language generation, 2024. URL https://arxiv.org/abs/2406.05213. — [arXiv:2406.05213](https://arxiv.org/abs/2406.05213) ↗
  56. Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv preprint arXiv:2107.09224, 2022. — [arXiv:2107.09224](https://arxiv.org/abs/2107.09224) find
  57. Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating llm hallucinations via conformal abstention. arXiv preprint arXiv:2405.01563, 2024. — [arXiv:2405.01563](https://arxiv.org/abs/2405.01563) find
  58. Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=7DbIyQlfaO. ↗
  59. Gal Yona, Roee Aharoni, and Mor Geva. Narrowing the knowledge evaluation gap: Open-domain question answering with multi-granularity answers. arXiv preprint arXiv:2401.04695, 2024. find
  60. Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. Sac3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
  61. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In In International Conference on Machine Learning, 2021.
  62. Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. Knowing what LLMs do not know: A simple yet effective self-detection method. arXiv preprint arXiv:2310.17918, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yadkori-2024-believe-believe-llm/