Elamur Research⌕
☰
home / papers / 2024 / Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?

Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?

Paper · 2024 · Gal Yona, Roee Aharoni and Mor Geva

Summary

LLMs often state false facts in a confident, fluent tone, which can lead users to over-trust them. The authors argue that hedging in a response should reflect the model's own uncertainty, not external correctness. They define faithful response uncertainty, a per-example score equal to one minus the average gap between how decisively each assertion is worded and the model's internal confidence in it. Both needless hedging and missing hedging lower the score. Decisiveness is the probability a reader would assign to an assertion being true based only on its wording. Confidence is estimated through consistency: the share of resampled answers that do not contradict the original assertion. Gemini Ultra acts as a judge and scores both quantities. The authors test Gemini Nano, Pro and Ultra, gpt-3.5-turbo and gpt-4-turbo on unambiguous questions from PopQA and Natural Questions. Models answer decisively even when they are clearly uncertain. When prompted to hedge, the hedges they produce barely track their actual confidence. The authors conclude that better alignment is needed before LLMs can be trusted to communicate uncertainty in words.

Key points

  • Confidence is computed from k = 20 extra sampled answers per question. Few-shot prompts extract the assertions and detect contradictions between them.
  • The judge's decisiveness scores for hedging phrases (e.g., "Highly likely") agree with human survey perceptions from Fagen-Ulmschneider (2023). Its confidence scores reached a correlation of 0.97 with author labels on 100 examples.
  • Four prompting setups were compared under greedy decoding: Vanilla, Granularity, Uncertainty and Uncertainty+.
  • The main metric is conditional mean faithful generation (cMFG). Most model–method–dataset combinations scored close to the 0.5 baseline. The best scores were 0.70 for Gemini Ultra and 0.63 for GPT-4.
  • Only the Uncertainty-style prompts reduced mean decisiveness, and decisiveness still correlated only weakly with confidence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓Tel Aviv UniversityTel Aviv · Israel82 works · 10,655 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Gal Yona, Roee Aharoni and Mor Geva. 2024. Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.443

@inproceedings{yona2024can,
  title = {{Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?}},
  author = {Gal Yona, Roee Aharoni and Mor Geva},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.443},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.443}
}

References

54 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018.
  3. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734.
  4. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703. — [arXiv:2307.15703](https://arxiv.org/abs/2307.15703) find
  5. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2024. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200.
  6. Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–21. find
  7. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827. find
  8. A Philip Dawid. 1982. The well-calibrated bayesian. Journal of the American Statistical Association, 77(379):605–610. find
  9. Marie-Catherine De Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational linguistics, 38(2):301–333.
  10. Ran El-Yaniv et al. 2010. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5). find
  11. Wade Fagen-Ulmschneider. 2023. Perception of probability words. Ms., UIUC, 05-24-2023. find
  12. Bruce Fraser. 2010. Pragmatic competence: The case of hedging. In New approaches to hedging, pages 15–34. Brill. find
  13. Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. Advances in neural information processing systems, 30. find
  14. Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904. find
  15. Gemini-Team. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  16. Anastasia Giannakidou. 1999. Affective dependencies. Linguistics and Philosophy, 22:367–421. find
  17. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? evaluating uncertainty in neural text generators against human production variability. arXiv preprint arXiv:2305.11707. find
  18. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. find
  19. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  20. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.
  21. Marie Juanchich, Amélie Gourdon-Kanhukamwe, and Miroslav Sirota. 2017. “i am uncertain” vs “it is uncertain”. how linguistic markers of the uncertainty source affect uncertainty communication. Judgment and Decision Making, 12(5):445–465. find
  22. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  23. Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. 2024. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612.
  24. Sunnie SY Kim, Q Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. 2024. " i’m not sure, but...": Examining the impact of large language models’ uncertainty expression on user reliance and trust. arXiv preprint arXiv:2405.00623. — [arXiv:2405.00623](https://arxiv.org/abs/2405.00623) find
  25. Svenja Kranich. 2011. To hedge or not to hedge: the use of epistemic modal expressions in popular science in english texts, english–german translations, and german original texts. find
  26. Lea Krause, Wondimagegnhue Tufa, Selene Baez Santamaria, Angel Daza, Urja Khurana, and Piek Vossen. 2023. Confidently wrong: Exploring the calibration and expression of (un)certainty of large language models in a multilingual setting. In Proceedings of the Workshop on Multimodal, Multilingual Natura… find
  27. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. find
  28. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic…
  29. George Lakoff. 1973. Hedges: A study in meaning criteria and the logic of fuzzy concepts. Journal of philosophical logic, 2(4):458–508.
  30. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
  31. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. find
  32. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462. find
  33. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
  34. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.
  35. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. find
  36. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. find
  37. Paul D Windschitl and Gary L Wells. 1996. Measuring psychological uncertainty: Verbal versus numeric methods. Journal of Experimental Psychology: Applied, 2(4):343. find
  38. Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. find
  39. Hongshen Xu, Zichen Zhu, Da Ma, Situo Zhang, Shuai Fan, Lu Chen, and Kai Yu. 2024. Rejection improves reliability: Training llms to refuse unknown questions using rl from knowledge feedback. arXiv preprint arXiv:2403.18349. — [arXiv:2403.18349](https://arxiv.org/abs/2403.18349) find
  40. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.
  41. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645. find
  42. Samir Passi and Mihaela Vorvoreanu. 2022. Overreliance on ai literature review. Microsoft Research. find
  43. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070.
  44. Alexandre Piché, Aristides Milios, Dzmitry Bahdanau, and Chris Pal. 2024. Llms can learn self-restraint through iterative self-reflection. arXiv preprint arXiv:2405.13022. find
  45. Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023. Alignment for honesty. arXiv preprint arXiv:2312.07000. find
  46. Gal Yona, Roee Aharoni, and Mor Geva. 2024. Narrowing the knowledge evaluation gap: Open-domain question answering with multi-granularity answers. arXiv preprint arXiv:2401.04695. find
  47. Hiyori Yoshikawa and Naoaki Okazaki. 2023. Selective-LAMA: Selective prediction for confidence-aware evaluation of language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2017–2028, Dubrovnik, Croatia. Association for Computational Linguistics. find
  48. Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024. Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation. arXiv preprint arXiv:2402.09267.
  49. Alf C Zimmer. 1983. Verbal vs. numerical processing of subjective probabilities. In Advances in psychology, volume 16, pages 159–182. Elsevier. find
  50. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2023a. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401.
  51. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023b. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975. find
  52. Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q Vera Liao, and Jennifer Wortman Vaughan. 2023. Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in ai-powered code completions. arXiv preprint arXiv:2302.07248. — [arXiv:2302.07248](https://arxiv.org/abs/2302.07248) find
  53. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  54. Thomas S Wallsten, David V Budescu, Rami Zwick, and Steven M Kemp. 1993. Preferences and reasons for communicating probabilistic information in verbal or numerical terms. Bulletin of the Psychonomic Society, 31(2):135–138. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yona-2024-can-large-language-models-faithfully/