Elamur Research⌕
☰
home / papers / 2023 / Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Paper · 2023 · Lorenz Kuhn, Yarin Gal and Sebastian Farquhar

Summary

The paper addresses how far the answers of large language models can be trusted in free-form text generation, particularly in question-answering systems. The authors show that ordinary entropy computed from token probabilities gives a misleading picture. The same idea can be phrased in different ways, so a model appears "uncertain" even when it is only hesitating over the wording. To correct this, they propose semantic entropy. For a single question, several answers are sampled from the model and grouped into clusters by meaning: two answers are considered equivalent if each logically follows from the other (bidirectional entailment). The probabilities of the answers within each cluster are summed, and entropy is then computed over the distribution of meanings. The method requires no training, additional data or annotation, and works with a single off-the-shelf model without any modifications to it. It was evaluated on OPT models of various sizes and on the TriviaQA (closed-book) and CoQA (open-book) datasets. Semantic entropy predicts whether the model's answer is correct better than the baseline methods, and its advantage grows with model size. In practice, it offers a simple way to flag unreliable LLM answers.

Key points

  • Comparison with baselines: semantic entropy was compared with predictive entropy, length-normalised entropy (a single-model variant of the method of Malinin & Gales), lexical similarity based on Rouge-L, p(True) following Kadavath et al. (few-shot, 10 examples), and margin probability. Performance was measured with AUROC.
  • A toy example with a question about the capital of France: if the answers "Paris" and "It's Paris" are merged into a single cluster, entropy drops from 0.94 to 0.33. This shows that the model's actual uncertainty is low.
  • CoQA (correctness criterion Rouge-L > 0.3): AUROC is ≈0.767 for semantic entropy and ≈0.753 for normalised entropy, with model answer accuracy of ≈0.82. The conclusions hardly depend on the choice of correctness metric, with the exception of exact matching.
  • Margin probability (the difference between the probabilities of the two best answers from beam search) achieves an AUROC of only 0.54 on CoQA with the 13B model. The reason is that the method does not distinguish paraphrases from genuinely different answers.
  • Model size and sampling: the experiments cover OPT models from 1.3B/2.7B up to 30B; larger models are more accurate, and semantic entropy performs better with them. The authors also show that answer sampling must balance diversity against accuracy. Reliable uncertainty estimation requires far fewer samples than previous work assumed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 214 works

Cite

Cite as Lorenz Kuhn, Yarin Gal and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. arXiv:2302.09664.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.09664 · arXiv 2302.09664

@misc{kuhn2023semantic,
  title = {{Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation}},
  author = {Lorenz Kuhn, Yarin Gal and Sebastian Farquhar},
  year = {2023},
  doi = {10.48550/arxiv.2302.09664},
  eprint = {2302.09664},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.09664}
}

References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXiv, 2016. 1
  2. Ion Androutsopoulos and Prodromos Malakasiotis. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research, 38:135–187, 2010. 6
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 1
  5. Peter W Culicover. Paraphrase generation and information retrieval from stored text. Mech. Transl. Comput. Linguistics, 11(3-4):78–88, 1968. 6 find
  6. Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295– 302, 2020. 6 find
  7. Samuel Fernando and Mark Stevenson. A semantic similarity approach to paraphrase detection. In Proceedings of the 11th annual research colloquium of the UK special interest group for computational linguistics, pp. 45–52. Citeseer, 2008. 6 find
  8. Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim G J Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. Benchmarking Bayesian Deep Learning with Diabetic Retinopathy Diagnosis. Bayesian Deep Learning Workshop at NeurIPS, 2019. 7 find
  9. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 539–555, 2020. 2, 7
  10. Yarin Gal et al. Uncertainty in deep learning. PhD thesis, 2016. 1, 2 find
  11. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André FT Martins. Uncertainty-aware machine translation evaluation. arXiv preprint arXiv:2109.06352, 2021. 6 find
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020a. 5
  13. Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. arXiv preprint arXiv:2012.11747, 2020b. 6 find
  14. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved Problems in ML Safety. arXiv, 2022. 1
  15. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. 1
  16. Fuad Issa, Marco Damonte, Shay B Cohen, Xiaohui Yan, and Yi Chang. Abstract meaning representation for paraphrase detection. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.… find
  17. Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint arXiv:2102.08501, 2021. 3 — [arXiv:2102.08501](https://arxiv.org/abs/2102.08501) find
  18. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977, 2021. 6 find
  19. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. 7
  20. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. 1, 2, 3, 6, 7, 8, 9, 18 find
  21. Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017. 1, 2
  22. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2796–2804. PMLR, 2018. 1 find
  23. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. 1 find
  24. Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 605–612, 2004. 7
  25. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022a. 2, 3, 6 find
  26. Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4160–4173, 2022b. 6, 8, 19 find
  27. David Mackay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, 2003. 4 find
  28. Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018. 3 find
  29. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020. 1, 2, 3, 4, 5, 7, 8 find
  30. Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. Regression prior networks. arXiv preprint arXiv:2006.11590, 2020. 3 — [arXiv:2006.11590](https://arxiv.org/abs/2006.11590) find
  31. Sabrina J Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. arXiv preprint arXiv:2012.14983, 2020. 6 — [arXiv:2012.14983](https://arxiv.org/abs/2012.14983) find
  32. Kenton Murray and David Chiang. Correcting length bias in neural machine translation. arXiv preprint arXiv:1808.10006, 2018. 3, 4
  33. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pp. 3956–3965. PMLR, 2018. 6
  34. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. 1 find
  35. Sebastian Padó, Daniel Cer, Michel Galley, Dan Jurafsky, and Christopher D Manning. Measuring machine translation quality as semantic equivalence: A metric based on entailment features. Machine Translation, 23(2):181–193, 2009. 6 find
  36. Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. 7
  37. Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1–39, 2022. 6 find
  38. Richard Socher, Eric Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in neural information processing systems, 24, 2011. 6 find
  39. Jeff Speaks. Theories of Meaning. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, 2021. 3 find
  40. Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672, 2021. 6
  41. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690, 2021. 6 — [arXiv:2104.14690](https://arxiv.org/abs/2104.14690) find
  42. Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. Uncertainty estimation and reduction of pre-trained models for text regression. Transactions of the Association for Computational Linguistics, 10:680–696, 2022. 6
  43. Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814, 2017. 6
  44. Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017. 5
  45. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632, 2014. 6 find
  46. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 2, 7 *Table 3: Illustration of semantic, syntactic, and lexical e…
  47. Paris is the capital of France. Berlin is the capital of France. France’s capital is Paris. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuhn-2023-semantic-uncertainty-linguistic-invariances-uncertainty/