Elamur Research⌕
☰
home / papers / 2026 / Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing

Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing

Paper · 2026 · Dennis Ulmer, Alexandra Lorson, Ivan Titov et al.

Summary

LLMs often deliver wrong answers in a confident tone, which erodes user trust. Ulmer, Lorson, Titov and Hardmeier argue that verbalized uncertainty in NLP has largely ignored how people actually signal and interpret doubt. They propose anthropomimetic uncertainty: machine uncertainty should partly imitate human linguistic behavior to be intuitive and trustworthy.

The paper does four things:

  • Reviews the linguistics literature. It covers epistemic markers, hedging and strengthening, the social motives behind them (politeness, power, accountability) and verbal versus numerical registers.
  • Surveys and annotates NLP work. Most studies rely on simple prompting and numeric confidence only.
  • Analyzes corpora. Frequency counts show that uncertainty marker usage depends strongly on text domain.
  • Probes five commercial LLMs. Their uncertainty expressions shift with context and domain and are poorly calibrated.

The authors then discuss machine-specific issues such as anthropomorphization and sycophancy. They outline research directions: new notions of calibration, personalization, multilingual resources, explaining uncertainty, and trust factors beyond confidence. The work matters because it gives a principled basis for designing uncertainty communication in chat interfaces.

Key points

  • Motivating example. In a 2025 Columbia Journalism Review test, ChatGPT was wrong 67% of the time and hedged in only 7.5% of replies. It never refused to answer.
  • Human vagueness. People map the word "possible" to probabilities between 17% and 89%, averaging about 42%.
  • LLM findings. In the five commercial LLMs tested, calibration errors reach up to 44%. Their use of strengtheners and weakeners ranges from ideal to anti-correlated with actual correctness.
  • Multilingual gap. Uncertainty-marker resources with confidence annotations are scarce for English and nearly absent for other languages, which the authors flag as a gap.
  • Publication details. The paper appeared in TACL vol. 14 (2026), pp. 1526–1561. Code is at the GitHub repo kaleidophon/anthropomimetic-uncertainty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓University of GroningenGroningen · Netherlands70 works · 5,142 citations🎓IT University of CopenhagenCopenhagen · Denmark27 works · 2,406 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Dennis Ulmer, Alexandra Lorson, Ivan Titov and Christian Hardmeier. 2026. Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.739

@article{ulmer2026anthropomimetic,
  title = {{Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing}},
  author = {Dennis Ulmer, Alexandra Lorson, Ivan Titov and Christian Hardmeier},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.739},
  url = {https://doi.org/10.1162/tacl.a.739}
}

References

243 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maria Barrett, Max Müller-Eberstein, Elisa Bassignana, Amalie Brogaard Pauli, Mike Zhang, and Rob van der Goot. 2024. Can Humans Identify Domains? In Proceedings Of The 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC/Coling 2024, 20-25 May, 2… find
  2. Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, and Verena Rieser. 2021. Alexa, Google, Siri: What Are Your Pronouns? Gender And Anthropomorphism In The Design And Perception Of Conversational Assistants. arXiv preprint arXiv:2106.02578. find
  3. Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. 2024. Faithfulness Vs. Plausibility: On The (Un)Reliability Of Explanations From Large Language Models. arXiv preprint arXiv:2402.04614. — [arXiv:2402.04614](https://arxiv.org/abs/2402.04614) find
  4. Anastasios N. Angelopoulos and Stephen Bates. 2023. Conformal Prediction: A Gentle Introduction. Foundations and Trends® in Machine Learning, 16(4):494–591. find
  5. Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf. ↗
  6. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty In Natural Language Generation: From Theory To Applications. arXiv preprint arXiv:2307.15703. — [arXiv:2307.15703](https://arxiv.org/abs/2307.15703) find
  7. Yavuz Faruk Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Avestimehr, and Sai Praneeth Karimireddy. 2025. Reconsidering LLM Uncertainty Estimation Methods in the Wild. In Proceedings Of The 63rd Annual Meeting Of The Association For Computational Linguistics (Volume 1… find
  8. Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. 2024. find
  9. Linguistic Calibration Of Long-Form Generations. In Forty-First International Conference On Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  10. Jeffrey Basoah, Daniel Chechelnitsky, Tao Long, Katharina Reinecke, Chrysoula Zerva, Kaitlyn Zhou, Mark Díaz, and Maarten Sap. 2025. find
  11. Not Like Us, Hunty: Measuring Perceptions And Behavioral Effects Of Minoritized Anthropomorphic Cues In LLMs. In Proceedings Of The 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2025, Athens, Greece, June 23-26, 2025, pages 710– 745. ACM. find
  12. Elisa Bassignana, Amanda Cercas Curry, and Dirk Hovy. 2025. The AI Gap: How Socioeconomic Status Affects Language Technology Interactions. In Proceedings Of The 63rd Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1…
  13. Andrea Beltrama, Stephanie Solt, and Heather Burnett. 2023. Context, Precision, And Social Perception: A Sociopragmatic Study. Language in Society, 52(5):805–835. find
  14. Brian Bergstrom, Bianca Moehlmann, and Pascal Boyer. 2006. Extending The Testimony Problem: Evaluating The Truth, Scope, And Source Of Cultural Information. Child development, 3(77):531–538. find
  15. Ruth Beyth-Marom. 1982. How Probable Is Probable? A Numerical Translation Of Verbal Probability Expressions. Journal of Forecasting, 1(3):257–269. find
  16. Jean-François Bonnefon and Gaëlle Villejoubert. 2006. Tactful Or Doubtful?: Expectations Of Politeness Explain The Severity Bias In The Interpretation Of Probability Phrases. Psychological Science, 17(9):747–751. find
  17. Mirko Borszukovszki, Ivo Pascal De Jong, and Matias Valdenegro-Toro. 2025. Know What You Do Not Know: Verbalized Uncertainty Estimation Robustness On Corrupted Images In Vision-Language Models. arXiv preprint arXiv:2504.03440. find
  18. Chris Brooks and Louis Williams. 2021. The Impact Of Personality Traits On Attitude To Financial Risk. Research in International Business and Finance, 58:101501. find
  19. Penelope Brown and Stephen Levinson. 1978. Universals In Language Usage: Politeness Phenomena. In E. Goody, editor, Questions And Politeness. Strategies In Social Interactions, pages 56–311. Cambridge University Press. ence on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelve… find
  20. Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2025. Magicore: Multi-Agent, Iterative, Coarse-To-Fine Refinement For Reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, Novembe…
  21. Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025. Social Sycophancy: A Broader Understanding Of LLM Sycophancy. arXiv preprint arXiv:2505.13995.
  22. Margarida Campos, António Farinhas, Chrysoula Zerva, Mário A.T. Figueiredo, and André F.T. Martins. 2024. Conformal Prediction For Natural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 12:1497–1516. find
  23. Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning From Human Preferences. Advances in neural information processing systems, 30. find
  24. María Victoria Carro. 2024. Flattering To Deceive: The Impact Of Sycophantic Behavior On User Trust In Large Language Model. arXiv preprint arXiv:2412.02802. — [arXiv:2412.02802](https://arxiv.org/abs/2412.02802) find
  25. Dominic A. Clark. 1990. Verbal Uncertainty Expressions: A Critical Review Of Two Decades Of Research. Current Psychology, 9(3):203– 235. find
  26. Tapabrata Chakraborti, Christopher R.S. Banerji, Ariane Marandon, Vicky Hellon, Robin Mitra, Brieuc Lehmann, Leandra Bräuninger, Sarah McGough, Cagatay Turkay, Alejandro F. Frangi, Ginestra Bianconi, Weizi Li, Owen Rackham, Deepak Parashar, Chris Harbron, and Ben MacArthur. 2025. Personalized Uncert… find
  27. Cohere Labs. 2024. C4AI-Command-R-Plus-08-2024 . https://huggingface.co/ CohereLabs/c4ai-command-r-plus-08-2024. ↗
  28. Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. 2024. Finetuning Language Models To Emit Linguistic Expressions Of Uncertainty. arXiv preprint arXiv:2409.12180. — [arXiv:2409.12180](https://arxiv.org/abs/2409.12180) find
  29. Hanjie Chen and Yangfeng Ji. 2022. Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Confer-Michelle Cohn, Mahima Pushkarna, Gbolahan O. Olanubi, Joseph M. Moran, Danie… find
  30. Clara Colombatto, Jonathan Birch, and Stephen Fleming. 2025. The Influence Of Mental State Attributions On Trust In Large Language Models. Communications Psychology, 3. find
  31. BNC Consortium et al. 2007. British National Corpus. Oxford Text Archive Core Collection. find
  32. Robots: Concept, Construction And Modelling. International Journal of Advanced Robotic Systems, 9(5):209. find
  33. Ajeya Cotra. 2021. Why AI Alignment Could Be Hard With Modern Deep Learning. Cold Takes. find
  34. Janet Cotterill. 2007. ‘I Think He Was Kind Of Shouting Or Something’: Uses And Abuses Of Vagueness In The British Courtroom. In Vague Language Explor Ed, pages 97–114. Springer. find
  35. Chris Cummins. 2015. Constraints On Numerical Expressions, volume 5. Oxford University Press. find
  36. Adam Dahlgren Lindström, Leila Methnani, Lea Krause, Petter Ericson, Íñigo Martínez de Rit-uerto de Troya, Dimitri Coelho Mollo, and Roel Dobbe. 2025. Helpful, Harmless, Honest? Sociotechnical Limits Of AI Alignment And Safety Through Reinforcement Learning From Human Feedback. Ethics and Informatio… find
  37. DeepSeek-AI. 2025. Deepseek-R1: Incentivizing Reasoning Capability In LLMs Via Reinforcement Learning. arXiv preprint arXiv:2501.12948.
  38. Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory Or Epistemic? Does It Matter? Structural safety, 31(2):105–112. find
  39. Alicia DeVrio, Myra Cheng, Lisa Egede, Alexandra Olteanu, and Su Lin Blodgett. 2025. find
  40. A Taxonomy Of Linguistic Expressions That Contribute To Anthropomorphism Of Language Technologies. In Proceedings Of The 2025 Chi Conference on Human Factors in Computing Systems, CHI 2025, Yokohama, Japan, 26 April 2025- 1 May 2025, pages 430:1–430:18. ACM. find
  41. Mandeep K. Dhami and David R. Mandel. 2022. Communicating Uncertainty Using Words And Numbers. Trends in Cognitive Sciences, 26(6):514–526. find
  42. Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. 2023. A Diachronic Perspective On User Trust In AI under Uncertainty. In Proceedings Of The 2023 Conference On Empirical Methods In Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5567–5580.… find
  43. Alan Diamond, Rob Knight, David Devereux, and Owen Holland. 2012. find
  44. Anthropomimetic Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. find
  45. Enhancing Chat Language Models By Scaling High-Quality Instructional Conversations. In Proceedings Of The 2023 Conference On Empirical Methods In Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3029–3051. Association for Computational Linguistics.
  46. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can LLM be a Personalized Judge? In Findings Of The Association For Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 10126–10141. Association for Computational Linguistics. find
  47. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting Attention To Relevance: Towards The Uncertainty Estimation Of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V… find
  48. Bryan Eikema, Evgenia Ilia, José G.C. de Souza, Chrysoula Zerva, and Wilker Aziz. 2025. Teaching Language Models To Faithfully Express Their Uncertainty. arXiv preprint arXiv:2510.12587. — [arXiv:2510.12587](https://arxiv.org/abs/2510.12587) find
  49. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2024. GPTs Are GPTs: Labor Market Impact Potential Of LLMs. Science, 384(6702):1306–1308. find
  50. Ido Erev and Brent L. Cohen. 1990. Verbal Versus Numerical Probabilities: Efficiency, Biases, And The Preference Paradox. Organizational behavior and human decision processes, 45(1):1–18. find
  51. European Commission. 2009. Guideline On The Readability Of The Labelling And Package Leaflet Of Medicinal Products For Human Use. https://health.ec.europa.eu/ document / download / d8612682 -ad17 - 40e3 - 8130 - 23395ec80380_en. Accessed: 11/07/2025. ↗
  52. Wader Fagen-Ulmschneider. 2019. Perception Of Probability Words. https://waf.cs.illinois.edu/visualizations/Perception - of -Probability - Words/. Accessed: 02/05/2025. ↗
  53. Constanza Fierro, Negar Foroutan, Desmond Elliott, and Anders Søgaard. 2025. How Do Multilingual Language Models Remember Facts. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 16052–16106. A… find
  54. Bruce Fraser. 1975. Hedged Performatives. In Speech Acts, pages 187–210. Brill. find
  55. Mirta Galesic and Rocio Garcia-Retamero. 2010. Statistical Numeracy For Health: A Cross-Cultural Comparison With Probabilistic National Samples. Archives of internal medicine, 170(5):462–468. find
  56. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset Of Diverse Text For Language Modeling. arXiv preprint arXiv:2101.00027. find
  57. Raymond W. Gibbs Jr. and Gregory A. Bryant. 2008. Striving For Optimal Relevance When Answering Questions. Cognition, 106(1):345– 369. find
  58. Irving John Good. 1971. 46656 Varieties Of Bayesians. American Statistician, 25(5):62. find
  59. Herbert Paul Grice. 1975. Logic And Conversation. Syntax and semantics, 3:43–58. find
  60. Tobias Groot and Matias Valdenegro-Toro. 2024. Overconfidence Is Key: Verbalized Uncertainty Evaluation In Large Language And Vision-Language Models. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pages 145–171. find
  61. Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauer-mann. 2023. Sources Of Uncertainty In Machine Learning–A Statisticians’ View. arXiv preprint arXiv:2305.16703. — [arXiv:2305.16703](https://arxiv.org/abs/2305.16703) find
  62. Vinayaka Gude. 2025. Factors Influencing Chat-GPT Adoption For Product Research And Information Retrieval. Journal of Computer Information Systems, 65(2):222–231. find
  63. Nuno Miguel Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations In Large Multilingual Translation Models. Trans. Assoc. Comput. Linguistics, 11:1500–1517.
  64. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration Of Modern Neural Networks. In International Conference On Machine Learning, pages 1321–1330. PMLR. find
  65. Sophia Hager, David Mueller, Kevin Duh, and Nicholas Andrews. 2025. Uncertainty Distillation: Teaching Language Models To Express Semantic Confidence. arXiv preprint arXiv:2503.14749.
  66. Adam J. L. Harris and Adam Corner. 2011. Communicating Environmental Risks: Clarifying The Severity Effect In Interpretations Of Verbal Probability Expressions. Journal of experimental psychology. Learning, memory, and cognition, 37 6:1571–8. find
  67. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  68. Juyeon Heo, Miao Xiong, Christina Heinze-Deml, and Jaya Narain. 2025. Do LLMs Estimate Uncertainty Well In Instruction-Following? In The Thirteenth International Conference On Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  69. Daniel Hershcovich, Stella Frank, Heather C. Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges And Strategies In Cross-Cultura…
  70. Arthur Hoarau, Sébastien Destercke, Yusuf Sale, Paul Hofman, and Eyke Hüllermeier. 2024. Measures Of Uncertainty: A Quantitative Analysis. https://arthur - hoarau.fr / files / Measures_of_Uncertainty.pdf. find
  71. Owen Holland and Rob Knight. 2006. The Anthropomimetic Principle. In Proceedings Of The Aisb06 Symposium On Biologically Inspired Robotics, pages 1–8. Citeseer. find
  72. Janet Holmes. 1982. Expressing Doubt And Certainty In English. RELC journal, 13(2):9–28. find
  73. Thomas Holtgraves. 2010. Social Psychology And Language: Words, Utterances, And Conversations. In Handbook Of Social Psychology. John Wiley & Sons, Inc. find
  74. Thomas Holtgraves. 2014. Interpreting Uncertainty Terms. Journal of personality and social psychology, 107(2):219. find
  75. Thomas Holtgraves and Audrey Perdew. 2016. Politeness And The Communication Of Uncertainty. Cognition, 154:1–10. find
  76. Richard Holton. 1994. Deciding To Trust, Coming To Believe. Australasian Journal of Philosophy, 72(1):63–76. find
  77. Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric And Epistemic Uncertainty In Machine Learning: An Introduction To Concepts And Methods. Machine learning, 110(3):457– 506. find
  78. Anders Humlum and Emilie Vestergaard. 2024. The Adoption Of ChatGPT. https://papers.ssrn.com/sol3/Delivery.cfm?abstractid=4807516. ↗
  79. Ken Hyland. 1999. Disciplinary Discourses: Writer Stance In Research Articles. find
  80. Ken Hyland. 2005. Stance And Engagement: A Model Of Interaction In Academic Discourse. Discourse Studies, 7(2):173–192. find
  81. Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. find
  82. Formalizing Trust In Artificial Intelligence: Prerequisites, Causes And Goals Of Human Trust In AI. In Proceedings Of The 2021 ACM Conference On Fairness, Accountability, And Transparency, pages 624–635. find
  83. Carel J.M. Jansen and Mathijs M.W. Pollmann. 2001. On Round Numbers: Pragmatic Aspects Of Numerical Expressions. Journal of quantitative linguistics, 8(3):187–201. find
  84. Klaudia Jaźwińska and Aisvarya Chandrasekar. 2025. AI Search Has A Citation Problem. https://www.cjr.org/tow_center/we - compared - eight - ai - search -engines - theyre - all - bad - at -citing-news.php. Accessed: 20/03/2025. ↗
  85. Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. 2025. Calibrating Verbal Uncertainty As A Linear Feature To Reduce Hallucinations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pro…
  86. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings Of The 55th Annual Meeting Of The Association For Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4…
  87. Marie Juanchich and Miroslav Sirota. 2015. A Direct And Comprehensive Test Of Two Postulates Of Politeness Theory Applied To Uncertainty Communication. Judgment and Decision Making, 10(3):232–240. find
  88. Marie Juanchich and Miroslav Sirota. 2020. Most Family Physicians Report Communicating The Risks Of Adverse Drug Reactions In Words (Vs. Numbers). Appl. Cogn. Psychol., (34):526—-534. find
  89. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… find
  90. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katie Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024. Large Language Models Must Be Taught To Know What They Don’t Know. In Advances In Neural Information Processing Systems 38: Annual Conferenc… find
  91. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models. In Proceedings Of The 16th Conference Of The European Chapter Of The Association For Computational Linguistics: Main Volume, Eacl 2021, Online, April 19 - 2…
  92. Maurice G. Kendall. 1938. A New Measure Of Rank Correlation. Biometrika, 30(1-2):81–93. find
  93. Christopher Kennedy. 2011. Ambiguity And Vagueness: An Overview. Semantics: An international handbook of natural language meaning, 1:507–535. find
  94. Sunnie S.Y. Kim, Qing Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. 2024. " I’m Not Sure, But...": Examining the Impact Of Large Language Models’ Uncertainty Expression On User Reliance And Trust. In Proceedings Of The 2024 ACM Conference On Fairness, Accountability… find
  95. Michael Kirchhof, Gjergji Kasneci, and Enkelejda Kasneci. 2025. Position: Uncertainty Quantification Needs Reassessment For Large Language Model Agents. In Forty-Second International Conference On Machine Learning Position Paper Track. find
  96. Lea Krause, Wondimagegnhue Tufa, Selene Báez Santamaría, Angel Daza, Urja Khurana, and Piek Vossen. 2023. Confidently Wrong: Exploring The Calibration And Expression Of (Un)Certainty Of Large Language Models In A Multilingual Setting. In Proceedings Of The Workshop On Multimodal, Multilingual Natura… find
  97. Manfred Krifka. 2007. Approximate Interpretation Of Number Words. Humboldt-Universität zu Berlin, Philosophische Fakultät II. find
  98. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. find
  99. Semantic Uncertainty: Linguistic Invariances For Uncertainty Estimation In Natural Language Generation. In The Eleventh International Conference On Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  100. Abhishek Kumar, Robert Morabito, Sanzhar Um-bet, Jad Kabbara, and Ali Emami. 2024. Confidence Under The Hood: An Investigation Into The Confidence-Probability Alignment In Large Language Models. In Proceedings Of The 62nd Annual Meeting Of The Association For Computational Linguistics (Volume 1: Lon… find
  101. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  102. Peter Lasersohn. 1999. Pragmatic Halos. Language, pages 522–551. find
  103. Daniel Lassiter. 2010. Gradable Epistemic Modals, Probability, And Scale Structure. In Semantics And Linguistic Theory, pages 197– 215. 7-11 November, 2021, pages 175–184. Association for Computational Linguistics. find
  104. Daniel Lassiter. 2017. Graded Modality: Qualitative And Quantitative Perspectives. Oxford University Press, Oxford. find
  105. Yibo Li, Miao Xiong, Jiaying Wu, and Bryan Hooi. 2025. Conftuner: Training Large Language Models To Express Their Confidence Verbally. arXiv preprint arXiv:2508.18847. — [arXiv:2508.18847](https://arxiv.org/abs/2508.18847) find
  106. Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2025a. Are LLM-Judges Robust To Expressions Of Uncertainty? Investigating The Effect Of Epistemic Markers On LLM-Based Evaluation. In Proceedings Of The 2025 Conference Of The Nations Of The Americas Chapter Of The Association Fo… find
  107. Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. 2025b. find
  108. The Impact Of Generative AI On Critical Thinking: Self-Reported Reductions In Cognitive Effort And Confidence Effects From A Survey Of Knowledge Workers. In Chi 2025. find
  109. Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. 2025. find
  110. Taming Overconfidence In LLMs: Reward Calibration In RLHF. In The Thirteenth International Conference On Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  111. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — Datasets: A Community Library for Natural Language Processing
  112. Chin-Yew Lin. 2004. Rouge: A Package For Automatic Evaluation Of Summaries. In Text Summarization Branches Out, pages 74–81.
  113. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. find
  114. Teaching Models To Express Their Uncertainty In Words. Trans. Mach. Learn. Res., 2022. find
  115. Yen-Liang Lin. 2013. Vague Language And Interpersonal Communication: An Analysis Of Adolescent Intercultural Conversation. International Journal of Society, Culture & Language, 1(2):69–81. find
  116. Dawn Liu, Marie Juanchich, Miroslav Sirota, and Sheina Orbell. 2020. The Intuitive Use Of Contextual Information In Decisions Made With Verbal And Numerical Quantifiers. Quarterly Journal of Experimental Psychology, 73(4):481–494. find
  117. Gabrielle Kaili-May Liu, Gal Yona, Avi Caciularu, Idan Szpektor, Tim G. J. Rudner, and Arman Cohan. 2025. Metafaith: Faithful Natural Language Uncertainty Expression In LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, Novembe…
  118. Jiayu Liu, Qing Zong, Weiqi Wang, and Yangqiu Song. Revisiting Epistemic Markers In Confidence Estimation: Can Markers Accurately Reflect Large Language Models’ Uncertainty? In Proceedings Of The 63rd Annual Meeting Of The Association For Computational Linguistics (Volume 2: Short Papers), ACL 2025,… — Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?
  119. Shudong Liu, Zhaocong Li, Xuebo Liu, Runzhe Zhan, Derek Wong, Lidia Chao, and Min Zhang. 2024. Can LLMs Learn Uncertainty On Their Own? Expressing Uncertainty Effectively In A Self-Training Manner. In Proceedings Of The 2024 Conference On Empirical Methods In Natural Language Processing, pages 21635… find
  120. Erik Løhre and Karl Halvor Teigen. 2016. There Is A 60% Probability, But I Am 70% Certain: Communicative Consequences Of External And Internal Expressions Of Uncertainty. Thinking & Reasoning, 22(4):369–396. find
  121. Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024a. Are Self-Explanations From Large Language Models Faithful? In Findings Of The Association For Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 295–337. Association for Computational Linguistic… find
  122. Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. 2024b. Interpretability Needs A New Paradigm. arXiv preprint arXiv:2405.05386. — [arXiv:2405.05386](https://arxiv.org/abs/2405.05386) find
  123. Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, and Maartje Ter Hoeve. 2025. On The Way To LLM Personalization: Learning To Remember User Conversations. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 61–77. find
  124. Michael McCarthy and Ronald Carter. 2006. Explorations In Corpus Linguistics. Cambridge University Press Cambridge. find
  125. Hugo Mercier and Dan Sperber. 2009. Intuitive And Reflective Inferences. In Jonathan Evans and Keith Frankish, editors, In Two Minds: Dual Processes And Beyond, pages 149–170. find
  126. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration. Trans. Assoc. Comput. Linguistics, 10:857–872. find
  127. Tim Miller. 2019. Explanation In Artificial Intelligence: Insights From The Social Sciences. Artificial intelligence, 267:1–38.
  128. Sandeep Mishra and Martin L. Lalumière. 2011. Individual Differences In Risk-Propensity: Associations Between Personality And Behavioral Measures Of Risk. Personality and Individual Differences, 50(6):869–873. find
  129. Mistral AI. 2025. Medium Is The New Large. . https://mistral.ai/news/mistral-medium-3. ↗
  130. Sarah Moss. 2015. On The Semantics And Pragmatics Of Epistemic Vocabulary. Semantics and pragmatics, 8:5–1.
  131. Frederick Mosteller and Cleo Youtz. 1990. Quantifying Probabilistic Expressions. Statistical science, pages 2–12. find
  132. Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. 2024. Benchmarking Uncertainty Disentanglement: Specialized Uncertainties For Specialized Tasks. Advances in Neural Information Processing Systems, 37:50972–51038. find
  133. Gijs Mulder, Gert-Jan Schoenmakers, and Helen De Hoop. 2022. The Influence Of First And Second Language On The Acquisition Of Pragmatic Markers In Spanish: Evidence From An Experimental Study. Isogloss, 8(1):1–23. find
  134. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In Proceedings Of The AAAI Conference On Artificial Intelligence, volume 29. find
  135. Manisha Natarajan and Matthew Gombolay. 2020. find
  136. Effects Of Anthropomorphism And Accountability On Trust In Human Robot Interaction. In Proceedings Of The 2020 ACM/IEEE International Conference On Human-Robot Interaction, HRI ’20, page 33–42, New York, NY, USA. Association for Computing Machinery. find
  137. Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024a. When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation. In Findings Of The Association For Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 1… find
  138. Shiyu Ni, Keping Bi, Lulu Yu, and Jiafeng Guo. 2024b. Are Large Language Models More Honest In Their Probabilistic Or Verbalized Confidence? In China Conference On Information Retrieval, pages 124–135. Springer. find
  139. Stephen Obadinma and Xiaodan Zhu. 2025. On The Robustness Of Verbal Confidence Of LLMs In Adversarial Attacks. arXiv preprint arXiv:2507.06489. — [arXiv:2507.06489](https://arxiv.org/abs/2507.06489) find
  140. Elinor Ochs. 1976. The Universality Of Conversational Postulates. Language in Society, 5(1):67–80. find
  141. OpenAI. 2024. GPT-4o System Card. preprint arXiv:2410.21276. arXiv Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training Language Models To Follow Instructions With Human Feedback. Advances…
  142. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra Of Human Behavior. In Proceedings Of The 36th Annual ACM Symposium On User Interface Software And Technology, pages 1–22. find
  143. Antonín Pavlíček, Aneta Bobenič Hintošová, and František Sudzina. 2021. Impact Of Personality Traits And Demographic Factors On Risk Attitude. SAGE Open, 11(4):21582440211066917. find
  144. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. find
  145. Discovering Language Model Behaviors With Model-Written Evaluations. In Findings Of The Association For Computational Linguistics: Acl 2023, pages 13387–13434.
  146. Jakub Podolak and Rajeev Verma. 2025. Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals In LLMs. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 247–258.
  147. James Pustejovsky. 2017. The Semantics Of Lexical Underspecification. Folia linguistica, 51(s1000):1–25. find
  148. Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. 2025. find
  149. Measuring What Makes You Unique: Difference-Aware User Modeling For Enhancing LLM Personalization. In Findings Of The Association For Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 21258–21277. Association for Computational Linguistics. find
  150. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model Is Secretly A Reward Model. Advances in Neural Information Processing Systems, 36:53728–53741.
  151. Neil Rathi, Dan Jurafsky, and Kaitlyn Zhou. 2025. Humans Overrely On Overconfident Language Models, Across Languages. arXiv preprint arXiv:2507.06306. find
  152. Ilaria Riccioni, Andrzej Zuczkowski, Roberto Burro, and Ramona Bongelli. 2022. The Italian Epistemic Marker Mi Sa [To Me It Knows] Compared To So [I Know], Non So [I Don’T Know], Non So Se [I Don’T Know Whether], Credo [I Believe], Penso [I Think]. PloS one, 17(9):e0274694. find
  153. Mauricio Rivera, Jean-François Godbout, Reihaneh Rabbany, and Kellin Pelrine. 2024. Combining Confidence Elicitation And Sample-Based Methods For Uncertainty Quantification In Misinformation Mitigation. In Proceedings Of The 1st Workshop On Uncertainty-Aware NLP (UncertaiNLP 2024), pages 114–126. find
  154. Anna Ruskan and Audronė Šolienė. 2017. Evidential And Epistemic Adverbials In Lithuanian: Evidence From Intra-Linguistic And Cross-Linguistic Analysis. Kalbotyra, 70:127– 152. find
  155. Paul A. Ruud, Daniel Schunk, and Joachim K. Winter. 2014. Uncertainty Causes Rounding: An Experimental Study. Experimental Economics, 17:391–413. find
  156. Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R. Narasimhan, and Vishvak Murahari. 2025. Per-sonagym: Evaluating Persona Agents And LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, N… find
  157. Sascha Schneider, Alexandra Häßler, Tanja Haber-meyer, Maik Beege, and Günter Daniel Rey. 2018. The More Human, The Higher The Performance? Examining The Effects Of Anthropomorphism On Learning With Media. Journal of Educational Psychology, 111. find
  158. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. find
  159. Glenn Shafer. 1976. A Mathematical Theory Of Evidence, volume 42. Princeton university press. find
  160. Glenn Shafer. 1978. Non-Additive Probabilities In The Work Of Bernoulli And Lambert. Archive for History of Exact sciences, 19:309– 370. find
  161. Matthew Shardlow and Piotr Przybyła. 2024. Deanthropomorphising NLP: Can A Language Model Be Conscious? PLoS One, 19(12):e0307521. find
  162. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. find
  163. Towards Understanding Sycophancy In Language Models. In The Twelfth International Conference On Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  164. Anthony Sicilia, Mert Inan, and Malihe Alikhani. 2025a. find
  165. Accounting For Sycophancy In Language Model Uncertainty Estimation. In Findings Of The Association For Computational Linguistics: Naacl 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 7851–7866. Association for Computational Linguistics. find
  166. Anthony Sicilia, Mert Inan, and Malihe Alikhani. 2025b. find
  167. Accounting For Sycophancy In Language Model Uncertainty Estimation. In Findings Of The Association For Computational Linguistics: Naacl 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 7851–7866. Association for Computational Linguistics. find
  168. Anthony Sicilia, Hyunwoo Kim, Khyathi Raghavi Chandu, Malihe Alikhani, and Jack Hessel. 2024. Deal, Or No Deal (Or Who Knows)? Forecasting Uncertainty In Conversations Using Large Language Models. In Findings Of The Association For Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual m… find
  169. Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. 2024a. Rethinking Interpretability In The Era Of Large Language Models. arXiv preprint arXiv:2402.01761.
  170. Shivalika Singh, Freddie Vargus, Daniel D’Souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün, Ifeom… — Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
  171. Dan Sperber. 2001. An Evolutionary Perspective On Testimony And Argumentation. Philosophical Topics, 29(1/2):401–413. find
  172. Dan Sperber, Fabrice Clément, Christophe Heintz, Olivier Mascaro, Hugo Mercier, Gloria Origgi, Torr, and Chang Xu. 2025b. Can Large Language Models Express Uncertainty Like Human? arXiv preprint arXiv:2509.24202. and Deirdre Wilson. 2010. Epistemic Vigilance. Mind and Language, 25(4):359–393. — [arXiv:2509.24202](https://arxiv.org/abs/2509.24202) find
  173. Dan Sperber and Deirdre Wilson. 1995. Relevance: Communication And Cognition, 2 edition. Blackwell Publishing. find
  174. Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. 2024. LACIE: Listener-aware fine-tuning for calibration in large language models. Advances in Neural Information Processing Systems, 37:43080–43106. find
  175. Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas W. Mayer, and Padhraic Smyth. 2025. What Large Language Models Know And What People Think They Know. Nature Machine Intelligence, pages 1–11. find
  176. Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. 2025. Large Language Models Are Overconfident And Amplify Human Bias. arXiv preprint arXiv:2505.02151. — [arXiv:2505.02151](https://arxiv.org/abs/2505.02151) find
  177. Yoo Yeon Sung, Eve Fleisig, Yu Hou, Ishan Upadhyay, and Jordan Lee Boyd-Graber. 2025. Grace: A Granular Benchmark For Evaluating Model Calibration Against Human Calibration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025,… find
  178. Eric Eric Peter Swanson. 2006. Interactions With Context. Ph.D. thesis, Massachusetts Institute of Technology. find
  179. Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Quantifying Uncertainty In Natural Language Explanations Of Large Language Models. In International Conference On Artificial Intelligence And Statistics, pages 1072–1080. PMLR. find
  180. Linwei Tao, Yi-Fan Yeh, Minjing Dong, Tao Huang, Philip Torr, and Chang Xu. 2025a. Revisiting Uncertainty Estimation And Calibration Of Large Language Models. arXiv preprint arXiv:2505.23854. — [arXiv:2505.23854](https://arxiv.org/abs/2505.23854) find
  181. Linwei Tao, Yi-Fan Yeh, Bo Kai, Minjing Dong, Tao Huang, Tom A. Lamb, Jialin Yu, Philip H.S. find
  182. Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. find
  183. When To Trust LLMs: Aligning Confidence With Response Quality. In Findings Of The Association For Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 5984–5996. Association for Computational Linguistics. find
  184. Gemini Team. 2023. Gemini: A Family Of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805.
  185. Gemini Team. 2024. Gemini 1.5: Unlocking Multimodal Understanding Across Millions Of Tokens Of Context. arXiv preprint arXiv:2403.05530.
  186. Karl Halvor Teigen. 2023. Dimensions Of Uncertainty Communication: What Is Conveyed By Verbal Terms And Numeric Ranges. Current Psychology, 42(33):29122–29137. find
  187. Teknium. 2023. OpenHermes 2.5: An Open Dataset Of Synthetic Data For Generalist LLM Assistants. https://huggingface.co/ datasets/teknium/OpenHermes-2.5. ↗
  188. Elizabeth R. Tenney, Robert J. MacCoun, Barbara A. Spellman, and Reid Hastie. 2007. Calibration Trumps Confidence As A Basis For Witness Credibility. Psychological Science, 18(1):46–50. find
  189. Elizabeth R. Tenney, Barbara A. Spellman, and Robert J. MacCoun. 2008. The Benefits Of Knowing What You Know (And What You Don’T): How Calibration Affects Credibility. Journal of Experimental Social Psychology, 44(5):1368–1375. find
  190. Ole Teutloff, Johanna Einsiedler, Otto Kässi, Fabian Braesemann, Pamela Mishkin, and R. Maria del Rio-Chanona. 2025. Winners And Losers Of Generative AI: Early Evidence Of Shifts In Freelancer Demand. Journal of Economic Behavior & Organization, page 106845. find
  191. Arthur Thuy and Dries F. Benoit. 2024. Explainability Through Uncertainty: Trustworthy Decision-Making With Neural Networks. find
  192. European Journal of Operational Research, 317(2):330–340. tive Principles Of Relevance. In Experimental Pragmatics, pages 141–171. Springer. find
  193. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. find
  194. Just Ask For Calibration: Strategies For Eliciting Calibrated Confidence Scores From Language Models Fine-Tuned With Human Feedback. In Proceedings Of The 2023 Conference On Empirical Methods In Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5433–5442. Association for… find
  195. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking Uncertainty Quantificat…
  196. Bob van Tiel, Uli Sauerland, and Michael Franke. 2022. Meaning And Use In The Expression Of Estimative Probability. Open Mind, 6:250–263. find
  197. Hannah Vogel, Sebastian Appelbaum, Heidemarie Haller, and Thomas Ostermann. 2022. The Interpretation Of Verbal Probabilities: A Systematic Literature Review And Meta-Analysis. German Medical Data Sciences 2022–Future Medicine: More Precise, More Integrative, More Sustainable!, pages 9–16. find
  198. Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. 2024. Uncertainty-Based Abstention In LLMs Improves Safety And Reduces Hallucinations. arXiv preprint arXiv:2404.10960. — [arXiv:2404.10960](https://arxiv.org/abs/2404.10960) find
  199. Indrit Troshani, Sally Rao Hill, Claire Sherman, and Damien Arthur and. 2021. Do We Trust In AI? Role Of Anthropomorphism And Intelligence. Journal of Computer Information Systems, 61(5):481–491. find
  200. UK Ministry of Defence. 2023. Defence Intelligence – Communicating Probability. https://www.gov.uk/government/news/defence - intelligence -communicating - probability. Accessed: 11/07/2025. ↗
  201. Dennis Ulmer. 2024. On Uncertainty In Natural Language Processing. Ph.D. thesis, IT-Universitetet i København. find
  202. Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. 2024. Calibrating Large Language Models Using Their Generations Only. In Proceedings Of The 62nd Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16,… find
  203. Jean-Baptiste Van Der Henst and Dan Sperber. 2004. find
  204. Testing The Cognitive And Communica-Eva Thue Vold. 2006. Epistemic Modality Markers In Research Articles: A Cross-Linguistic And Cross-Disciplinary Study. International journal of applied linguistics, 16(1):61–87. find
  205. Colin Vullioud, Fabrice Clément, Thom Scott-Phillips, and Hugo Mercier. 2017. Confidence As An Expression Of Commitment: Why Misplaced Expressions Of Confidence Backfire. Evolution and Human Behavior, 38(1):9–17. find
  206. Thomas S. Wallsten, David V. Budescu, Rami Zwick, and Steven M. Kemp. 1993. Preferences And Reasons For Communicating Probabilistic Information In Verbal Or Numerical Terms. Bulletin of the Psychonomic Society, 31(2):135–138. find
  207. Thomas S. Wallsten, Samuel Fillenbaum, and James A. Cox. 1986. Base Rate Effects On The Interpretations Of Probability And Frequency Expressions. Journal of Memory and Language, 25(5):571–587. find
  208. Cheng Wang, Gyuri Szarvas, Georges Balazs, Pavel Danchenko, and Patrick Ernst. 2024a. Calibrating Verbalized Probabilities For Large Language Models. arXiv preprint arXiv:2410.06707. — [arXiv:2410.06707](https://arxiv.org/abs/2410.06707) find
  209. Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024b. Understanding User Experience In Large Language Model Interactions. arXiv preprint arXiv:2401.08329. — [arXiv:2401.08329](https://arxiv.org/abs/2401.08329) find
  210. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. find
  211. Self-Consistency Improves Chain Of Thought Reasoning In Language Models. In The Eleventh International Conference On Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  212. David Watson, Joshua O’Hara, Niek Tax, Richard Mudd, and Ido Guy. 2023. Explaining Predictive Uncertainty With Information Theoretic Shapley Values. Advances in Neural Information Processing Systems, 36:7330–7350. find
  213. Elke U. Weber and Denis J. Hilton. 1990. Contextual Effects In The Interpretations Of Probability Words: Perceived Base Rate And Severity Of Events. Journal of Experimental Psychology: Human Perception and Performance, 16(4):781–789. find
  214. Caroline J. Wesson and Briony D. Pulford. 2009. Verbal Expressions Of Confidence And Doubt. Psychological Reports, 105(1):151–160. find
  215. Robin Willink and Rod White. 2012. Disentangling Classical And Bayesian Approaches To Uncertainty Analysis. New Zeland: Measurement Standards Laboratory. find
  216. Paul D. Windschitl and Gary L. Wells. 1996. Measuring Psychological Uncertainty: Verbal Versus Numeric Methods. Journal of Experimental Psychology: Applied, 2(4):343. find
  217. Greg Woodin, Bodo Winter, Jeannette Littlemore, Marcus Perlman, and Jack Grieve. 2024. Large-Scale Patterns Of Number Use In Spoken And Written English. Corpus Linguistics and Linguistic Theory, 20(1):123–152. find
  218. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation Of Confidence Elicitation In LLMs. In The Twelfth International Conference On Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open… find
  219. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024. find
  220. SaySelf: Teaching LLMs To Express Confidence With Self-Reflective Rationales. In Proceedings Of The 2024 Conference On Empirical Methods In Natural Language Processing, pages 5985–5998. find
  221. Weihao Xuan, Qingcheng Zeng, Heli Qi, Junjue Wang, and Naoto Yokoya. 2025. Seeing Is Believing, But How Much? A Comprehensive Analysis Of Verbalized Calibration In Vision-Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, C…
  222. Seth Yalcin. 2007. Epistemic Modals. 116(464):983–1026. find
  223. Mind, Seth Yalcin. 2010. Probability Operators. Philosophy Compass, 11(5):916–937. find
  224. Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024a. On Verbalized Confidence Scores For LLMs. arXiv preprint arXiv:2412.14737. — [arXiv:2412.14737](https://arxiv.org/abs/2412.14737) find
  225. Haoyan Yang, Yixuan Wang, Xingyin Xu, Hanyuan Zhang, and Yirong Bian. 2024b. Can We Trust LLMs? Mitigate Overconfidence Bias In LLMs Through Knowledge Transfer. arXiv preprint arXiv:2405.16856. — [arXiv:2405.16856](https://arxiv.org/abs/2405.16856) find
  226. Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2024c. Alignment For Honesty. Advances in Neural Information Processing Systems, 37:63565–63598. find
  227. Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2024. Cognitive Mirage: A Review of Hallucinations in Large Language Models. In Proceedings Of The First International Openkg Workshop: Large Knowledge-Enhanced Models Co-Locacted With The International Joint Conference On Artificial Inte… find
  228. Luke Yoffe, Alfonso Amayuelas, and William Yang Wang. 2024. DebUnc: Mitigating Hallucinations In Large Language Model Agent Communication With Uncertainty Estimations. arXiv preprint arXiv:2407.06426. 2979. Association for Computational Linguistics. — [arXiv:2407.06426](https://arxiv.org/abs/2407.06426) find
  229. Gal Yona, Roee Aharoni, and Mor Geva. 2024. Can Large Language Models Faithfully Express Their Intrinsic Uncertainty In Words? In Proceedings Of The 2024 Conference On Empirical Methods In Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 7752–7764. Association for…
  230. Lili Zhao, Yang Wang, Qi Liu, Mengyun Wang, Wei Chen, Zhichao Sheng, and Shijin Wang. 2025a. Evaluating Large Language Models Through Role-Guide And Self-Reflection: A Comparative Study. In The Thirteenth International Conference On Learning Representations. find
  231. Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, and Minjoon Seo. 2025. Reasoning Models Better Express Their Confidence. arXiv preprint arXiv:2505.14489. — [arXiv:2505.14489](https://arxiv.org/abs/2505.14489) find
  232. George Udny Yule. 1912. On The Methods Of Measuring Association Between Two Attributes. Journal of the Royal Statistical Society, 75(6):579–652. find
  233. Qingcheng Zeng, Weihao Xuan, Leyang Cui, and Rob Voigt. 2025. Thinking Out Loud: Do Reasoning Models Know When They’re Right? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 1394–1407. Association for Com…
  234. Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier. 2025. Atomic Calibration Of LLMs In Long-Form Generations. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the… find
  235. Mozhi Zhang, Mianqiu Huang, Rundong Shi, Lin-sen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu. 2024. find
  236. Calibrating The Confidence Of Large Language Models By Eliciting Fidelity. In Proceedings Of The 2024 Conference On Empirical Methods In Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 2959– Yunpu Zhao, Rui Zhang, Junbin Xiao, Ruibo Hou, Jiaming Guo, Zihao Zhang,… find
  237. Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Nouha Dziri, Dan Jurafsky, and Maarten Sap. 2025. REL-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human La… find
  238. Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. 2024. Relying On The Unreliable: The Impact Of Language Models’ Reluctance To Express Uncertainty. In Proceedings Of The 62Nd Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand,… find
  239. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. find
  240. Navigating The Grey Area: How Expressions Of Uncertainty And Overconfidence Affect Language Models. In Proceedings Of The 2023 Conference On Empirical Methods In Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5506–5524. Association for Computational Linguistics. find
  241. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. find
  242. Fine-Tuning Language Models From Human Preferences. arXiv preprint arXiv:1909.08593. find
  243. Brian J. Zikmund-Fisher, Dylan M. Smith, Peter A. Ubel, and Angela Fagerlin. 2007. Validation Of The Subjective Numeracy Scale: Effects Of Low Numeracy On Comprehension Of Risk Communications And Utility Elicitations. Medical Decision Making, 27(5):663–671. *Figure 7: Distributions of the most frequ… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ulmer-2026-anthropomimetic-uncertainty-what-verbalized-uncertainty/