Elamur Research⌕
☰
home / papers / 2026 / Confidence Estimation for LLMs in Multi-turn Interactions

Confidence Estimation for LLMs in Multi-turn Interactions

Paper · 2026 · Caiqi Zhang, Ruihan Yang, Xiaochen Zhu et al.

Summary

Most work on LLM confidence estimation looks at single questions answered once. This paper studies a different case: multi-turn dialogue, where useful information arrives over several turns. The authors argue that a good confidence signal in this setting must do two things. It should be calibrated at each turn, and it should rise steadily as evidence builds up.

To measure these properties, they introduce two metrics:

  • InfoECE, an Expected Calibration Error that groups turns by normalized information level, so dialogues of different lengths can be compared.
  • Kendall's τ, applied to measure how consistently confidence increases over turns.

They build testbeds for two kinds of questions:

  • Underspecified questions, generated with a new "Hinter–Guesser" paradigm in which clues are revealed step by step.
  • Hard but fully specified questions, taken from existing incremental QA benchmarks.

Across four open-source models, standard methods (verbalized confidence, self-consistency, P(true)) are poorly calibrated and do not reliably increase over turns. The proposed logit-based probe P(SUFFICIENT) asks whether the available evidence is enough to answer. It performs better, but the problem remains largely open. This matters for agents that must decide when to ask a clarifying question and when to act.

Key points

  • Models and methods: The study evaluates Llama3.1-8B, Llama3.1-70B, Qwen2.5-7B and Qwen2.5-72B on the GUESS and 20Q datasets. Five confidence methods are compared: VANILLA-VERB, COT-VERB, SC, P(TRUE) and P(SUFFICIENT). Post-hoc rescaling such as temperature scaling is deliberately left out.
  • Placebo-hint test: Uninformative "placebo" hints were added to check whether confidence rises because of real information or simply because more turns have passed. P(SUFFICIENT) best separated genuine clues from filler. Verbalized confidence was unstable; for example, Qwen2.5-7B on GUESS dropped 9.29 points after an informative hint.
  • Ground truth versus provisional answers: Confidence increased more consistently when scored against the ground-truth answer than against the model's provisional answer at each turn.
  • Multi-turn dialogue versus single-turn summary: Accuracy barely differed between receiving clues over several turns and receiving them in one summary prompt (mean gap under 1 point). Confidence behaved very differently, however. P(SUFFICIENT) dropped sharply with summaries, for example from 63.13 to 13.23 for Qwen2.5-7B on 20Q.
  • Related work: A concurrent study looks at calibration under adversarial persuasion. This paper covers the complementary cooperative case, where confidence should increase as information accumulates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Fudan UniversityShanghai · China167 works · 20,987 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Caiqi Zhang, Ruihan Yang, Xiaochen Zhu, Chengzu Li, Tiancheng Hu, Yijiang River Dong, Deqing Yang and Nigel Collier. 2026. Confidence Estimation for LLMs in Multi-turn Interactions. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1280

@inproceedings{zhang2026confidence,
  title = {{Confidence Estimation for LLMs in Multi-turn Interactions}},
  author = {Caiqi Zhang, Ruihan Yang, Xiaochen Zhu, Chengzu Li, Tiancheng Hu, Yijiang River Dong, Deqing Yang and Nigel Collier},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1280},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1280}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Our research follows standard ethical guidelines. We verified the licenses of all software and datasets used in this study. We introduce new multi-turn evaluation datasets built from existing resources and automated generation. Despite best efforts, the data may contain Western-centric biases, and w… find
  2. Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. 2023. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. Preprint, arXiv:2311.18232. — [arXiv:2311.18232](https://arxiv.org/abs/2311.18232) find
  3. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Associatio…
  4. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can LLM be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126–10141, Miami, Florida, USA. Association for Computational Linguistics. find
  5. Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. BotChat: Evaluating LLMs’ capabilities of having multi-turn dialogues. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3184–3200, Mexico City, Mexico. Asso… find
  6. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  7. Tiancheng Hu, Benjamin Minixhofer, and Nigel Collier. 2025. Navigating the alignment-calibration trade-off: A pareto-superior frontier via model merging. arXiv preprint arXiv:2510.17426. find
  8. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  9. Michael Kirchhof, Gjergji Kasneci, and Enkelejda Kasneci. 2025. Position: Uncertainty quantification needs reassessment for large-language model agents. Preprint, arXiv:2505.22655. — [arXiv:2505.22655](https://arxiv.org/abs/2505.22655) find
  10. Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. MT-eval: A multi-turn capabilities evaluation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,…
  11. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open… find
  12. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. Llms get lost in multi-turn conversation. Preprint, arXiv:2505.06120.
  13. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. find
  14. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. Preprint, arXiv:2305.19187. find
  15. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. find
  16. Meta. 2024. Llama 3 model card. find
  17. Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, and Timothy Baldwin. 2025. A head to predict and a head to question: Pre-trained uncertainty quantification heads for hallucination d… find
  18. Yoo Yeon Sung, Eve Fleisig, Yu Hou, Ishan Upadhyay, and Jordan Lee Boyd-Graber. 2025. GRACE: A granular benchmark for evaluating model calibration against human calibration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1958… find
  19. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conferen… find
  20. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics, 7:387– 401.
  21. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. Preprint, arXiv:2309.10691.
  22. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Sen Yang, Nigel Collier, Dong Yu, and Deqing Yang. 2025a. LoGU: Long-form generation with uncertainty expressions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18947–1…
  23. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Dong Yu, Nigel Collier, and Deqing Yang. 2025b. UNCLE: Benchmarking uncertainty expressions in long-form generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30328–30344, Suzhou, China. As… find
  24. Zihao Yi, Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. 2024. A survey on recent advances in llm-based multi-turn dialogue systems. Preprint, arXiv:2402.18013.
  25. Boxuan Zhang and Ruqi Zhang. 2025. CoT-UQ: Improving response-wise uncertainty quantification in LLMs with chain-of-thought. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26114–26133, Vienna, Austria. Association for Computational Linguistics. find
  26. Caiqi Zhang, Zhijiang Guo, and Andreas Vlachos. 2024a. Do we need language-specific fact-checking models? the case of Chinese. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1899–1914, Miami, Florida, USA. Association for Computational Linguistics. find
  27. Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024b. LUQ: Long-text uncertainty quantification for LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA. Association for Computational Linguistics.
  28. Caiqi Zhang, Chang Shu, Ehsan Shareghi, and Nigel Collier. 2025a. All roads lead to Rome: Graph-based confidence estimation for large language model reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31802–31812, Suzhou, China. Association for… find
  29. Caiqi Zhang, Ruihan Yang, Zhisong Zhang, Xinting Huang, Sen Yang, Dong Yu, and Nigel Collier. 2024c. Atomic calibration of llms in long-form generations. find
  30. Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Nigel Collier, and Andreas Vlachos. 2025b. Reinforcement learning for better verbalized confidence in long-form generation. Preprint, arXiv:2505.23912. — [arXiv:2505.23912](https://arxiv.org/abs/2505.23912) find
  31. Zhaohan Zhang, Chengzhengxu Li, Xiaoming Liu, Chao Shen, Ziquan Liu, and Ioannis Patras. 2026. Confidence should be calibrated more than one turn deep. arXiv preprint arXiv:2604.05397.
  32. Zhaohan Zhang, Ziquan Liu, and Ioannis Patras. 2025c. Grace: A generative approach to better confidence elicitation in large language models. arXiv preprint arXiv:2509.09438. — [arXiv:2509.09438](https://arxiv.org/abs/2509.09438) find
  33. Ej Zhou, Caiqi Zhang, Tiancheng Hu, Chengzu Li, Nigel Collier, Ivan Vulić, and Anna Korhonen. 2025. Beyond the final layer: Intermediate representations for better multilingual calibration in large language models. Preprint, arXiv:2510.03136. — [arXiv:2510.03136](https://arxiv.org/abs/2510.03136) find
  34. Xiaochen Zhu, Caiqi Zhang, Tom Stafford, Nigel Collier, and Andreas Vlachos. 2025. Conformity in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3854–3872, Vienna, Austria. Association for Computational…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-confidence-estimation-llms-multi-turn/