Elamur Research⌕
☰
home / papers / 2026 / Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features

Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features

Paper · 2026 · Alberto Testoni and Iacer Calixto

Summary

Before LLMs can be trusted in clinical question answering, they need uncertainty estimates that are both well calibrated and good at separating right answers from wrong ones. Earlier evaluations reported only pooled scores, covered few and dated models, and seldom looked at clinical specialty or question type. The authors benchmark ten open-source LLMs (general-purpose, biomedical and reasoning-tuned), plus some proprietary ChatGPT models, on 1,810 English clinical multiple-choice questions spanning eleven specialties. Models answer in free form, and their outputs are then mapped onto the answer options. The study compares several uncertainty methods: token-probability scores, G-NLL, verbalized confidence elicitation, Semantic Entropy and conformal prediction. It also tests a new, cheap single-pass approach that builds features from the behaviour visible in reasoning traces. Semantic Entropy performs best but needs many generations, and most single-pass scores do poorly on both calibration and discrimination. Reasoning models often lead, but their advantage depends on specialty and question type. The reasoning-trace features come close to sampling-based quality. The main lesson is that pooled evaluation can hide reliability gaps, so model choice should fit the specific clinical setting.

Key points

  • Data: 870 questions from S-MedQA cover six common specialties at 145 questions each. Another 940 come from MedExQA and cover five underrepresented fields, such as Clinical Psychology and Speech Language Pathology.
  • Question types: six categories were labelled automatically with Llama-3.1-70B-Instruct. A manual check on 100 questions matched 94% of the time. The categories deal with identifying a condition (Diagnosis), choosing therapy (Treatment), selecting tests (Diagnostic Test), explaining terms (Definition), and carrying out procedures or operations (Procedure/Operation), with a catch-all Other group for the rest.
  • Models:
    • General-purpose: Llama-3.1/3.2/3.3 variants, Mistral-7B and Mixtral-8x7B.
    • Reasoning: DeepSeek-R1-70B and s1.1-32B.
    • Biomedical: Bio-Medical-Llama-3-8B and Llama3-OpenBioLLM-70B.
  • Semantic Entropy was the most robust method. G-NLL underperformed despite strong results reported outside the clinical domain, and no post-hoc calibration was applied to any method.
  • Code: released at github.com/albertotestoni/mind-the-gap-clinical-qa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksQuestion answering · 441 works

Cite

Cite as Alberto Testoni and Iacer Calixto. 2026. Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.106

@inproceedings{testoni2026mind,
  title = {{Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features}},
  author = {Alberto Testoni and Iacer Calixto},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.106},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.106}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emre Can Acikgoz, Osman Batur İnce, Rayene Bench, Arda Anıl Boz, İlker Kesen, Aykut Erdem, and Erkut Erdem. 2024. Hippocrates: An open-source framework for advancing large language models in healthcare. Preprint, arXiv:2404.16621. — [arXiv:2404.16621](https://arxiv.org/abs/2404.16621) find
  2. Felix J Dorfner, Amin Dada, Felix Busch, Marcus R Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Jacqueline Lammert, Lisa C Adams, and 1 others. 2024. Biomedical large languages models seem not to be superior to generalist models on unseen medical data. arXiv preprint arXiv:240… — [arXiv:2408.13833](https://arxiv.org/abs/2408.13833) find
  3. Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter. 2024. Rethinking uncertainty estimation in natural language generation. arXiv preprint arXiv:2412.15176. — [arXiv:2412.15176](https://arxiv.org/abs/2412.15176) find
  4. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  5. Malaikannan Sankarasubbu Ankit Pal. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/ OpenBioLLM-Llama3-70B. ↗
  6. Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy Miller, Danielle S Bitterman, Guanhua Chen, Anoop Mayampurath, Matthew M Churpek, and Majid Afshar. 2025. Uncertainty estimation in diagnosis generation from large language models: next-word probability is not pre-test probability. JAMIA… find
  7. Zahra Atf, Seyed Amir Ahmad Safavi-Naini, Peter R Lewis, Aref Mahjoubfar, Nariman Naderi, Thomas R Savage, and Ali Soroush. 2025. The challenge of uncertainty quantification of large language models in medicine. arXiv preprint arXiv:2504.05278. — [arXiv:2504.05278](https://arxiv.org/abs/2504.05278) find
  8. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703. — [arXiv:2307.15703](https://arxiv.org/abs/2307.15703) find
  9. Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3. find
  10. Tareen Dawood, Chen Chen, Baldeep S Sidhu, Bram Ruijsink, Justin Gould, Bradley Porter, Mark K Elliott, Vishal Mehta, Christopher A Rinaldi, Esther Puyol-Antón, and 1 others. 2023. Uncertainty aware training to improve deep learning model calibration for classification of cardiac mr images. Medical… find
  11. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  12. Biraja Ghoshal, Allan Tucker, Bal Sanghera, and Wai Lup Wong. 2021. Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection. Computational Intelligence, 37(2):701–734. find
  13. Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? evaluating uncertainty in neural text generators against human production variability. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14349–14371,… find
  14. Tobias Groot and Matias Valdenegro Toro. 2024. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pages 145–171, Mexico City, Mexico. Association for Com… find
  15. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. find
  16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  17. Wataru Hashimoto, Hidetaka Kamigaito, and Taro Watanabe. 2024. Are data augmentation methods in named entity recognition applicable for uncertainty estimation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18852–18867. find
  18. Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look before you leap: An exploratory study of uncertainty analysis for large language models. IEEE Transactions on Software Engineering. find
  19. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  20. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  21. Yusong Ke, Hongru Lin, Yuting Ruan, Junya Tang, and Li Li. 2025. Correctness coverage evaluation for medical multiple-choice question answering based on the enhanced conformal prediction framework. Mathematics, 13(9):1538. find
  22. Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Medexqa: Medical question answering benchmark with multiple explanations. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 167–181. find
  23. Benjamin Kompa, Jasper Snoek, and Andrew L Beam. 2021. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4(1):4. find
  24. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations. find
  25. Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107–6117.
  26. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… find
  27. Meta AI. 2024a. Llama 3 models. https://ai.meta.com/blog/meta-llama-3. ↗
  28. Meta AI. 2024b. Meta llama 3.1 70b instruct. https://huggingface.co/meta-llama/ Llama-3.1-70B-Instruct. ↗
  29. Mistral AI. 2023. Mistral-7b and mixtral-8x7b models. https://mistral.ai/news/ mixtral-of-experts. ↗
  30. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2…
  31. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR.
  32. Mauricio Rivera, Jean-François Godbout, Reihaneh Rabbany, and Kellin Pelrine. 2024. Combining confidence elicitation and sample-based methods for uncertainty quantification in misinformation mitigation. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 114–126, St… find
  33. Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. 2025. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics… find
  34. Rosanne Schoonbeek, Jessica Workum, Stephanie CE Schuit, Job Doornberg, Tom P Van Der Laan, and Charlotte MHHT Bootsma-Robroeks. 2024. Completeness, correctness and conciseness of physician-written versus large language model generated patient summaries integrated in electronic health records. Avail… find
  35. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, a… find
  36. Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. 2025. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys. find
  37. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  38. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930– 1940. find
  39. Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Oh. 2024. Calibrating large language models using their generations only. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15440–15459, Bangkok, Thailand. Associati… find
  40. Dennis Ulmer, Alexandra Lorson, Ivan Titov, and Christian Hardmeier. 2025. Anthropomimetic uncertainty: What verbalized uncertainty in language models is missing. arXiv preprint arXiv:2507.10587.
  41. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, and 1 others. 2024. Adapted large language models can outperform medical experts in clinical text summarization. Nature med…
  42. Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. 2024a. PMC-LLaMA: toward building open-source language models for medicine. J Am Med Inform Assoc, 31(9):1833–1843. find
  43. Jiaxin Wu, Yizhou Yu, and Hong-Yu Zhou. 2024b. Uncertainty estimation of large language models in medical question answering. arXiv preprint arXiv:2407.08662. — [arXiv:2407.08662](https://arxiv.org/abs/2407.08662) find
  44. Yijun Xiao and William Yang Wang. 2019. Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7322–7329. find
  45. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations. find
  46. Xinlan Yan, Di Wu, Yibin Lei, Christof Monz, and Iacer Calixto. 2026. What does neuro mean to cardio? investigating the role of clinical specialty data in medical llms. In The 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026). find
  47. Yongjin Yang, Haneul Yoo, and Hwaran Lee. 2025. Maqa: Evaluating uncertainty quantification in llms regarding data uncertainty. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5846–5863. find
  48. Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems, 37:15356–15385. find
  49. Chrysoula Zerva, Taisiya Glushkova, Ricardo Rei, and André FT Martins. 2022. Disentangling uncertainty in machine translation evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8622–8641. find
  50. Shuang Zhou, Jiashuo Wang, Zidu Xu, Song Wang, David Brauer, Lindsay Welton, Jacob Cogan, Yuen-Hei Chung, Lei Tian, Zaifu Zhan, and 1 others. 2025. Uncertainty-aware large language models for explainable disease diagnosis. npj Digital Medicine, 8(1):690. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/testoni-2026-mind-gap-benchmarking-llm-uncertainty/