Elamur Research⌕
☰
home / papers / 2026 / ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making

ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making

Paper · 2026 · Yusuke Watanabe, Yohei Kobashi, Takeshi Kojima et al.

Summary

Clinicians often have to decide with partial data, so a core skill is recognizing whether the available facts already settle a question. Existing medical and abstention benchmarks mostly assume complete inputs or treat missing data as a reason to abstain. They therefore cannot show whether a model knows when a judgment is actually possible. The authors build ClinDet-Bench from 16 guideline-backed clinical scoring systems, such as CHADS2. Synthetic cases are labeled by comparing the lowest and highest attainable total score with the decision threshold. Each model is tested only on systems it explains correctly, which separates reasoning failures from missing knowledge. Recent LLMs do well when all inputs are given. With missing inputs, however, they both jump to unsupported conclusions and refuse to answer when an answer can be derived. Error analysis shows that models fill gaps with plausible values, for example treating unmentioned findings as absent, instead of checking every possibility. The authors conclude that complete-information benchmarks overstate the clinical safety of LLMs.

Key points

  • Benchmark design: ClinDet-Bench has two tasks.
    • An Explanation Task checks knowledge of each scoring system using a one-shot prompt.
    • A Clinical Decision Task asks for 'Met', 'Not met', or 'Unable to determine'.
  • Case labels: Cases are Complete (S_min = S_max), Incomplete-Determinable (S_min ≥ T or S_max < T), or Incomplete-Undeterminable (S_min < T ≤ S_max). Each scoring system has about 12 cases, two per scenario–label combination.
  • Settings tested: Three prompting setups were evaluated: Base, Chain-of-Thought, and a "Safe" prompt that encourages abstention when uncertain. Post-hoc self-evaluation was also tested, and temperature was fixed at 1.0.
  • Main findings:
    • Models more often reached a verdict too early than declined to answer when they didn't need to.
    • Across models and prompts, reducing one error type increased the other.
    • Few-shot prompts with worked reasoning improved accuracy, but the authors consider this a non-scalable workaround.
  • Availability: The data is fully synthetic, contains no patient records, and is released under the MIT License on GitHub (yusukewatanabe1208/ClinDet_Benchmark).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

Cite

Cite as Yusuke Watanabe, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yasushi Okuno and Yutaka Matsuo. 2026. ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.47

@inproceedings{watanabe2026clindet,
  title = {{ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making}},
  author = {Yusuke Watanabe, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yasushi Okuno and Yutaka Matsuo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.47},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.47}
}

References

68 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R. Goodwin, Sonya E. Shooshan, and Dina Demner-Fushman. 2019. Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers. In MEDINFO 2019: Health and Wellbeing e-Networks for All, pages 25–29. IOS Press. find
  2. A. Alvarado. 1986. A practical score for the early diagnosis of acute appendicitis. Annals of Emergency Medicine, 15(5):557–564. find
  3. V. Apgar. 1953. A proposal for a new method of evaluation of the newborn infant. Current Researches in Anesthesia & Analgesia, 32(4):260–267. find
  4. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". arXiv preprint. ArXiv:2309.12288 [cs]. find
  5. E. H. Bishop. 1964. PELVIC SCORING FOR ELEC-TIVE INDUCTION. Obstetrics and Gynecology, 24:266–268. find
  6. O. Blatchford, W. R. Murray, and M. Blatchford. 2000. A risk score to predict need for treatment for upper-gastrointestinal haemorrhage. Lancet (London, England), 356(9238):1318–1321. find
  7. R. C. Bone, R. A. Balk, F. B. Cerra, R. P. Dellinger, A. M. Fein, W. A. Knaus, R. M. Schein, and W. J. Sibbald. 1992. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physici… find
  8. R. M. Centor, J. M. Witherspoon, H. P. Dalton, C. E. Brody, and K. Link. 1981. The diagnosis of strep throat in adults in the emergency room. Medical Decision Making: An International Journal of the Society for Medical Decision Making, 1(3):239–246. find
  9. Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. 2024. Can AI Assistants Know What They Don’t Know? arXiv preprint. ArXiv:2401.13275 [cs]. — [arXiv:2401.13275](https://arxiv.org/abs/2401.13275) find
  10. Pat Croskerry. 2003. The importance of cognitive errors in diagnosis and strategies to minimize them. Academic Medicine: Journal of the Association of American Medical Colleges, 78(8):775–780. find
  11. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
  12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LL…
  13. Salomone Di Saverio, Mauro Podda, Belinda De Simone, Marco Ceresoli, Goran Augustin, Alice Gori, Marja Boermeester, Massimo Sartelli, Federico Coc-colini, Antonio Tarasconi, Nicola De’ Angelis, Dieter G. Weber, Matti Tolonen, Arianna Birindelli, Walter Biffl, Ernest E. Moore, Michael Kelly, Kjetil S… find
  14. Moritoki Egi, Hiroshi Ogura, Tomoaki Yatabe, Kazuaki Atagi, Shigeaki Inoue, Toshiaki Iba, Yasuyuki Kakihana, Tatsuya Kawasaki, Shigeki Kushimoto, Yasuhiro Kuroda, Joji Kotani, Nobuaki Shime, Takumi Taniguchi, Ryosuke Tsuruta, Kent Doi, Matsuyuki Doi, Taka-Aki Nakada, Masaki Nakane, Seitaro Fujishima… find
  15. European Association for the Study of the Liver. 2018. EASL Clinical Practice Guidelines for the management of patients with decompensated cirrhosis. Journal of Hepatology, 69(2):406–460. find
  16. Laura Evans, Andrew Rhodes, Waleed Alhazzani, Massimo Antonelli, Craig M. Coopersmith, Craig French, Flávia R. Machado, Lauralyn Mcintyre, Marlies Ostermann, Hallie C. Prescott, Christa Schorr, Steven Simpson, W. Joost Wiersinga, Fayez Alshamsi, Derek C. Angus, Yaseen Arabi, Luciano Azevedo, Richard… find
  17. B. F. Gage, A. D. Waterman, W. Shannon, M. Boechler, M. W. Rich, and M. J. Radford. 2001. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA, 285(22):2864–2870. find
  18. Mark L. Graber, Nancy Franklin, and Ruthanna Gordon. 2005. Diagnostic Error in Internal Medicine. Archives of Internal Medicine, 165(13):1493–1499. find
  19. Gerhard Hindricks, Tatjana Potpara, Nikolaos Dagres, Elena Arbelo, Jeroen J. Bax, Carina Blomström-Lundqvist, Giuseppe Boriani, Manuel Castella, Gheorghe-Andrei Dan, Polychronis E. Dilaveris, Laurent Fauchier, Gerasimos Filippatos, Jonathan M. Kalman, Mark La Meir, Deirdre A. Lane, Jean-Pierre Lebea… find
  20. Alok A. Khorana, Nicole M. Kuderer, Eva Culakova, Gary H. Lyman, and Charles W. Francis. 2008. Development and validation of a predictive model for chemotherapy-associated thrombosis. Blood, 111(10):4902–4907. find
  21. Jerry R. Hobbs, Mark E. Stickel, Douglas E. Appelt, and Paul Martin. 1993. Interpretation as abduction. Artificial Intelligence, 63(1):69–142. find
  22. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv preprint. ArXiv:2205.11916 [cs].
  23. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. arXiv preprint. ArXiv:2311.08718 [cs]. — [arXiv:2311.08718](https://arxiv.org/abs/2311.08718) find
  24. C. W. Imrie, I. S. Benjamin, J. C. Ferguson, A. J. McKay, I. Mackenzie, J. O’Neill, and L. H. Blumgart. 1978. A single-centre double-blind trial of Trasylol therapy in primary acute pancreatitis. The British Journal of Surgery, 65(5):337–341. find
  25. Renata Iskander, Hannah Moyer, Karine Vigneault, Salaheddin M. Mahmud, and Jonathan Kimmelman. 2024. Survival Benefit Associated With Participation in Clinical Trials of Anticancer Drugs: A Systematic Review and Meta-Analysis. JAMA, 331(24):2105– 2113. find
  26. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv preprint. ArXiv:2009.13081 [cs].
  27. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. arXiv preprint. ArXiv:1909.06146 [cs].
  28. A. C. KAKAS, R. A. KOWALSKI, and F. TONI. 1992. Abductive Logic Programming. Journal of Logic and Computation, 2(6):719–770. find
  29. Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations.
  30. Nigel S. Key, Alok A. Khorana, Nicole M. Kuderer, Kari Bohlke, Agnes Y. Y. Lee, Juan I. Arcelus, Sandra L. Wong, Edward P. Balaban, Christopher R. Flowers, Leigh E. Gates, Ajay K. Kakkar, Margaret A. Tempero, Shilpi Gupta, Gary H. Lyman, and Anna Falanga. 2023. Venous Thromboembolism Prophylaxis and… find
  31. Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. 2025. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv preprint. ArXiv:2506.09038 [cs]. find
  32. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  33. Loren Laine, Alan N. Barkun, John R. Saltzman, Myriam Martel, and Grigorios I. Leontiadis. 2021. ACG Clinical Guideline: Upper Gastrointestinal and Ulcer Bleeding. The American Journal of Gastroenterology, 116(5):899–917. find
  34. W. S. Lim, M. M. van der Eerden, R. Laing, W. G. Boersma, N. Karalus, G. I. Town, S. A. Lewis, and J. T. Macfarlane. 2003. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax, 58(5):377–382. find
  35. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. arXiv preprint. ArXiv:2205.14334 [cs]. find
  36. Sravanthi Machcha, Sushrita Yerra, Sahil Gupta, Aishwarya Sahoo, Sharmin Sultana, Hong Yu, and Zonghai Yao. 2026. Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty. arXiv preprint. ArXiv:2601.12471 [cs].
  37. Sravanthi Machcha, Sushrita Yerra, Sharmin Sultana, Hong Yu, and Zonghai Yao. 2025. Do Large Language Models Know When Not to Answer in Medical QA? In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 27–35, Suzhou, China. Association for Computational Linguistics. find
  38. Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. 2025. Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9329–9345, Abu Dhabi, UAE. A…
  39. Marina Mancoridis, Bec Weeks, Keyon Vafa, and Sendhil Mullainathan. 2025. Potemkin Understanding in Large Language Models. arXiv preprint. ArXiv:2506.21521 [cs]. — [arXiv:2506.21521](https://arxiv.org/abs/2506.21521) find
  40. Liam G. McCoy, Rajiv Swamy, Nidhish Sagar, Minjia Wang, Stephen Bacchi, Jie Ming Nigel Fong, Nigel C.K. Tan, Kevin Tan, Thomas A. Buckley, Peter Brodeur, Leo Anthony Celi, Arjun K. Manrai, Aloysius Humbert, and Adam Rodman. 2025. Assessment of Large Language Models in Clinical Reasoning: A Novel Ben… find
  41. Joshua P. Metlay, Grant W. Waterer, Ann C. Long, Antonio Anzueto, Jan Brozek, Kristina Crothers, Laura A. Cooley, Nathan C. Dean, Michael J. Fine, Scott A. Flanders, Marie R. Griffin, Mark L. Me-tersky, Daniel M. Musher, Marcos I. Restrepo, and Cynthia G. Whitney. 2019. Diagnosis and Treatment of Ad… find
  42. Naoyuki Miyashita, Toshiharu Matsushima, Mikio Oka, and null Japanese Respiratory Society. 2006. The JRS guidelines for the management of community-acquired pneumonia in adults: an update and new recommendations. Internal Medicine (Tokyo, Japan), 45(7):419–428. find
  43. Hiroshi Mukae, Naoki Iwanaga, Nobuyuki Horita, Kosaku Komiya, Takaya Maruyama, Yuichiro Shindo, Yoshifumi Imamura, Kazuhiro Yatera, Yoshihiro Yamamoto, Katsunori Yanagihara, Nobuaki Shime, Kazuyoshi Senda, Hiroshi Takahashi, Futoshi Higa, Tetsuya Matsumoto, Makoto Miki, Shinji Teramoto, Hiroki Tsuka… find
  44. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  45. Richard E. Nisbett and Timothy D. Wilson. 1977. The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35(4):250–256. find
  46. Emily C. O’Brien, DaJuanicia N. Simon, Laine E. Thomas, Elaine M. Hylek, Bernard J. Gersh, Jack E. find
  47. Ansell, Peter R. Kowey, Kenneth W. Mahaffey, Paul Chang, Gregg C. Fonarow, Michael J. Pencina, Jonathan P. Piccini, and Eric D. Peterson. 2015. The ORBIT bleeding score: a simple bedside score to assess bleeding risk in atrial fibrillation. European Heart Journal, 36(46):3258–3264. find
  48. Committee on Fetus and Newborn. 2015. The Apgar Score. Obstetrics and Gynecology, 126(4):e52–e55. find
  49. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-HALT: Medical Domain Hallucination Test for Large Language Models. arXiv preprint. ArXiv:2307.15343 [cs].
  50. Stephen G. Pauker and Jerome P. Kassirer. 1980. The Threshold Approach to Clinical Decision Making. New England Journal of Medicine, 302(20):1109– 1117. find
  51. Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked. arXiv preprint. ArXiv:2308.07308 [cs]. find
  52. Ron Pisters, Deirdre A. Lane, Robby Nieuwlaat, Cees B. de Vos, Harry J. G. M. Crijns, and Gregory Y. H. Lip. 2010. A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey. Chest, 138(5):1093–1100. find
  53. R. N. Pugh, I. M. Murray-Lyon, J. L. Dawson, M. C. Pietroni, and R. Williams. 1973. Transection of the oesophagus for bleeding oesophageal varices. The British Journal of Surgery, 60(8):646–649. find
  54. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. arXiv preprint. ArXiv:1806.03822 [cs].
  55. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, and 48 others. 202… find
  56. Stanford T. Shulman, Alan L. Bisno, Herbert W. Clegg, Michael A. Gerber, Edward L. Kaplan, Grace Lee, Judith M. Martin, Chris Van Beneden, and Infectious Diseases Society of America. 2012. Clinical practice guideline for the diagnosis and management of group A streptococcal pharyngitis: 2012 update… find
  57. Mervyn Singer, Clifford S. Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R. find
  58. Bernard, Jean-Daniel Chiche, Craig M. Coopersmith, Richard S. Hotchkiss, Mitchell M. Levy, John C. Marshall, Greg S. Martin, Steven M. Opal, Gordon D. Rubenfeld, Tom van der Poll, Jean-Louis Vincent, and Derek C. Angus. 2016. The Third International Consensus Definitions for Sepsis and Septic Shock… find
  59. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others… find
  60. Wenting Zhao, Justin T. Chiu, Jena Hwang, Faeze Brahman, Jack Hessel, Sanjiban Choudhury, Yejin Choi, Xiang Lorraine Li, and Alane Suhr. 2024a. UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations. In Proceedings of the 2024 Conference of the North American Chapter of the Associati… find
  61. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024b. WildChat: 1M ChatGPT Interaction Logs in the Wild. arXiv preprint. ArXiv:2405.01470 [cs].
  62. Bonpei Takase, Takanori Ikeda, Wataru Shimizu, Haruhiko Abe, Takeshi Aiba, Masaomi Chinushi, Shinji Koba, Kengo Kusano, Shinichi Niwano, Naohiko Takahashi, Seiji Takatsuki, Kaoru Tanno, Eiichi Watanabe, Koichiro Yoshioka, Mari Amino, Tadashi Fujino, Yu-Ki Iwasaki, Ritsuko Kohno, Toshio Kinoshita, an… find
  63. Scott Tenner, John Baillie, John DeWitt, Santhi Swaroop Vege, and American College of Gastroenterology. 2013. American College of Gastroenterology guideline: management of acute pancreatitis. The American Journal of Gastroenterology, 108(9):1400– 1415; 1416. find
  64. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. arXiv preprint. ArXiv:2212.10509 [cs].
  65. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint. ArXiv:2201.11903 [cs].
  66. Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2025. Know Your Limits: A Survey of Abstention in Large Language Models. Transactions of the Association for Computational Linguistics, 13:529–556. find
  67. B. U. Wu, R. S. Johannes, X. Sun, Y. Tabak, D. L. Conwell, and P. A. Banks. 2008. The early prediction of mortality in acute pancreatitis: a large population-based study. Gut, 57(12):1698–1703. find
  68. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2024. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. arXiv preprint. ArXiv:2306.09841 [cs].

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/watanabe-2026-clindet-bench-beyond-abstention-evaluating/