Elamur Research⌕
☰
home / papers / 2026 / Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study

Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study

Paper · 2026 · Angelo Ziletti and Leonardo D'Ambrosi

Summary

When an LLM translates clinical questions into SQL over EHR or claims databases, varied outputs can mean two different things. The question itself may admit several readings, and the user should be asked to clarify. Or the model may be unreliable even for one fixed reading, and a human should check the result. Ziletti and D'Ambrosi (Bayer AG) introduce CLUES, which separates these two cases.

CLUES treats generation as a two-step pipeline. First, the system produces several natural-language interpretations of the question. Then, for each interpretation, it generates, executes and verbalizes several SQL answers. Interpretations and answers become nodes of a bipartite graph, with edge weights from LLM-judged semantic similarity. An ambiguity score is computed from the interpretation block. An instability score is computed from the Schur complement, which removes the part of answer similarity that the interpretations already explain.

CLUES predicts failures better than Kernel Language Entropy (KLE) on AmbigQA/SituatedQA and on a new clinical Text-to-SQL benchmark. In deployment, where interpretations are generated on the fly, it stays competitive and adds a diagnosis of the error source. This supports routing each query to either clarification or review.

Key points

  • Two scores instead of one. Uncertainty is split into ambiguity (H_I) and conditional instability (H_R|I). The baseline, KLE-style result entropy (H_R), ignores where answers came from.
  • How the scores are computed.
    • Answer-level structure comes from a Schur complement with ε = 10⁻³ added for invertibility.
    • The result is projected to positive semidefinite form by clipping negative eigenvalues, so H_R|I ≥ 0.
    • A heat-diffusion kernel is applied with τ = 10. This value was chosen by requiring near-zero entropy for identical items, and results are robust for τ between 2 and 20.
  • Black-box compatible. Similarity is judged by an LLM prompt asking whether two texts would yield equivalent SQL (or the same factual answer), so any black-box LLM can be used.
  • Error concentration. Queries high in both ambiguity and instability make up 25% of the data but account for 51% of errors, which makes triage efficient.
  • Released resources. A clinical Text-to-SQL dataset with multiple interpretations per question and an interpretation-generation procedure for epidemiological queries are available at github.com/Bayer-Group/clues-clinical-nlp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BayerLeverkusen · Germany4 works · 401 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Semantic parsing & meaning representation · 193 worksCode & programming languages · 168 works

Cite

Cite as Angelo Ziletti and Leonardo D'Ambrosi. 2026. Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4qse3ioqyh8s

@inproceedings{ziletti2026disentangling,
  title = {{Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study}},
  author = {Angelo Ziletti and Leonardo D'Ambrosi},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4qse3ioqyh8s},
  url = {https://doi.org/10.63317/4qse3ioqyh8s}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. 2024. Distinguishing the knowable from the unknowable with language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  2. Anthropic AI. 2023. Model card and evaluations for claude models. https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf . Accessed: February 15, 2024. ↗
  3. Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press, Cambridge, England. find
  4. Spiros Denaxas, Kenan Direk, Arturo Gonzalez-Izquierdo, Maria Pikoula, Aylin Cakiroglu, Jason Moore, Harry Hemingway, and Liam Smeeth. 2017. Methods for enhancing the reproducibility of biomedical research findings using electronic health records. BioData Mining, 10:31. find
  5. Naihao Deng, Yulong Chen, and Yue Zhang. 2022. Recent advances in text-to-SQL: A survey of what we have and what we expect. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2166–2187, Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detectin… find
  6. Darius Foodeei, Simin Fan, and Martin Jaggi. 2025. Semantic uncertainty in advanced decoding methods for llm generation. arXiv preprint arXiv:2506.17296. — [arXiv:2506.17296](https://arxiv.org/abs/2506.17296) find
  7. Gemini Team. 2023. Gemini: A family of highly capable multimodal models. Technical report, Google. Accessed: February 15, 2024.
  8. Shrestha Ghosh, Moritz Schneider, Carina Reinicke, and Carsten Eickhoff. 2025. A survey on LLM-assisted clinical trial recruitment. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Compu… find
  9. Yashvir S. Grewal, Edwin V. Bonilla, and Thang D. Bui. 2024. Improving uncertainty quantification in large language models via semantic embeddings. arXiv preprint arXiv:2410.22685. — [arXiv:2410.22685](https://arxiv.org/abs/2410.22685) find
  10. Sebastian G. Gruber and Florian Buettner. 2024. A bias-variance-covariance decomposition of kernel scores for generative models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  11. Nicholas J. Higham. 2002. Computing the nearest correlation matrix—a problem from finance. IMA Journal of Numerical Analysis, 22(3):329–343. find
  12. Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457–506. find
  13. Yongrae Jo, Seongyun Lee, Minju Seo, Sung Ju Hwang, and Moontae Lee. 2024. LG AI research & KAIST at EHRSQL 2024: Self-training large language models with pseudo-labeled unanswerable questions for a reliable text-to-SQL system on EHRs. In Proceedings of the 6th Clinical Natural Language Processing W… find
  14. Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977.
  15. Daeyoung Kim, Seongsu Bae, Seungho Kim, and Edward Choi. 2022. Uncertainty-aware text-to-program for question answering on structured electronic health records. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 138–151.… find
  16. Sangryul Kim, Donghee Han, and Sehyun Kim. 2024. ProbGate at EHRSQL 2024: Enhancing SQL query generation accuracy through probabilistic threshold filtering and error handling. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 687–696, Mexico City, Mexico. Association for… find
  17. Mathew J Koretsky, Maya Willey, et al. 2025. BiomedSQL: Text-to-SQL for scientific reasoning on biomedical knowledge bases. arXiv preprint arXiv:2505.20321. — [arXiv:2505.20321](https://arxiv.org/abs/2505.20321) find
  18. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. International Conference on Learning Representations (ICLR).
  19. Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong C Lee, and Edward Choi. 2023. EHRSQL: A practical text-to-SQL benchmark for electronic health records. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. find
  20. Xiaomin Li, Zhou Yu, Ziji Zhang, Yingying Zhuang, Swair Shah, Narayanan Sadagopan, and Anurag Beniwal. 2025a. Semantic volume: Quantifying and detecting both external and internal uncertainty in llms. arXiv preprint arXiv:2502.21239. — [arXiv:2502.21239](https://arxiv.org/abs/2502.21239) find
  21. Zhaoye Li, Siyuan Shen, Wenjing Yang, Ruochun Jin, Huan Chen, Ligong Cao, and Jing Ren. 2025b. Enhancing uncertainty quantification in large language models through semantic graph density. In Proceedings of the Forty-First Conference on Uncertainty in Artificial Intelligence, UAI ’25. JMLR.org. find
  22. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics. find
  23. Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: fine-grained uncertainty quantification for llms from semantic similarities. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Cur… find
  24. OHDSI Community. 2024. Atlas: Open source software for observational data analysis. https://github.com/OHDSI/Atlas . Accessed: 2026-01-28. ↗
  25. OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark… — gpt-oss-120b & gpt-oss-20b Model Card
  26. Shalini Raghuvanshi, Yashs Tiwari, and Abhishek Yadav. 2025. Integrating token-level uncertainty, bidirectional nli, and semantic entropy for robust hallucination detection in large language models. In 2025 International Conference on Artificial intelligence and Emerging Technologies (ICAIET), pages… find
  27. Jaehee Ryu, Seonhee Cho, Gyubok Lee, and Edward Choi. 2024. EHR-SeqSQL : A sequential text-to-SQL dataset for interactively exploring electronic health records. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16388–16407, Bangkok, Thailand. Association for Computational… find
  28. Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H… — Kimi K2: Open Agentic Intelligence
  29. Irina Saparina and Mirella Lapata. 2024. Ambrosia: a benchmark for parsing ambiguous questions into database queries. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  30. Irina Saparina and Mirella Lapata. 2025. Disambiguate first, parse later: Generating interpretations for ambiguity resolution in semantic parsing. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16825–16839, Vienna, Austria. Association for Computational Linguistics. find
  31. J Schur. 1917. Über potenzreihen, die im innern des einheitskreises beschränkt sind. J. Reine Angew. Math., 1917(147):205–232. find
  32. Haoyi Song, Ruihan Ji, Naichen Shi, Fan Lai, and Raed Al Kontar. 2025. Inv-entropy: A fully probabilistic framework for uncertainty quantification in language models. arXiv preprint arXiv:2506.09684. — [arXiv:2506.09684](https://arxiv.org/abs/2506.09684) find
  33. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong G… — Qwen3 Technical Report
  34. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018…
  35. Fuzhen Zhang, editor. 2005. The schur complement and its applications, 2005 edition. Numerical Methods and Algorithms. Springer, New York, NY. find
  36. Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  37. Angelo Ziletti and Leonardo DAmbrosi. 2024. Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 47–53, Mexico City, Mexico. Association for Computational Ling… find
  38. Angelo Ziletti and Leonardo D’Ambrosi. 2025. Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-sql generation. arXiv preprint arXiv:2502.21107. — [arXiv:2502.21107](https://arxiv.org/abs/2502.21107) find
  39. Meredith N. Zozus, Rachel L. Richesson, Anita Walden, Jessie D. Tenenbaum, and W. E. Hammond. 2016. Research reproducibility in longitudinal multi-center studies using data from electronic health records. AMIA Joint Summits on Translational Science Proceedings, 2016:279– 285. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ziletti-2026-disentangling-ambiguity-instability-large-language/