Elamur Research⌕
☰
home / papers / 2026 / CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

Paper · 2026 · Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang et al.

Summary

Benchmarks for NL2SQL usually assume that each question contains one source of ambiguity and that users answer clarification requests cooperatively. Real deployments are messier: one question can be confusing in several ways at once, and users may give vague or useless replies. The Oracle team behind CLARITY built a framework that automatically turns executable SQL from Spider and BIRD into ambiguous or unanswerable questions, with no human annotation.

The pipeline works in stages:

  • A rule-based SQL parser picks target columns or values.
  • An adaptive schema retriever groups similar schema elements.
  • An LLM invents "pivot terms" that are either ambiguous or unanswerable, and LLM judges must all approve them.
  • The pivot terms are woven into the original question, and the result is expanded into single- or multi-turn dialogues.

Each instance also records span-level pivot terms and the schema target groups they point to. This metadata lets evaluators check whether a system finds the right schema elements, not just whether it flags a problem. Tested on CLARITY, strong LLMs and the AmbiSQL detector usually notice that a question is ambiguous but often fail to locate and resolve the cause. This failure grows much worse when several ambiguities occur together.

Key points

  • The framework covers four A/U modes (col_amb, val_amb, col_unans, val_unans) and three user response types (helpful, partial, unhelpful). Ambiguity is further split into lexical and semantic, and into uni-facet and multi-facet cases.
  • AmbiSQL detects column ambiguity on Spider almost perfectly (DA 99.6%), but its match accuracy is low:
    • 57.8% MA for uni-facet questions on Spider.
    • About 20% MA for multi-facet questions on Spider.
    • 5.4–7.0% MA for multi-facet questions on BIRD.
  • In multi-turn SQL prediction, the evaluated LLMs included GPT-4o, GPT-4.1, GPT-5, Grok-3 Mini Fast and LLaMA-3.3. Results across these models were:
    • Models typically handle unanswerable questions better than ambiguous ones.
    • Multi-facet ambiguity is harder than uni-facet ambiguity.
    • Verbose clarifications help most.
    • No single model was best in every category.
  • Few-shot prompts built from CLARITY examples with metadata significantly outperform zero-shot and standard few-shot prompting (paired t-tests, p < 0.005).
  • Because the framework needs no human annotation, companies can generate ambiguous and unanswerable test variants of their own enterprise datasets at low cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢OracleAustin · United States7 works · 1,167 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksDialogue systems · 624 works

Cite

Cite as Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang, Xiaoxiao Ma, Shawn Chang Xu, Fahimeh Saleh, Poorya Zaremoodi, Avirup Sil and Katrin Kirchhoff. 2026. CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.86

@inproceedings{sarwar2026clarity,
  title = {{CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems}},
  author = {Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang, Xiaoxiao Ma, Shawn Chang Xu, Fahimeh Saleh, Poorya Zaremoodi, Avirup Sil and Katrin Kirchhoff},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.86},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.86}
}

References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-sql generation under ambiguity. arXiv preprint arXiv:2310.13659. find
  2. Hasan Alp Caferoğlu, Mehmet Serhat Çelik, and Özgür Ulusoy. 2025. Sing-sql: A synthetic data generation framework for in-domain text-to-sql translation. arXiv preprint arXiv:2509.25672. — [arXiv:2509.25672](https://arxiv.org/abs/2509.25672) find
  3. Kaiwen Chen, Yueting Chen, Nick Koudas, and Xiaohui Yu. 2025. Reliable text-to-sql with adaptive abstention. Proceedings of the ACM on Management of Data, 3(1):1–30. find
  4. Zhongjun Ding, Yin Lin, and Tianjing Zeng. 2025. Am-bisql: Interactive ambiguity detection and resolution for text-to-sql. arXiv preprint arXiv:2508.15276. — [arXiv:2508.15276](https://arxiv.org/abs/2508.15276) find
  5. Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, and 1 others. 2025. Practiq: A practical conversational text-to-sql dataset with ambiguous and unanswerable queries. In Proceedings of the 2025 Conference of th…
  6. Shaoming Duan, Youxuan Wu, Chuanyi Liu, Yuhao Zhang, Zirui Wang, Peiyi Han, Shengyuan Yu, Liang Yan, and Yingwei Liang. 2025. Dsqg-syn: Synthesizing high-quality data for text-to-sql parsing by domain specific question generation. In Findings of the Association for Computational Linguistics: NAACL 2… find
  7. Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jianyang Jianyang, and Xiaobin Tan. 2025. Sqlforge: Synthesizing reliable and diverse data to enhance text-to-sql reasoning in llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452. find
  8. Ziming Guo, Chao Ma, Yinggang Sun, Tiancheng Zhao, Guangyao Wang, and Hai Huang. 2024. Evaluating and enhancing llms for multi-turn text-to-sql with multiple question types. arXiv preprint arXiv:2412.17867. — [arXiv:2412.17867](https://arxiv.org/abs/2412.17867) find
  9. Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of llm-based text-to-sql. IEEE Transactions on Knowledge and Data Engineering. find
  10. Yiming Huang, Jiyu Guo, Wenxin Mao, Cuiyun Gao, Peiyi Han, Chuanyi Liu, and Qing Ling. 2025. Exploring the landscape of text-to-sql with large language models: Progresses, challenges and opportunities. arXiv preprint arXiv:2505.23838. — [arXiv:2505.23838](https://arxiv.org/abs/2505.23838) find
  11. Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, and 1 others. 2025. Bird-interact: Re-imagining text-to-sql evaluation for large language models via lens of dynamic interactions. arXiv preprint arXiv:2510.05318. — [arXiv:2510.05318](https://arxiv.org/abs/2510.05318) find
  12. Shivasankari Kannan, Yeounoh Chung, Amita Gondi, Tristan Swadell, and Fatma Ozcan. 2025. High-fidelity and complex test data generation for real-world sql code generation services. arXiv preprint arXiv:2504.17203. — [arXiv:2504.17203](https://arxiv.org/abs/2504.17203) find
  13. Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. 2025. Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 337– 353.
  14. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The dawn of natural language to sql: Are we fully ready? arXiv preprint arXiv:2406.01265. — [arXiv:2406.01265](https://arxiv.org/abs/2406.01265) find
  15. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, and 1 others. 2025. Omnisql: Synthesizing high-quality text-to-sql data at scale. arXiv preprint arXiv:2503.02240. — [arXiv:2503.02240](https://arxiv.org/abs/2503.02240) find
  16. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can llm already serve as a database interface? a big bench for large-scale database…
  17. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025. A survey of text-to-sql in the era of llms: Where are we, and where are we going? IEEE Transactions on Knowledge and Data Engineering. find
  18. Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. 2025. nvbench 2.0: Resolving ambiguity in text-to-visualization through stepwise reasoning. arXiv preprint arXiv:2503.12880. — [arXiv:2503.12880](https://arxiv.org/abs/2503.12880) find
  19. Mihai Nadăs , , Laura Dios , an, and Andreea Tomescu. 2025. Synthetic data generation using large language models: Advances in text and code. IEEE Access. find
  20. Simone Papicchio, Luca Cagliero, and Paolo Papotti. 2025. Squab: Evaluating llm robustness to ambiguous and unanswerable questions in semantic parsing. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17937–17957. find
  21. Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. In Advances in Neural Information Processing Systems, volume 36, pages 36339–36348. Curran Associates, Inc.
  22. Mohammadreza Pourreza, Ruoxi Sun, Hailong Li, Lesly Miculicich, Tomas Pfister, and Sercan O Arik. 2024. Sql-gen: Bridging the dialect gap for text-to-sql via synthetic data and model merging. arXiv preprint arXiv:2408.12733. — [arXiv:2408.12733](https://arxiv.org/abs/2408.12733) find
  23. Irina Saparina and Mirella Lapata. 2024. Ambrosia: A benchmark for parsing ambiguous questions into database queries. Advances in Neural Information Processing Systems, 37:90600–90628. find
  24. Irina Saparina and Mirella Lapata. 2025. Disambiguate first parse later: Generating interpretations for ambiguity resolution in semantic parsing. arXiv preprint arXiv:2502.18448. find
  25. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Computing Surveys, 58(2):1–37. find
  26. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755. — [arXiv:2405.16755](https://arxiv.org/abs/2405.16755) find
  27. Kapil Vaidya, Abishek Sankararaman, Jialin Ding, Chuan Lei, Xiao Qin, Balakrishnan Narayanaswamy, and Tim Kraska. 2025. Odin: A nl2sql recommender to handle schema ambiguity. arXiv preprint arXiv:2505.19302. — [arXiv:2505.19302](https://arxiv.org/abs/2505.19302) find
  28. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.
  29. Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023. Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5701–5714. find
  30. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018…
  31. Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and Amr El Abbadi. 2024. Sphinteract: Resolving ambiguities in nl2sql through user interaction. Proceedings of the VLDB Endowment, 18(4):1145–1158. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sarwar-2026-clarity-framework-benchmark-conversational-language/