Elamur Research⌕
☰
home / papers / 2026 / Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

Paper · 2026 · Linzhuang Sun, Tianyu Guo, Hao Liang et al.

Summary

Most Text-to-SQL benchmarks test one-shot translation of a fixed question into a read-only query. Real database work is different: users refine goals over several turns and often need to change data, not just read it. The authors introduce DySQL-Bench to evaluate LLM agents in this setting. It contains 1,072 multi-turn tasks built on 13 databases from BIRD and SPIDER 2 and covers the full set of CRUD operations.

Tasks are created in two stages:

  1. Generation: records linked to a primary entity are packed into a hierarchical JSON tree, and GPT-4 writes instructions and golden action sequences from that tree.
  2. Quality control: LLM verifier committees check the tasks, missing parameters are filled in, executability is tested, and human experts give final approval.

During evaluation, an LLM-simulated user talks to the agent under test, which works on a live database. Success is judged by comparing hashes of the final database state against the state produced by the golden actions. Even GPT-4o struggles. This shows that dependable conversational database agents are still an open problem.

Key points

  • Scale and sources: 1,072 tasks over 13 databases (e.g., bowling, pagila, chinook, eu_soccer) drawn from BIRD and SPIDER 2, covering SELECT, UPDATE, INSERT and DELETE.
  • Generation cost: the tree-based Hierarchical Logic Tree generation cuts token cost and generation time by about 90% compared with iterative online sampling.
  • Verification: LLM committees use DeepSeek-r1 and Qwen3-235B-A22B-2507, followed by expert review, which the authors report yields 100% task correctness.
  • Main result: GPT-4o achieves 58.34% overall accuracy but only 23.81% on Pass^5. Its rare SQL errors do not translate into task success, which points to intent tracking and state planning as the main weaknesses.
  • Hallucinated results: some models invent execution results instead of waiting for the database's response. Rates are 44.28% for DeepSeek-V3, 26.15% for OmniSQL-32B and 0.09% for GPT-4o.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

🧭 Topics 2Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 worksCode & programming languages · 129 works

Cite

Cite as Linzhuang Sun, Tianyu Guo, Hao Liang, Ruitong Liu, Yuying Li, Qifeng Cai, Jingxuan Wei, Yuchen Wu, Bihui Yu, Xiangxiang Zhang, Wentao Zhang and Bin Cui. 2026. Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1654

@inproceedings{sun2026rethinking,
  title = {{Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration}},
  author = {Linzhuang Sun, Tianyu Guo, Hao Liang, Ruitong Liu, Yuying Li, Qifeng Cai, Jingxuan Wei, Yuchen Wu, Bihui Yu, Xiangxiang Zhang, Wentao Zhang and Bin Cui},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1654},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1654}
}

References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. Hallulens: Llm hallucination benchmark. arXiv preprint arXiv:2504.17550.
  2. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. tau2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. find
  3. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-sql generation under ambiguity. arXiv preprint arXiv:2310.13659. find
  4. Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, et al. 2025. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470. find
  5. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  7. Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of llm-based text-to-sql. IEEE Transactions on Knowledge and Data Engineering. find
  8. George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-sql. The VLDB Journal, 32(4):905–936. find
  9. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763. — [arXiv:2411.07763](https://arxiv.org/abs/2411.07763) find
  10. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, et al. 2025a. Omnisql: Synthesizing high-quality text-to-sql data at scale. arXiv preprint arXiv:2503.02240. — [arXiv:2503.02240](https://arxiv.org/abs/2503.02240) find
  11. Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023. Can llm already serve as a database interface. A big bench for large-scale database grounded text-to-sqls. CoRR, abs/2305.03111.
  12. Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, et al. 2025b. Swe-sql: Illuminating llm pathways to solve user sql issues in real-world applications. arXiv preprint arXiv:2506.18951. — [arXiv:2506.18951](https://arxiv.org/abs/2506.18951) find
  13. Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2025c. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717. — [arXiv:2504.04717](https://arxiv.org/abs/2504.04717) find
  14. Peixian Ma, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo. 2025. Sql-r1: Training natural language to sql reasoning model by reinforcement learning. arXiv preprint arXiv:2504.08600. — [arXiv:2504.08600](https://arxiv.org/abs/2504.08600) find
  15. Anna Mitsopoulou and Georgia Koutrika. 2025. Analysis of text-to-sql benchmarks: limitations, challenges and opportunities. In Proceedings 28th International Conference on Extending Database Technology, EDBT 2025, pages 199–212. OpenProceedings. org. find
  16. Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36:36339–36348.
  17. Zihao Yi, Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. 2024. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013.
  18. Mohammadreza Pourreza and Davood Rafiei. 2024. Dts-sql: Decomposed text-to-sql with small large language models. arXiv preprint arXiv:2402.01117. find
  19. Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. arXiv preprint arXiv:1909.05378.
  20. Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, Sercan Arik, et al. 2025. Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql. arXiv preprint arXiv:2503.23157. — [arXiv:2503.23157](https://arxiv.org/abs/2503.23157) find
  21. Patti Price. 1990. Evaluation of spoken language systems: The atis domain. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990. find
  22. Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, et al. 2022. A survey on text-to-sql parsing: Concepts, methods, and future directions. arXiv preprint arXiv:2208.13629. — [arXiv:2208.13629](https://arxiv.org/abs/2208.13629) find
  23. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  24. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Computing Surveys, 58(2):1–37. find
  25. Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Conghui He, Zenan Zhou, and Wentao Zhang. 2024. Beats: Optimizing llm mathematical capabilities with backverify and adaptive disambiguate based efficient tree search. arXiv preprint arXiv:2409.17972. — [arXiv:2409.17972](https://arxiv.org/abs/2409.17972) find
  26. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887.
  27. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055.
  28. Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, and Hangyu Mao. 2024. Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation. arXiv preprint arXiv:2403.02951. — [arXiv:2403.02951](https://arxiv.org/abs/2403.02951) find
  29. Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. A survey on multi-turn interaction capabilities of large language models. arXiv preprint arXiv:2501.09959. — [arXiv:2501.09959](https://arxiv.org/abs/2501.09959) find
  30. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583.
  31. Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, et al. 2023. Mac-sql: A multi-agent collaborative framework for text-to-sql. arXiv preprint arXiv:2312.11242.
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  33. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045.
  34. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-rethinking-text-sql-dynamic-multi/