Elamur Research⌕
☰
home / papers / 2026 / PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents

PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents

Paper · 2026 · Yuan Tian and Tianyi Zhang

Summary

Text-to-SQL models often fail because they misread what the database actually contains or overlook constraints in the question. PV-SQL is an agentic framework that targets both problems with two modules. The Probe module lets the LLM run small, LIMIT-bounded SELECT queries against the live database. From the results it learns value formats, column meanings and table links, for example that "California" is stored as "CA", and it collects these findings as grounding context. The Verify module uses pattern-matching rules to turn cues in the question, such as "unique" or "top-k", into a checklist that can be executed. It then repairs the SQL repeatedly until every check passes. The authors compared PV-SQL with seven baselines on BIRD, Mini-Dev and Spider using six base LLMs. PV-SQL ranked first on all three benchmarks and used fewer tokens than the best baseline. The work suggests that cheap, deterministic verification combined with exploration of database content works better than self-checking by the LLM.

Key points

  • An error study on BIRD used GPT-4o as a judge, which agreed with manual labels on 88% of 100 tasks. Database misinterpretation caused 41.3% of failures, question misinterpretation 24.8%, and synthesis errors 33.9%.
  • On BIRD, PV-SQL reaches 65.12% execution accuracy (EX) and a VES of 86.9. This is about 5 points of EX and 20.8% of VES above the strongest baseline, and 7.8 EX points above TS-SQL.
  • In the ablations, removing Probe lowers EX by 3.0 points on BIRD, 3.2 on Mini-Dev and 3.9 on Spider. Removing Repair lowers EX by 3.3, 3.0 and 0.7 respectively.
  • Replacing rule-based verification with LLM-based verification lowers accuracy by about 6%, doubles runtime and uses noticeably more tokens. The extracted constraints hold for gold SQL in 99.39% of cases. Repairs fix 90.82% of violations and break previously satisfied constraints in 8.71% of cases.
  • PV-SQL cuts database-misinterpretation errors by 42%, synthesis errors by 19% and question-misinterpretation errors by only 6%. Code is available on GitHub at magic-YuanTian/PV-SQL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Purdue University West LafayetteWest Lafayette · United States23 works · 1,417 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

🧭 Topics 3Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 worksCode & programming languages · 129 worksInterpretability · 130 works

Cite

Cite as Yuan Tian and Tianyi Zhang. 2026. PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1286

@inproceedings{tian2026pv,
  title = {{PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents}},
  author = {Yuan Tian and Tianyi Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1286},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1286}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arian Askari, Christian Poelitz, and Xinye Tang. 2024. MAGIC: Generating self-correction guidelines for in-context text-to-SQL. arXiv preprint arXiv:2406.12692. — [arXiv:2406.12692](https://arxiv.org/abs/2406.12692) find
  2. Hasan Alp Caferoğlu and Özgür Ulusoy. 2025. E-sql: Direct schema linking via question enrichment in text-to-sql. Preprint, arXiv:2409.16751. — [arXiv:2409.16751](https://arxiv.org/abs/2409.16751) find
  3. Jipeng Cen, Jiaxin Liu, Zhixu Li, and Jingjing Wang. 2024. SQLFixAgent: Towards semantic-accurate SQL generation via multi-agent collaboration. arXiv preprint arXiv:2406.13408. — [arXiv:2406.13408](https://arxiv.org/abs/2406.13408) find
  4. Saumya Chaturvedi, Aman Chadha, and Laurent Bindschaedler. 2025. Sql-of-thought: Multi-agentic text-to-sql with guided error correction. Preprint, arXiv:2509.00581. — [arXiv:2509.00581](https://arxiv.org/abs/2509.00581) find
  5. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. Preprint, arXiv:2207.10397. find
  6. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. find
  7. Zui Chen, Han Li, Xinhao Zhang, Xiaoyu Chen, Chun-yin Dong, Yifeng Wang, Xin Cai, Su Zhang, Ziqi Li, Chi Ding, Jinxu Li, Shuai Wang, Dousheng Zhao, Sanhai Gao, and Guangyi Liu. 2025. Rubiksql: Lifelong learning agentic knowledge base as an industrial nl2sql system. Preprint, arXiv:2508.17590. — [arXiv:2508.17590](https://arxiv.org/abs/2508.17590) find
  8. Nadezhda Chirkova, Tunde Oluwaseyi Ajayi, Seth Aycock, Zain Muhammad Mujahid, Vladana Per-lić, Ekaterina Borisova, and Markarit Vartampetian. 2025. Llm-as-a-qualitative-judge: automating error analysis in natural language generation. Preprint, arXiv:2506.09147. find
  9. Rushikesh Deotale, Adithya Srinivasan, Yuan Tian, Tianyi Zhang, Pavlos Vlachos, and Hector Gomez. 2026. All-fem: Agentic large language models fine-tuned for finite element methods. Preprint, arXiv:2603.21011. — [arXiv:2603.21011](https://arxiv.org/abs/2603.21011) find
  10. Zhongjun Ding, Yin Lin, and Tianjing Zeng. 2025. Am-bisql: Interactive ambiguity detection and resolution for text-to-sql. Preprint, arXiv:2508.15276. — [arXiv:2508.15276](https://arxiv.org/abs/2508.15276) find
  11. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. DAIL-SQL: Optimized llm prompt for text-to-SQL. In Proceedings of the VLDB Endowment, volume 17, pages 950–961. find
  12. Yue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya, Balakrishnan Narayanaswamy, and Tim Kraska. 2025. Sqlens: An end-to-end framework for error detection and correction in text-to-sql. Preprint, arXiv:2506.04494. — [arXiv:2506.04494](https://arxiv.org/abs/2506.04494) find
  13. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. Preprint, arXiv:2310.01798.
  14. Jihyung Lee, Jin-Seop Lee, Jaehoon Lee, YunSeok Choi, and Jee-Hyong Lee. 2025. DCG-SQL: Enhancing in-context learning for text-to-SQL with deep contextual schema link graph. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1539… find
  15. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thir…
  16. Jiawen Li, Zheng Ning, Yuan Tian, and Toby Jia jun Li. 2025. Alloy: Generating reusable agent workflows from user demonstration. Preprint, arXiv:2510.10049. — [arXiv:2510.10049](https://arxiv.org/abs/2510.10049) find
  17. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Li, Bei Hui, and Yongbin Li. 2024. Can LLM already serve as a database interface? a Big bench for large-scale database gro…
  18. Zhenwen Li and Tao Xie. 2024. Using LLM to select the right SQL query from candidates. arXiv preprint arXiv:2401.02115. — [arXiv:2401.02115](https://arxiv.org/abs/2401.02115) find
  19. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025a. A survey of text-to-sql in the era of llms: Where are we, and where are we going? Preprint, arXiv:2408.05109. — [arXiv:2408.05109](https://arxiv.org/abs/2408.05109) find
  20. Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong, Jinyang Gao, Yu Li, Bolin Ding, and Jingren Zhou. 2025b. Xiyan-sql: A novel multi-generator framework for text-to-sql. Preprint, arXiv:2507.04701. — [arXiv:2507.04701](https://arxiv.org/abs/2507.04701) find
  21. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  22. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36. find
  23. Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023. Lever: learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  24. Zheng Ning, Yuan Tian, Zheng Zhang, Tianyi Zhang, and Toby Jia-Jun Li. 2024. Insights into natural language database query errors: from attention misalignment to user handling strategies. ACM Trans. Interact. Intell. Syst., 14(4). find
  25. Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-sql generation. Preprint, arXiv:2405.15307. find
  26. Md Mahadi Hassan Rahaman and Mehmet Emre Gursoy. 2024. Evaluating sql understanding in large language models. Preprint, arXiv:2410.10680. — [arXiv:2410.10680](https://arxiv.org/abs/2410.10680) find
  27. Minseok Ren, Sijie Jin, Soobin Son, Yeonsu Kwon, Dong-Gun Lee, Hwi-Jun Yang, and Kyomin Jung. 2024. Purple: Making a large language model a better sql writer. Preprint, arXiv:2403.20014. — [arXiv:2403.20014](https://arxiv.org/abs/2403.20014) find
  28. Chetan Sharma, Ramasuri Narayanam, Soumyabrata Pal, Kalidas Yeturu, Shiv Kumar Saini, and Koyel Mukherjee. 2025. TTD-SQL: Tree-guided token decoding for efficient and schema-aware SQL generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Trac… find
  29. Jiawei Shen, Chengcheng Wan, Ruoyi Qiao, Jiazhen Zou, Hang Xu, Yuchen Shao, Yueling Zhang, Weikai Miao, and Geguang Pu. 2025. A study of in-context-learning-based text-to-sql errors. Preprint, arXiv:2501.09310. — [arXiv:2501.09310](https://arxiv.org/abs/2501.09310) find
  30. Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. 2024. Small LLMs are weak tool learners: A multi-LLM agent. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16658–16680, Miami, Florida, USA. Asso…
  31. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314.
  32. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. Preprint, arXiv:2405.16755. — [arXiv:2405.16755](https://arxiv.org/abs/2405.16755) find
  33. Zheng Ning, Zheng Zhang, Tianyi Sun, Yuan Tian, Tianyi Zhang, and Toby Jia-Jun Li. 2023. An empirical study of model errors and user error discovery and repair strategies in natural language database queries. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23… find
  34. Yuan Tian, Jonathan K. Kummerfeld, Toby Jia-Jun Li, and Tianyi Zhang. 2024. Sqlucid: Grounding natural language database queries with interactive explanations. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’24, New York, NY, USA. Association for Comp… find
  35. Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. 2024. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql. Preprint, arXiv:2410.01943. — [arXiv:2410.01943](https://arxiv.org/abs/2410.01943) find
  36. Yuan Tian, Daniel Lee, Fei Wu, Tung Mai, Kun Qian, Siddhartha Sahai, Tianyi Zhang, and Yunyao Li. 2025. Text-to-sql domain adaptation via human-llm collaborative data annotation. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI ’25, page 1398–1425, New York, NY… find
  37. Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Preprint, arXiv:2304.11015.
  38. Yuan Tian and Tianyi Zhang. 2025. Selective prompt anchoring for code generation. Preprint, arXiv:2408.09121. — [arXiv:2408.09121](https://arxiv.org/abs/2408.09121) find
  39. Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, and Tianyi Zhang. 2023. Interactive text-to-SQL generation via editable step-by-step explanations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16149–16166, Singapore. Ass… find
  40. Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2024. Mac-sql: A multi-agent collaborative framework for text-to-sql. Preprint, arXiv:2312.11242.
  41. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  42. Wenbo Xu, Haifeng Zhu, Liang Yan, Chuanyi Liu, Peiyi Han, Shaoming Duan, and Jeff Z. Pan. 2025. TS-SQL: Test-driven self-refinement for text-to-SQL. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2864–2889. find
  43. Fan Yang, Yuan Tian, and Jiansong Zhang. 2025. Supporting construction worker well-being with a multi-agent conversational ai system. Preprint, arXiv:2506.07997. — [arXiv:2506.07997](https://arxiv.org/abs/2506.07997) find
  44. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018…
  45. Hongxiang Zhang, Yuan Tian, and Tianyi Zhang. 2026. Attention-aligned reasoning for large language models. Preprint, arXiv:2510.03223. find
  46. Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, and Yunyao Li. 2025. Evoschema: Towards text-to-sql robustness against schema evolution. Proc. VLDB Endow., 18(10):3655–3668. find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neu…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-pv-sql-synergizing-database-probing/