Elamur Research⌕
☰
home / papers / 2026 / VET: Verifiable Execution Tracing for Reliable Text-to-SQL Generation

VET: Verifiable Execution Tracing for Reliable Text-to-SQL Generation

Paper · 2026 · Dongyu Wang, Jingyu Li, Lan Zhang et al.

Summary

LLM-based text-to-SQL systems often hallucinate schema elements, data values or query logic. Their intermediate reasoning is either free text that nothing can check or code that runs only once at the end. VET (Verifiable Execution Tracing) is a training-free framework that replaces textual rationales with a chain of small Python operations, each run on the actual database.

The pipeline has three stages:

  1. Schema-constrained initialization. Schema linking narrows the reasoning to a set of candidate columns.
  2. Iterative executable reasoning. Each step's output, such as an intermediate DataFrame, is inspected. Errors or failed checks trigger a repair, and data probes resolve ambiguous values.
  3. SQL synthesis. The final query is built from the verified trace and accepted only if two checks pass. Its result must agree with the trace's final output, and a question reconstructed from it must stay close to the original.

Under comparable training-free conditions, VET attains 37.04% execution accuracy on Spider 2.0-lite and 70.93% on BIRD, improving most on difficult queries. The work shows that grounding each reasoning step in observable database state makes generation more reliable and easier to debug.

Key points

  • Formulation: generation is modeled as Y = G(V(E(Q, S), D)), where E builds the executable trace, V verifies it against database D, and G synthesizes the SQL.
  • Spider 2.0-lite with DeepSeek-Chat: the baseline scores 14.82% and full VET reaches 37.04%. Removing any single component lowers the score.
  • Ablation on BIRD-Dev (overall execution accuracy):
    • Full VET scores 68.38%.
    • Dropping execution feedback gives 63.04%, with challenging queries falling from 61.11% to 45.83%.
    • Dropping step-wise verification gives 63.56%.
    • Dropping the unified consistency check gives 63.30%.
    • Dropping schema constraints gives 66.30%.
  • Positioning against LEVER: the authors describe their verification as a "generative white-box" approach, in contrast to LEVER's trained discriminator that ranks candidates.
  • Limitations:
    • Step-wise execution adds inference latency.
    • Deployment involves a privacy–observability trade-off between a Secure mode and a Transparent/Auditing mode.
    • The authors recommend read-only database credentials and bounded execution budgets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksSemantic parsing & meaning representation · 185 worksCode & programming languages · 129 works

Cite

Cite as Dongyu Wang, Jingyu Li, Lan Zhang, Ganggang Yu and Liang Huang. 2026. VET: Verifiable Execution Tracing for Reliable Text-to-SQL Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1544

@inproceedings{wang2026vet,
  title = {{VET: Verifiable Execution Tracing for Reliable Text-to-SQL Generation}},
  author = {Dongyu Wang, Jingyu Li, Lan Zhang, Ganggang Yu and Liang Huang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1544},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1544}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, Wei Chen, and Xiang Bai. 2024. Rsl-sql: Robust schema linking in text-to-sql generation. Preprint, arXiv:2411.00073. — [arXiv:2411.00073](https://arxiv.org/abs/2411.00073) find
  2. Yongdong chi, Hanqing Wang, Zonghan Yang, Jian Yang, Xiao Yan, Yun Chen, and Guanhua Chen. 2025. Pi-sql: Enhancing text-to-sql with fine-grained guidance from pivot programming languages. Preprint, arXiv:2506.00912. find
  3. Yaxun Dai, Haiqin Yang, Mou Hao, and Pingfu Chao. 2025. PARSQL: Enhancing text-to-SQL through SQL parsing and reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 661–681, Vienna, Austria. Association for Computational Linguistics. find
  4. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023a. Text-to-sql empowered by large language models: A benchmark evaluation. Preprint, arXiv:2308.15363.
  5. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023b. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.
  6. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023c. Pal: Program-aided language models. Preprint, arXiv:2211.10435. find
  7. Omid Reza Heidari, Siobhan Reid, and Yassine Yaakoubi. 2025. Agentiql: An agent-inspired multi-expert framework for text-to-sql generation. Preprint, arXiv:2510.10661. — [arXiv:2510.10661](https://arxiv.org/abs/2510.10661) find
  8. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, and 1 others. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql work. find
  9. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024a. The dawn of natural language to sql: Are we fully ready? Proceedings of the VLDB Endowment, 17(11):33183331. find
  10. Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025a. Alpha-sql: Zero-shot text-to-sql using monte carlo tree search. Preprint, arXiv:2502.17248. — [arXiv:2502.17248](https://arxiv.org/abs/2502.17248) find
  11. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, and 1 others. 2024b. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.
  12. Miaoran Li, Jiangning Chen, Minghua Xu, and Xiaolong Wang. 2025b. Hallucination detection in structured query generation via llm self-debating. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16102–16113. find
  13. Hanbing Liu, Haoyang Li, Xiaokang Zhang, Ruotong Chen, Haiyong Xu, Tian Tian, Qi Qi, and Jing Zhang. 2025. Uncovering the impact of chain-of-thought reasoning for direct preference optimization: Lessons from text-to-sql. Preprint, arXiv:2502.11656. find
  14. Giang Nguyen, Ivan Brugere, Shubham Sharma, Sanjay Kariyappa, Anh Totti Nguyen, and Freddy Lecue. 2025. Interpretable llm-based table question answering. Preprint, arXiv:2412.12386. — [arXiv:2412.12386](https://arxiv.org/abs/2412.12386) find
  15. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pages 26106–26128. PMLR.
  16. Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36.
  17. Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-sql generation. arXiv preprint arXiv:2405.15307. find
  18. Ohad Rubin and Jonathan Berant. 2020. Smbop: Semi-autoregressive bottom-up semantic parsing. arXiv preprint arXiv:2010.12412. find
  19. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755. — [arXiv:2405.16755](https://arxiv.org/abs/2405.16755) find
  20. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942.
  21. Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025a. find
  22. MAC-SQL: A multi-agent collaborative framework for text-to-SQL. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540–557, Abu Dhabi, UAE. Association for Computational Linguistics.
  23. Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, LinZheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025b. Mac-sql: A multi-agent collaborative framework for text-to-sql. Preprint, arXiv:2312.11242.
  24. Yihan Wang, Peiyu Liu, and Xin Yang. 2025c. Linkalign: Scalable schema linking for real-world Preprint, large-scale multi-database text-to-sql. arXiv:2503.18596. — [arXiv:2503.18596](https://arxiv.org/abs/2503.18596) find
  25. Hanchen Xia, Feng Jiang, Naihao Deng, Cunxiang Wang, Guojiang Zhao, Rada Mihalcea, and Yue Zhang. 2024. r 3 : "this is my sql, are you with me?" a consensus-based multi-agent system for text-to-sql tasks. Preprint, arXiv:2402.14851. find
  26. Bo Yang, Yinfen Xia, Weisong Sun, and Yang Liu. 2025. Hallucination detection for llm-based text-to-sql generation via two-stage metamorphic testing. Preprint, arXiv:2512.22250. — [arXiv:2512.22250](https://arxiv.org/abs/2512.22250) find
  27. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. Preprint, arXiv:1809.08887.
  28. Bohan Zhai, Canwen Xu, Yuxiong He, and Zhewei Yao. 2025. Excot: Optimizing reasoning for Preprint, text-to-sql with execution feedback. arXiv:2503.19988. — [arXiv:2503.19988](https://arxiv.org/abs/2503.19988) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-vet-verifiable-execution-tracing-reliable/