Elamur Research⌕
☰
home / papers / 2026 / Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

Paper · 2026 · Hanqing Wang, Yongdong Chi, Jian Yang et al.

Summary

LLM-based Text-to-SQL systems score well on benchmarks such as Spider and BIRD, but they can still fail in hidden ways. Existing robustness tests rely on fixed, expert-written perturbation rules, so they only confirm failure types that people already expect. Here, "vulnerability" means brittleness under meaning-preserving input changes, not security attacks like SQL injection.

The authors introduce SAGE (Systematic Automated Guided Exploration), an iterative, agent-based framework that runs a closed loop:

  • Discovery. Starting from samples the target model already answers correctly, a Generator proposes vulnerability hypotheses and builds perturbed variants. It is guided by experience retrieved from a Vulnerability Codex. A Checker then confirms that each variant keeps the original execution result.
  • Evolution. A Summarizer turns the confirmed failures into generalized error archetypes and compresses them back into the Codex for the next round.

Across open-source LLMs, SAGE exposes far more failures than static expert baselines. The learned Codex also transfers between models, which suggests the weaknesses are shared rather than model-specific. Fine-tuning on SAGE-generated samples partly repairs these weaknesses, pointing toward a diagnose-and-fix cycle.

Key points

  • SAGE beats expert-curated static baselines by an average Vulnerability Exposure Rate (VER) margin of 26.33% on BIRD and 18.66% on Spider. Average VER across models is 76.98% on BIRD and 58.45% on Spider.
  • When each model uses its own evolved Codex, VER gains over the Expert baseline range from +10.91% to +18.47%.
  • Codices transfer across models. For example, a Codex built on OmniSQL raises VER on Gemma-3 by +13.74%.
  • For the remediation experiment, SAGE was run on 1,000 BIRD training samples against Gemma-3, producing 1,580 adversarial samples. These were used for SFT of Qwen2.5-Coder-7B-Instruct, with the following results:
    • EX on the adversarial set rose from 4.37% to 9.58%.
    • EX on the standard BIRD dev set rose from 42.63% to 52.87%.
    • Attempts per Discovery (ApD) rose from 3.61 to 4.59, meaning more attempts are now needed to find each failure.
  • Code is released at github.com/sustech-nlp/SAGE. The authors note two limitations: ablations are only at the framework level, and the Codex is updated at coarse intervals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beihang UniversityBeijing · China65 works · 8,452 citations🎓Southern University of Science and TechnologyShenzhen · China19 works · 1,337 citations🎓Shanghai University of Finance and EconomicsShanghai · China13 works · 825 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSoftware · 92 works

🧭 Topics 2Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 worksCode & programming languages · 129 works

Cite

Cite as Hanqing Wang, Yongdong Chi, Jian Yang, Lei Yang, Jiehui Zhao, Yun Chen and Guanhua Chen. 2026. Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.842

@inproceedings{wang2026beyond,
  title = {{Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL}},
  author = {Hanqing Wang, Yongdong Chi, Jian Yang, Lei Yang, Jiehui Zhao, Yun Chen and Guanhua Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.842},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.842}
}

References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch. 1995. Natural language interfaces to databases–an introduction. Natural language engineering, 1(1):29–81. find
  2. Arian Askari, Christian Poelitz, and Xinye Tang. 2025. Magic: Generating self-correction guideline for in-context text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23433–23441. find
  3. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-sql generation under ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053– 7074. find
  4. Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, and Bing Xiang. 2023a. Dr.spider: A diagnostic evaluation benchmark towards text-to-SQL r… find
  5. Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, and 1 others. 2023b. Dr. spider: A diagnostic evaluation benchmark towards text-to-sql robustness. arXiv preprint arXiv:2301.08881. — [arXiv:2301.08881](https://arxiv.org/abs/2301.08881) find
  6. Song Cheng, Qiannan Cheng, Linbo Jin, Lei Yi, and Guannan Zhang. 2025. Sqlord: A robust enterprise text-to-sql solution via reverse data generation and workflow decomposition. In Companion Proceedings of the ACM on Web Conference 2025, pages 919–923. find
  7. Yongdong Chi, Hanqing Wang, Yun Chen, Yan Yang, Jian Yang, Zonghan Yang, Xiao Yan, and Guanhua Chen. 2025. Pi-SQL: Enhancing text-to-SQL with fine-grained guidance from pivot programming languages. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25120– 25144, Suzhou,… find
  8. DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2021. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. Computational Linguistics, 47(2):309–332.
  9. Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, and Hao Zhang. 2025. Reforce: A text-to-sql agent with self-refinement, format restriction, and column exploration. In ICLR 2025 Workshop: VerifAI: AI Verification in the Wild. find
  10. Xiang Deng, Ahmed Hassan, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-grounded pretraining for text-to-sql. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
  11. Zhongjun Ding, Yin Lin, and Tianjing Zeng. 2025. Am-bisql: Interactive ambiguity detection and resolution for text-to-sql. arXiv preprint arXiv:2508.15276. — [arXiv:2508.15276](https://arxiv.org/abs/2508.15276) find
  12. Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Peng-sheng Huang. 2021a. Towards robustness of text-to-sql models against synonym substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation…
  13. Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. Exploring underexplored limitations of cross-domain text-to-sql generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926–8931.
  14. Yue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya, Balakrishnan Murali Narayanaswamy, and Tim Kraska. 2025. SQLens: An end-to-end framework for error detection and correction in text-to-SQL. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  15. Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jianyang Jianyang, and Xiaobin Tan. 2025. SQL-Forge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452, Vienna, Austria. Association fo… find
  16. Zijin Hong, Zheng Yuan, Qinggang Zhang, and 1 others. 2024. Next-generation database interfaces: A survey of llm-based text-to-sql. arXiv preprint arXiv:2406.08426. — [arXiv:2406.08426](https://arxiv.org/abs/2406.08426) find
  17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  18. Infly and InfTech. 2025. inf-rl-qwen-coder-32b-2746. https://huggingface.co/infly/ inf-rl-qwen-coder-32b-2746. Accessed: 2026-01-05. ↗
  19. Peng Lai, Zhihao Ou, Yong Wang, Longyue Wang, Jian Yang, Yun Chen, and Guanhua Chen. 2026. Biass-cope: Towards automated detection of bias in LLM-as-a-judge evaluation. In The Fourteenth International Conference on Learning Representations. find
  20. Fei Li and Hosagrahar V Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73–84. find
  21. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, and 1 others. 2025. Omnisql: Synthesizing high-quality text-to-sql data at scale. arXiv preprint arXiv:2503.02240. — [arXiv:2503.02240](https://arxiv.org/abs/2503.02240) find
  22. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067–13075.
  23. Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024a. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data, 2(3):1–28. find
  24. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, and 1 others. 2024b. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.
  25. Jinyang Li, Binyuan Hui, Ge Qu, and 1 others. 2023b. Bird: A big bench for large-scale database grounded text-to-sql. NeurIPS. find
  26. Hanbing Liu, Haoyang Li, Xiaokang Zhang, Ruotong Chen, Haiyong Xu, Tian Tian, Qi Qi, and Jing Zhang. 2025. Uncovering the impact of chain-of-thought reasoning for direct preference optimization: Lessons from text-to-sql. arXiv preprint arXiv:2502.11656. find
  27. Xiping Liu and Zhao Tan. 2024. Epi-sql: Enhancing text-to-sql translation with error-prevention instructions. arXiv preprint arXiv:2404.14453. find
  28. Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hybrid ranking network for text-to-sql. arXiv preprint arXiv:2008.04759. — [arXiv:2008.04759](https://arxiv.org/abs/2008.04759) find
  29. Tanzim Mahmud, KM Azharul Hasan, Mahtab Ahmed, and Thwoi Hla Ching Chak. 2015. A rule based approach for nlp based query processing. In 2015 2nd international conference on electrical information and communication technologies (EICT), pages 78–82. IEEE. find
  30. Rajaswa Patil, Manasi Patwardhan, Shirish Karande, Lovekesh Vig, and Gautam Shroff. 2023. Exploring dimensions of generalizability and few-shot transfer for text-to-sql semantic parsing. In Transfer Learning for Natural Language Processing Workshop, pages 103–114. PMLR. find
  31. Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022. Towards robustness of text-to-sql models against natural and realistic adversarial table perturbation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… find
  32. Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36:36339–36348.
  33. Ge Qu, Jinyang Li, Bowen Qin, Xiaolong Li, Nan Huo, Chenhao Ma, and Reynold Cheng. 2025. SHARE: An SLM-based hierarchical action CorREction assistant for text-to-SQL. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11268– 1129… find
  34. Ariana Sahitaj, Markus Nilles, Ralf Schenkel, and Vera Schmitt. 2025. Utilising large language models for adversarial attacks in text-to-sql: A perpetrator and victim approach. In Datenbanksysteme für Business, Technologie und Web (BTW 2025), pages 919–931. Gesellschaft für Informatik, Bonn. find
  35. Irina Saparina and Mirella Lapata. 2024. Ambrosia: A benchmark for parsing ambiguous questions into database queries. Advances in Neural Information Processing Systems, 37:90600–90628. find
  36. Jiawei Shen, Chengcheng Wan, Ruoyi Qiao, Jiazhen Zou, Hang Xu, Yuchen Shao, Yueling Zhang, Weikai Miao, and Geguang Pu. 2025. A study of in-context-learning-based text-to-sql errors. arXiv preprint arXiv:2501.09310. — [arXiv:2501.09310](https://arxiv.org/abs/2501.09310) find
  37. Lei Sheng, Shuai-Shuai Xu, and Wei Xie. 2025. Base-sql: A powerful open source text-to-sql baseline approach. arXiv preprint arXiv:2502.10739. — [arXiv:2502.10739](https://arxiv.org/abs/2502.10739) find
  38. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Comput. Surv. Just Accepted. find
  39. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  40. Hanchen Xia, Feng Jiang, Naihao Deng, Cunxiang Wang, Guojiang Zhao, Rada Mihalcea, and Yue Zhang. 2024. rˆ3:" this is my sql, are you with me?" a consensus-based multi-agent system for text-to-sql tasks. arXiv preprint arXiv:2402.14851. — [arXiv:2402.14851](https://arxiv.org/abs/2402.14851) find
  41. Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Ruijie Guo. 2025. Opensearch-sql: Enhancing text-to-sql with dynamic few-shot and consistency alignment. Preprint, arXiv:2502.14913. — [arXiv:2502.14913](https://arxiv.org/abs/2502.14913) find
  42. Bo Xu, Shufei Li, Yifei Wu, Shouang Wei, Ming Du, Hongya Wang, and Hui Song. 2024. Chain-of-program prompting with open-source large language models for text-to-sql. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  43. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436. find
  44. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv:2505.09388. arXiv preprint Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. Tabert: Pretraining for joint und…
  45. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, and 1 others. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In 2018 Conference on Empirical Methods in Nat…
  46. Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, and Hangyu Mao. 2024. Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation. arXiv preprint arXiv:2403.02951. — [arXiv:2403.02951](https://arxiv.org/abs/2403.02951) find
  47. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  48. He Zhu, Yifan Ding, Yicheng Tao, Zhiwen Ruan, Yixia Li, Wenjia Zhang, Yun Chen, and Guanhua Chen. 2025a. FANNO: Augmenting high-quality instruction data with open-sourced LLMs only. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17633–17653, Vienna, Austria. Associatio… find
  49. He Zhu, Zhiwen Ruan, Junyou Su, Xingwei He, Yun Chen, Wenjia Zhang, and Guanhua Chen. 2025b. Tag-instruct: Controlled instruction complexity enhancement through structure-based augmentation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17708–17729, Vienna, Austria. A… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-static-rules-automated-discovery/