Elamur Research⌕
☰
home / papers / 2026 / Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation

Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation

Paper · 2026 · Marko Sterbentz, Michael Glass, Nhan H Pham et al.

Summary

The paper targets a known weakness of GRPO-based reinforcement learning for text-to-SQL. Because GRPO trains only on samples from the model's current policy, it tends to reinforce reasoning the model already has instead of teaching new strategies. The authors bring mixed-policy GRPO, previously studied mainly for math reasoning, to text-to-SQL for the first time. In this setup, externally generated off-policy reasoning traces are mixed into training alongside the model's own rollouts, and rewards come from SQL execution. They compare several ways of building these traces from existing question–SQL data, starting from SynSQL. They also propose Iterative Error Correction (IEC). In IEC, a model first writes error-avoiding hints using the gold query. A separate generation step then sees only those hints, never the gold SQL. The hints are revised after each failed attempt until the query executes correctly. Mixed-policy training beat both the base model and on-policy GRPO on Spider and BIRD, with the biggest improvements on the harder BIRD benchmark. The study also finds that which trace source works best depends on the training algorithm, so data generation and optimization should be designed together.

Key points

  • Average execution accuracy across Spider Dev, Spider Test and BIRD Dev:
    • base model: 68.63%
    • on-policy GRPO: 69.25%
    • best LUFFY run (SynSQL CoT + Direct Sampling): 73.36%, roughly +4.7 points over the base model and +4.1 over on-policy GRPO
  • The largest gains appear on BIRD Dev. LUFFY with the original SynSQL CoT reaches 56.06%, compared with 48.70% for the base model and 51.49% for on-policy GRPO.
  • In the simplified mixed-policy method, replacing most SynSQL CoT traces with IEC traces (91% IEC) gives the best result for that method: 71.50% average and 55.02% on BIRD Dev.
  • Every LUFFY-trained variant outperformed both baselines on all three benchmarks. However, under LUFFY, IEC substitution did not help, and the original or Direct Sampling traces worked better.
  • Training only on the examples where Direct Sampling produced a valid trace still beat on-policy GRPO by 1.43 points, while using 40% less data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semantic parsing & meaning representation · 193 worksCode & programming languages · 168 works

Cite

Cite as Marko Sterbentz, Michael Glass, Nhan H Pham, Dharmashankar Subramanian and Kristian J Hammond. 2026. Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation. Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.surgellm-1.20

@inproceedings{sterbentz2026mixed,
  title = {{Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation}},
  author = {Marko Sterbentz, Michael Glass, Nhan H Pham, Dharmashankar Subramanian and Kristian J Hammond},
  booktitle = {Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.surgellm-1.20},
  url = {https://doi.org/10.18653/v1/2026.surgellm-1.20}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed M Adly, Mostafa Samy, and Amr Fawzy. 2025. Gazal-r1: Achieving state-of-the-art medical reasoning with parameter-efficient two-stage training. arXiv preprint arXiv:2506.21594. — [arXiv:2506.21594](https://arxiv.org/abs/2506.21594) find
  2. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. Gepa: Reflective prompt evoluti… find
  3. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  4. Peixian Ma, Xialie Zhuang, Chengjin Xu, Xuhui Jiang, Ran Chen, and Jian Guo. 2025. Sql-r1: Training natural language to sql reasoning model by reinforcement learning. arXiv preprint arXiv:2504.08600. — [arXiv:2504.08600](https://arxiv.org/abs/2504.08600) find
  5. Mingzhe Du, Luu Anh Tuan, Yue Liu, Yuhao Qing, Dong Huang, Xinyi He, Qian Liu, Zejun Ma, and See-kiong Ng. 2025. Afterburner: Reinforcement learning facilitates self-improving code efficiency optimization. arXiv preprint arXiv:2505.23387. — [arXiv:2505.23387](https://arxiv.org/abs/2505.23387) find
  6. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: iterative refinement with… find
  7. Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, Xiaoqian Jiao, Chun Yong Chong, Shan Gao, and Michael Lyu. 2025. The prompt alchemist: Automated llm-tailored prompt optimization for test case generation. arXiv preprint arXiv:2501.01329. — [arXiv:2501.01329](https://arxiv.org/abs/2501.01329) find
  8. Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx. 2025. Adaptive guidance accelerates reinforcement learning of reasoning models. arXiv preprint arXiv:2506.13923. — [arXiv:2506.13923](https://arxiv.org/abs/2506.13923) find
  9. Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin, Hong Chi, James Cao, Peng Chen, and Ming Zhou. 2018. Question generation from SQL queries improves neural semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1597–1607, Brussels, Belgium… find
  10. Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, and Sercan "O. Arik. 2025. Reasoning-sql: Reinforcement learning with sql tailored partial rewards for reasoning-enhanced text-to-sql. Preprint, arXiv:2503.23157. — [arXiv:2503.23157](https://arxiv.org/abs/2503.23157) find
  11. Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jian Yang, and Xiaobin Tan. 2025. SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452, Vienna, Austria. Association for Computa… find
  12. Mingqian He, Yongliang Shen, Wenqi Zhang, Qiuying Peng, Jun Wang, and Weiming Lu. 2025. Star-sql: Self-taught reasoner for text-to-sql. arXiv preprint arXiv:2502.13550.
  13. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. Omnisql: Synthesizing high-quality text-to-sql data at scale. Proc. VLDB Endow., 18(11):4695–4709. find
  14. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Ma Chenhao, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can llm already serve as a database interface? a big bench for large-scale database groun…
  15. Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968.
  16. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  17. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc.
  18. Marko Sterbentz, Kevin Cushing, Cameron Barrie, and Kristian J Hammond. 2026. Ringsql: Generating synthetic data with schema-independent templates for text-to-sql reasoning models. arXiv preprint arXiv:2601.05451. — [arXiv:2601.05451](https://arxiv.org/abs/2601.05451) find
  19. Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. 2021. Learning to synthesize data for semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2760–2766, Online. Associatio… find
  20. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
  21. Nathaniel Weir, Prasetya Utama, Alex Galakatos, Andrew Crotty, Amir Ilkhechi, Shekar Ramaswamy, Rohin Bhushan, Nadja Geisler, Benjamin Hättasch, Steffen Eger, Ugur Cetintemel, and Carsten Binnig. 2020. Dbpal: A fully pluggable nl2sql training pipeline. In Proceedings of the 2020 ACM SIGMOD Internati… find
  22. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. 2024. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 1126. find
  23. Kun Wu, Lijie Wang, Zhenghua Li, Ao Zhang, Xinyan Xiao, Hua Wu, Min Zhang, and Haifeng Wang. 2021. Data augmentation with hierarchical SQL-to-question generation for cross-domain text-to-SQL parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 897… find
  24. Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I Wang, Wen-tau Yih, and Ziyu Yao. 2023. Learning to simulate natural language feedback for interactive semantic parsing. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3149–3170. find
  25. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945.
  26. Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. 2024. Synthesizing text-to-sql data from weak and strong llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7864–7875. find
  27. Zhewei Yao, Guoheng Sun, Lukasz Borchmann, Gaurav Nuti, Zheyu Shen, Minghang Deng, Bohan Zhai, Hao Zhang, Ang Li, and Yuxiong He. 2026. Arctic-text2sql-r1: Simple rewards, strong reasoning in text-to-sql. Preprint, arXiv:2505.20315.
  28. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845. find
  29. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018…
  30. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the thirteenth national conference on Artificial intelligence-Volume 2, pages 1050–1055.
  31. Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. 2025a. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXiv:2502.19655. find
  32. Xiaotian Zhang, Yuan Wang, Zhaopeng Feng, Ruizhe Chen, Zhijie Zhou, Yan Zhang, Hongxia Xu, Jian Wu, and Zuozhu Liu. 2025b. Med-u1: Incentivizing unified medical reasoning in llms via large-scale reinforcement learning. arXiv preprint arXiv:2506.12307. — [arXiv:2506.12307](https://arxiv.org/abs/2506.12307) find
  33. Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. 2025. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912. find
  34. Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869– 6882, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sterbentz-2026-mixed-policy-grpo-text-sql/