Elamur Research⌕
☰
home / papers / 2026 / ROSE: An Intent-Centered Evaluation Metric for NL2SQL

ROSE: An Intent-Centered Evaluation Metric for NL2SQL

Paper · 2026 · Wenqi Pei, Shizheng Hou, Boyan Li et al.

Summary

The paper argues that Execution Accuracy (EX), the standard NL2SQL metric, gives misleading scores. EX only checks whether a predicted query returns the same result as one reference SQL. It therefore fails in three situations: correct queries with different output formatting or column order, valid alternative readings of ambiguous questions, and wrong gold SQL in benchmarks such as BIRD.

The authors propose ROSE (ReasOning ScorE), which judges whether a prediction fulfils the user's intent rather than whether it matches the reference. It uses two reasoning-model stages:

  • SQL Prover evaluates the prediction using only the question and database, without seeing the gold SQL.
  • Adversarial Refuter then uses the gold SQL as evidence to try to overturn that verdict. It can also label items as ambiguous or as having faulty ground truth.

To validate the metric, they built ROSE-VEC, a set of samples with expert consensus labels. ROSE agrees with the experts more closely than any other metric tested. Re-scoring 19 NL2SQL systems shows that the gap between intent-level correctness and reference matching grows as models improve. This matters because benchmark rankings may increasingly reward imitation of reference style rather than genuine understanding.

Key points

  • ROSE-VEC contains 585 expert-consensus samples, kept only where two annotators fully agreed. It is released together with ROSE (github.com/CedricPei/ROSE).
  • Compared with the next-best metric, ROSE gains roughly 24% in Cohen's Kappa and about 14% in accuracy.
  • Erroneous gold SQL (GoldX) causes about 45% of EX–ROSE disagreements, and ambiguous questions (AmbQ) about 10%. Earlier work (NL2SQL-BUGs) found a 6.91% error rate in BIRD Dev gold SQL.
  • The re-evaluation of 19 methods yields four findings:
    • Base-model strength matters more than system engineering.
    • The gap between ROSE and EX widens as models get stronger.
    • Most of that gap comes from benchmark flaws.
    • Fine-tuning, mostly on BIRD Train, shrinks the gap by teaching the dataset's SQL style. For prompting methods, the gap on Simple and Moderate questions is about four times larger.
  • The authors list three limitations: ROSE depends on the backbone LLM, costs more and is slower than EX, and consensus filtering may bias ROSE-VEC toward easier cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Wenqi Pei, Shizheng Hou, Boyan Li, Chen Han, Zhichao Shi and Yuyu Luo. 2026. ROSE: An Intent-Centered Evaluation Metric for NL2SQL. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.257

@inproceedings{pei2026rose,
  title = {{ROSE: An Intent-Centered Evaluation Metric for NL2SQL}},
  author = {Wenqi Pei, Shizheng Hou, Boyan Li, Chen Han, Zhichao Shi and Yuyu Luo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.257},
  url = {https://doi.org/10.18653/v1/2026.acl-long.257}
}

References

42 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Serge Abiteboul, Richard Hull, and Victor Vianu. 1995. Foundations of Databases. find
  2. Arize AI. 2024. Text to SQL: Evaluating SQL Generation with LLM as a Judge. find
  3. Benjamin G. Ascoli, Yasoda Sai Ram Kandikonda, and Jinho D. Choi. 2024. ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models. Preprint, arXiv:2407.07313. — [arXiv:2407.07313](https://arxiv.org/abs/2407.07313) find
  4. Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, Wei Chen, and Xiang Bai. 2024. RSL-SQL: Robust Schema Linking in Text-to-SQL Generation. arXiv preprint arXiv:2411.00073. — [arXiv:2411.00073](https://arxiv.org/abs/2411.00073) find
  5. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. find
  6. Defog.ai. 2023. Introducing SQL-Eval: A Framework for Evaluating LLM-Generated SQL. find
  7. Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. Preprint, arXiv:2307.07306. — [arXiv:2307.07306](https://arxiv.org/abs/2307.07306) find
  8. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment, 17(5):1132–1145.
  9. Google DeepMind. 2025. Gemini 2.5: Our most intelligent AI model. find
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, and Others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633638. \#\# Acknowledgements Yang He, Pinhan Zhao, Xinyu Wang, and Yuepeng Wang.… find
  11. This paper was supported by the NSF of China (62402409); Youth S&T Talent Support Programme of Guangdong Provincial Association for Science and Technology (SKXRC2025461); the Shizheng Hou, Wenqi Pei, Nuo Chen, Quang-Trung Ta, Peng Lu, and Beng Chin Ooi. 2026. NL2SQL-Bench: A Modular Benchmarking Fra… find
  12. Heegyu Kim, Taeyang Jeon, Seunghwan Choi, Seungtaek Choi, and Hyunsouk Cho. 2025. FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics. Association for C… find
  13. Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. 2026. DeepEye-SQL: a software-engineering-inspired text-to-sql framework. Proc. ACM Manag. Data, 4(3). find
  14. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024a. The Dawn of Natural Language to SQL: Are We Fully Ready? Proceedings of the VLDB Endowment, 17(11):3318–3331. find
  15. Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025a. Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. In Forty-Second International Conference on Machine Learning, ICML. find
  16. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025b. OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale. Proceedings of the VLDB Endowment, 18(11):46954709. find
  17. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067–13075.
  18. Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024b. CodeS: Towards Building Open-source Language Models for Text-to-SQL. Proc. ACM Manag. Data, 2(3). find
  19. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023b. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grou…
  20. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025a. A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going? IEEE Transactions on Knowledge and Data Engineering, 37(10):5735–5754. find
  21. Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. 2025b. NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation. Preprint, arXiv:2503.11984. — [arXiv:2503.11984](https://arxiv.org/abs/2503.11984) find
  22. Yuyu Luo, Guoliang Li, Ju Fan, Chengliang Chai, and Nan Tang. 2025. Natural Language to SQL: State of the Art and Open Problems. Proceedings of the VLDB Endowment, 18(12):5466–5471. find
  23. Yuyu Luo, Xuedi Qin, Nan Tang, and Guoliang Li. 2018a. DeepEye: Towards Automatic Data Visualization. In ICDE, pages 101–112. IEEE Computer Society. find
  24. Yuyu Luo, Xuedi Qin, Nan Tang, Guoliang Li, and Xinran Wang. 2018b. DeepEye: Creating Good Data Visualizations by Keyword Search. In SIGMOD Conference, pages 1733–1736. ACM. find
  25. Brian W. Matthews. 1975. Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451. find
  26. OpenAI. 2025. Introducing o3 and o4-mini. find
  27. Wenqi Pei, Hailing Xu, Henry Hengyuan Zhao, Shizheng Hou, Chen Han, Zining Zhang, Luo Pingyi, and Bingsheng He. 2025. Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Models. In Proceedings of the 14th International Joint Conference on Natural Lan… find
  28. Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation. In Findings of the Association for Computational Linguistics, pages 5456–5471. Association for Co… find
  29. Irina Saparina and Mirella Lapata. 2024. AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. Preprint, arXiv:2406.19073. — [arXiv:2406.19073](https://arxiv.org/abs/2406.19073) find
  30. Lei Sheng and Shuai-Shuai Xu. 2025. CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning. Preprint, arXiv:2505.13271. find
  31. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. CHESS: Contextual Harnessing for Efficient SQL Synthesis. Preprint, arXiv:2405.16755. — [arXiv:2405.16755](https://arxiv.org/abs/2405.16755) find
  32. C. J. van Rijsbergen. 1979. Information Retrieval, 2 edition. Butterworths, London. find
  33. Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Ruijie Guo. 2025. OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment. arXiv preprint arXiv:2502.14913. — [arXiv:2502.14913](https://arxiv.org/abs/2502.14913) find
  34. Tao Yu, Rui Zhang, Michihiro Yasunaga, Kai Tan, Xi Victoria Lin, Suyi Li, James Jiang, Zhou Li, Shanelle Yao, Yi Chern Ma, and Others. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empiri…
  35. Yi Zhan, Longjie Cui, Han Weng, Guifeng Wang, Yu Tian, Boyi Liu, Yingxiang Yang, Xiaoming Yin, Jiajun Xie, and Yang Sun. 2025. Towards Database-Free Text-to-SQL Evaluation: A Graph-Based Metric for Functional Correctness. In Proceedings of the 31st International Conference on Computational Linguisti… find
  36. Feng Zhang, Jidong Zhai, Xipeng Shen, Dalin Wang, Zheng Chen, Onur Mutlu, Wenguang Chen, and Xiaoyong Du. 2020. TADOC: Text analytics directly on compression. The VLDB Journal, 30(2):163188. find
  37. Yuxin Zhang, Meihao Fan, Ju Fan, Mingyang Yi, Yuyu Luo, Jian Tan, and Guoliang Li. 2025. Reward-SQL: Boosting Text-to-SQL via Stepwise Reasoning and Process-Supervised Rewards. Preprint, arXiv:2505.04671. — [arXiv:2505.04671](https://arxiv.org/abs/2505.04671) find
  38. Fuheng Zhao, Jiayue Chen, Lawrence Lim, Ishtiyaque Ahmad, Divyakant Agrawal, and Amr El Abbadi. 2025a. LLM-SQL-Solver: Can LLMs Determine SQL Equivalence? Preprint, arXiv:2312.10321. — [arXiv:2312.10321](https://arxiv.org/abs/2312.10321) find
  39. Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divy Agrawal, and Amr El Abbadi. 2025b. Sphinteract: Resolving Ambiguities in NL2SQL through User Interaction. Proceedings of the VLDB Endowment, 18(4). find
  40. Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. Preprint, arXiv:2010.02840. find
  41. Yizhang Zhu, Runzhi Jiang, Boyan Li, Nan Tang, and Yuyu Luo. 2025. EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing. Preprint, arXiv:2503.22402. — [arXiv:2503.22402](https://arxiv.org/abs/2503.22402) find
  42. Yizhang Zhu, Liangwei Wang, Chenyu Yang, Xiaotian Lin, Boyan Li, Wei Zhou, Xinyu Liu, Zhangyang Peng, Tianqi Luo, Yu Li, Chengliang Chai, Chong Chen, Shimin Di, Ju Fan, Ji Sun, Nan Tang, Fugee Tsung, Jiannan Wang, Chenglin Wu, and 6 others. 2026. A Survey of Data Agents: Emerging Paradigm or Oversta… — [arXiv:2510.23587](https://arxiv.org/abs/2510.23587) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pei-2026-rose-intent-centered-evaluation-metric/