Elamur Research⌕
☰
home / papers / 2026 / GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL

GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL

Paper · 2026 · Yanning Su, Yuhang Zhou, Yang Fang et al.

Summary

NL2GQL is the task of mapping natural-language questions to graph database queries. It has lacked an evaluation resource that is large, spans many domains, and covers more than one query dialect. Earlier datasets usually target one field, such as finance or medicine, or only Cypher/nGQL, and they rely on costly manual annotation.

The authors build GQLBench with an automated two-part pipeline:

  • Converter. It reuses established NL2SQL datasets. A hybrid rule-plus-LLM procedure turns relational databases into property graphs: join tables become edges, an LLM names and orients the edges, and redundant edge types are then merged. SQL queries are rewritten into Cypher and ISO GQL through a shared GQL AST.
  • Generator. It adds graph-native structures that relational data rarely contains, such as cycles and long paths. Queries come from templates, and an LLM writes the matching questions.

Execution-based experiments show that current LLMs find the benchmark hard. Even gemini-3-flash averages only 35.40% execution accuracy over the two dialects. GQLBench therefore exposes gaps in structural graph reasoning and in handling the newer ISO standard.

Key points

  • Coverage: GQLBench has over 20k questions per dialect. The Cypher version spans 311 graphs (140.7M entities, 15.0M relations), and the ISO GQL version spans 260 graphs (56.0M entities, 12.9M relations).
  • ISO GQL implementation: ISO GQL is executed via NebulaGraph GQL (Nebula). For gemini-3-flash and Qwen3-Max, syntax mistakes account for over 70% of Nebula failures.
  • Graph patterns: In a fine-grained analysis on 4,000 sampled questions, simple patterns such as a single node or edge reach roughly 40% average EX. Accuracy falls sharply for cycles and long paths.
  • Advanced functions: Queries without advanced functions average 28.55% EX, while subqueries collapse to 0.50% on average.
  • Models evaluated: The models include gemini-3-flash, Qwen3-Max, GPT-4o-mini, Qwen3-32B, DeepSeek-V3.2-chat, Qwen3-8B, GPT-OSS-20B and Llama-3.1-8B-Instruct. LoRA fine-tuning was also explored, and it substantially reduces syntax errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Yanning Su, Yuhang Zhou, Yang Fang, Sen Liu, Guangnan Ye and Hongfeng Chai. 2026. GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{su2026gqlbench,
  title = {{GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL}},
  author = {Yanning Su, Yuhang Zhou, Yang Fang, Sen Liu, Guangnan Ye and Hongfeng Chai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1476},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1476}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  2. Orri Erling, Alex Averbuch, Josep Larriba-Pey, Hassan Chafi, Andrey Gubichev, Arnau Prat, Minh-Duc Pham, and Peter Boncz. 2015. The ldbc social network benchmark: Interactive workload. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 619–630. find
  3. Yanlin Feng, Simone Papicchio, and Sajjadur Rahman. 2025. Cypherbench: Towards precise retrieval over full-scale modern knowledge graphs in the llm era. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8934–8958. find
  4. Aibo Guo, Xinyi Li, Guanchen Xiao, Zhen Tan, and Xiang Zhao. 2022. Spcql: A semantic parsing dataset for converting natural language into cypher. In Proceedings of the 31st ACM international conference on information & knowledge management, pages 3973– 3977. find
  5. Zirui Guo, Lianghao Xia, Yanhua Yu, Tian Ao, and Chao Huang. 2024. Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779, 2(3).
  6. Bernal J Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems, 37:59532–59569.
  7. Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From rag to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802. find
  8. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3.
  9. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  10. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, and 1 others. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763. — [arXiv:2411.07763](https://arxiv.org/abs/2411.07763) find
  11. Vladimir I Levenshtein and 1 others. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union. find
  12. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, and 1 others. 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330–4…
  13. Shunyang Li, Zhengyi Yang, Xianhang Zhang, Wenjie Zhang, and Xuemin Lin. 2021. Sql2cypher: automated data and query migration from rdbms to gdbms. In International Conference on Web Information Systems Engineering, pages 510–517. Springer. find
  14. Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, and Weining Qian. 2024a. Aligning large language models to a domain-specific graph database for nl2gql. In Proceedings of the 33rd ACM international conference on information and knowledge management, pages 1367–1377. find
  15. Yuanyuan Liang, Tingyu Xie, Gan Peng, Zihao Huang, Yunshi Lan, and Weining Qian. 2024b. Nat-nl2gql: a novel multi-agent framework for translating natural language to graph query language. arXiv preprint arXiv:2412.10434. — [arXiv:2412.10434](https://arxiv.org/abs/2412.10434) find
  16. Ilya Mazein, Adrien Rougny, Alexander Mazein, Ron Henkel, Lea Gütebier, Lea Michaelis, Marek Ostaszewski, Reinhard Schneider, Venkata Satagopam, Lars Juhl Jensen, and 1 others. 2024. Graph databases in systems biology: a systematic review. Briefings in Bioinformatics, 25(6):bbae561. find
  17. Lunyiu Nie, Shulin Cao, Jiaxin Shi, Jiuding Sun, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. 2022. Graphq ir: Unifying the semantic parsing of graph query languages with one intermediate representation. In Proceedings of the 2022 conference on empirical methods in natural language processing, page… find
  18. Makbule Gulcin Ozsoy, Leila Messallem, Jon Besga, and Gianandrea Minneci. 2025. Text2cypher: Bridging natural language and graph databases. In Proceedings of the Workshop on Generative AI and Knowledge Graphs (GenAIK), pages 100–108. find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  20. Liana Stanescu. 2021. A comparison between a relational and a graph database in the context of a recommendation system. In FedCSIS (Position Papers), pages 133–139. find
  21. Shikhar Vashishth, Prince Jain, and Partha Talukdar. 2018. Cesi: Canonicalizing open knowledge bases using embeddings and side information. In Proceedings of the 2018 World Wide Web Conference, pages 1317–1327. find
  22. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, and 1 others. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empir…
  23. Bowen Zhang and Harold Soh. 2024. Extract, define, canonicalize: An llm-based framework for knowledge graph construction. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 9820–9836.
  24. Ziyu Zhao, Wei Liu, Tim French, and Michael Stewart. 2023. Rel2graph: Automated mapping from relational databases to a unified property knowledge graph. arXiv preprint arXiv:2310.01080. — [arXiv:2310.01080](https://arxiv.org/abs/2310.01080) find
  25. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 400–410.
  26. Zijie Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang. 2025. Syn-thet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task. In Proceedings of the 31st international conference on computational linguistics, pages 672–692. find
  27. Yuhang Zhou, Yu He, Siyu Tian, Yuchen Ni, Zhangyue Yin, Xiang Liu, Chuanjun Ji, Sen Liu, Xipeng Qiu, Guangnan Ye, and 1 others. 2024. r3-nl2gql: A model coordination and knowledge graph alignment approach for nl2gql. In Findings of the association for computational linguistics: EMNLP 2024, pages 136… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2026-gqlbench-large-scale-cross-domain/