Elamur Research⌕
☰
home / papers / 2025 / Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic

Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic

Summary

Conversational text-to-SQL turns a series of linked user questions into database queries. Research on it has almost entirely used English, with some Chinese, and Arabic had no resource for this setting. The authors build Ar-SParC, an Arabic version of the English SParC benchmark. GPT-4.5-turbo produced draft translations of the questions, and native-speaker translators then post-edited them and checked that each SQL query still matched its Arabic question. They then run 40 prompting experiments with GPT-3.5-turbo and GPT-4.5-turbo, covering four ways of representing the question and schema and six strategies for choosing in-context examples. Their main methodological contribution is the GAT Corrector, which replaces the earlier GAT verifier. It is a GPT-3.5-Turbo model fine-tuned on 500 samples the team wrote, and it repairs outputs directly. It improved results in every one of the 40 runs and is reported to be faster and cheaper. The work gives Arabic its first benchmark for multi-turn database querying and a practical post-processing step that can raise LLM accuracy.

Key points

  • Ar-SParC size: 3,450 question sequences (about three questions each), 10,225 questions in total, spread over 160 databases and 116 domains.
  • Splits: train has 3,029 sequences, 9,025 questions and 140 DBs; dev has 421 sequences, 1,200 questions and 20 DBs. The two splits share no databases or domains. SParC's hidden test set was not translated.
  • Metrics: execution accuracy (EX) is scored per question, and interaction accuracy (IX) is scored per whole conversation.
  • GAT Corrector gains: in zero-shot runs, +1.9% EX and +1.9% IX on average; with in-context examples, +1.72% EX and +0.92% IX.
  • Ablation on a non-SQL task: the GAT Corrector made 3 errors in 100 cases, against 33 for the GAT verifier. The authors read this as evidence that the corrector suits Arabic better.

Interesting ideas

  • I1. Swapping a judge that only accepts or rejects an output for a small fine-tuned model that rewrites it improved every configuration and also saved time and cost. — For Mova (learn): a cheap tuned repairer could fix flagged parser or frame errors directly instead of only rejecting them, with its edits logged so each fix stays explainable.
  • I2. The dataset was built in three steps: an LLM drafts the translation, native experts post-edit it, and a final check confirms the formal target (SQL) still fits the translated text. — For Mova (learn): silver data could follow the same route, with LLM drafts reviewed by people and then verified against the UD tree or frame so that labels and text do not drift apart.
  • I3. Scoring both each turn (EX) and the whole conversation (IX) exposes how an early mistake spreads into later answers. — For Mova (answer): fable QA and multi-step word problems could report a whole-sequence score next to per-question accuracy, making error propagation through the world model visible.
  • I4. The train/dev split keeps databases and domains completely disjoint, so the scores measure transfer to unseen domains rather than memorised schemas. — For Mova (learn): evaluation could hold out entire genres or text sources, acting as a negative control against overfitting to familiar vocabulary.
  • I5. The authors compared corrector and verifier on a task outside the corrector's home task to check whether the advantage depends on language rather than task. — For Mova (learn): any repair or teacher component could be stress-tested on a different task before being trusted, to catch fixes that only work in one narrow setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.20677 · arXiv 2511.20677

@misc{almohaimeed2025prompt,
  title = {{Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic}},
  author = {Almohaimeed, Saleh and Alsofyani, May and Almohaimeed, Saad and Ghanim, Mansour Al and Wang, Liqiang},
  year = {2025},
  doi = {10.48550/arxiv.2511.20677},
  eprint = {2511.20677},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.20677}
}

References

43 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Yu et al., “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” *arXiv preprint arXiv:1809.08887*, 2018. ↗ our card
  2. P. Price, “Evaluation of spoken language systems: The ATIS domain,” in *Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990*, 1990. find
  3. J. Li et al., “Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs,” *Advances in Neural Information Processing Systems*, vol. 36, 2024. find our card
  4. OpenAI, “Introducing ChatGPT with GPT-3.5-turbo,” [Online]. Available: https://openai.com/blog/chatgpt-3-5-turbo/. [Accessed: 01-Jan-2024]. ↗
  5. OpenAI, “Models: GPT-4 and GPT-4 Turbo,” [Online]. Available: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. [Accessed: 01-Jan-2024]. ↗
  6. S. Almohaimeed, S. Almohaimeed, M. Al Ghanim, and L. Wang, “Ar-Spider: Text-to-SQL in Arabic,” in *Proc. 39th ACM/SIGAPP Symp. on Applied Computing*, 2024, pp. 1024–1030. find our card
  7. S. Almohaimeed, S. Almohaimeed, and L. Wang, “GAT-SQL: An advanced prompt engineering approach for effective text-to-SQL interactions,” in *2024 IEEE Congress on Evolutionary Computation (CEC)*, 2024, pp. 1–10. find
  8. S. Almohaimeed, S. Liu, M. Alsofyani, S. Almohaimeed, and L. Wang, “SIGMA: A dataset for text-to-code semantic parsing with statistical analysis,” in *2023 Int. Conf. on Machine Learning and Applications (ICMLA)*, 2023, pp. 851–857. find
  9. D. Bakshandaeva, O. Somov, E. Dmitrieva, V. Davydova, and E. Tutubalina, “PAUQ: Text-to-SQL in Russian,” in *Findings of the Assoc. for Computational Linguistics: EMNLP 2022*, 2022, pp. 2355–2376. find
  10. Q. Min, Y. Shi, and Y. Zhang, “A pilot study for Chinese SQL semantic parsing,” *arXiv preprint arXiv:1909.13293*, 2019. ↗
  11. A. T. Nguyen, M. H. Dao, and D. Q. Nguyen, “A pilot study of text-to-SQL semantic parsing for Vietnamese,” *arXiv preprint arXiv:2010.01891*, 2020. ↗
  12. M. A. José and F. G. Cozman, “mRAT-SQL+ GAP: A Portuguese text-to-SQL transformer,” in *10th Brazilian Conf. on Intelligent Systems (BRACIS)*, 2021, pp. 511–525. find
  13. M. Artetxe and H. Schwenk, “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond,” in *Trans. of the Assoc. for Computational Linguistics*, vol. 7, 2019, pp. 597–610. find our card
  14. L. Dou et al., “MultiSpider: Towards benchmarking multilingual text-to-SQL semantic parsing,” in *Proc. of the AAAI Conf. on Artificial Intelligence*, vol. 37, no. 11, 2023, pp. 12745–12753. find
  15. R. Cao et al., “LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,” *arXiv preprint arXiv:2106.01093*, 2021. ↗
  16. B. Hui et al., “S 2 SQL: Injecting syntax to question-schema interaction graph encoder for text-to-SQL parsers,” *arXiv preprint arXiv:2203.06958*, 2022. ↗
  17. A. Conneau et al., “Unsupervised cross-lingual representation learning at scale,” *CoRR*, vol. abs/1911.02116, 2019. [Online]. Available: http://arxiv.org/abs/1911.02116. ↗
  18. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” *CoRR*, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805. ↗
  19. B. Qin et al., “A survey on text-to-SQL parsing: Concepts, methods, and future directions,” *arXiv preprint arXiv:2208.13629*, 2022. ↗
  20. Z. Zhang et al., “Recent advances and challenges in task-oriented dialog systems,” *Science China Technological Sciences*, vol. 63, no. 10, 2020, pp. 2011–2027. find
  21. E. Dehaerne et al., “Code generation using machine learning: A systematic review,” *IEEE Access*, 2022. find
  22. V. Zhong, C. Xiong, and R. Socher, “Seq2SQL: Generating structured queries from natural language using reinforcement learning,” *arXiv preprint arXiv:1709.00103*, 2017. ↗
  23. T. Yu et al., “CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases,” *arXiv preprint arXiv:1909.05378*, 2019. ↗ our card
  24. T. Yu et al., “SParC: Cross-domain semantic parsing in context,” *arXiv preprint arXiv:1906.02285*, 2019. ↗ our card
  25. D. Q. Nguyen and A. T. Nguyen, “PhoBERT: Pre-trained language models for Vietnamese,” *arXiv preprint arXiv:2003.00744*, 2020. ↗ our card
  26. Y. Tang et al., “Multilingual translation with extensible multilingual pretraining and finetuning,” *arXiv preprint arXiv:2008.00401*, 2020. ↗
  27. T. B. Brown et al., “Language models are few-shot learners,” *Advances in Neural Information Processing Systems*, vol. 33, 2020. find
  28. J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in *Proc. of the 2014 Conf. on Empirical Methods in NLP (EMNLP)*, 2014, pp. 1532–1543. find
  29. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “ELECTRA: Pre-training text encoders as discriminators rather than generators,” *arXiv preprint arXiv:2003.10555*, 2020. ↗
  30. G. Lample and A. Conneau, “Cross-lingual language model pretraining,” in *Advances in Neural Information Processing Systems (NeurIPS)*, 2019. find
  31. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in *Proc. of the 2019 Conf. on Empirical Methods in NLP*, 2019, pp. 3982–3992. find
  32. M. Pourreza and D. Rafiei, “Din-SQL: Decomposed in-context learning of text-to-SQL with self-correction,” *Advances in Neural Information Processing Systems*, vol. 36, 2024. find our card
  33. L. Nan, Y. Zhao, W. Zou, N. Ri, J. Tae, E. Zhang, A. Cohan, and D. Radev, “Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies,” in *Findings of the Association for Computational Linguistics: EMNLP 2023*, 2023, pp. 14935–14956. find
  34. S. Chang and E. Fosler-Lussier, “How to prompt LLMs for text-to-SQL: A study in zero-shot, single-domain, and cross-domain settings,” in *NeurIPS 2023 Second Table Representation Learning Workshop*, 2023. find
  35. OpenAI, “SQL Translate,” *Online. Available: https://platform.openai.com/examples/default-sql-translate*, [Accessed: 24-Jul-2023]. ↗
  36. I. Higgins, N. Sonnerat, L. Matthey, A. Pal, C. P. Burgess, M. Bosnjak, M. Shanahan, M. Botvinick, D. Hassabis, and A. Lerchner, “SCAN: Learning hierarchical compositional visual concepts,” *arXiv preprint arXiv:1707.03389*, 2017. ↗
  37. X. V. Lin, C. Wang, L. Zettlemoyer, and M. D. Ernst, “NL2Bash: A corpus and semantic parser for natural language interface to the Linux operating system,” in *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, 2018. find
  38. J. M. Zelle and R. J. Mooney, “Learning to parse database queries using inductive logic programming,” in *Proceedings of the National Conference on Artificial Intelligence*, 1996, pp. 1050–1055. find
  39. J. Guo, Z. Si, Y. Wang, Q. Liu, M. Fan, J.-G. Lou, Z. Yang, and T. Liu, “Chase: A large-scale and pragmatic Chinese dataset for cross-database context-dependent text-to-SQL,” in *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, 2021, pp. 2316–2331. find
  40. S. Huang, L. Wang, Z. Li, Z. Liu, C. Dou, F. Yan, X. Xiao, H. Wu, and M. Zhang, “SeSQL: A high-quality large-scale session-level Chinese text-to-SQL dataset,” in *CCF International Conference on Natural Language Processing and Chinese Computing*, Springer, 2023, pp. 537– 550. find
  41. D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, “Text-to-SQL empowered by large language models: A benchmark evaluation,” *Proceedings of the VLDB Endowment*, vol. 17, no. 5, 2024, pp. 1132– 1145. find our card
  42. J. Liu, D. Shen, Y. Zhang, W. B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for GPT-3?,” in *Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures*, 2022, pp. 100–114. find our card
  43. All-mpnet-base-v2, “All-mpnet-base-v2: A sentence embedding model,” *Online. Available: [URL of the model description]*, 2023. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almohaimeed-2025-prompt-engineering-techniques-context-dependent/