Elamur Research⌕
☰
home / papers / 2024 / AraSpider: Democratizing Arabic-to-SQL

AraSpider: Democratizing Arabic-to-SQL

Summary

Text-to-SQL research has relied mostly on English benchmarks, which leaves Arabic speakers without a standard testbed for querying databases in natural language. The authors build AraSpider, an Arabic counterpart of the cross-domain Spider benchmark. To create it, ChatGPT 3.5 first translated the questions, with each prompt carrying the relevant database schema as context. Two native Arabic-speaking authors then post-edited the output and adapted it to Arabic cultural usage.

To choose a translation approach that others can reuse, they compared four multilingual translation models on a deliberately difficult set of 200 Spider questions. They then ran two strong Spider systems, ChatGPT 3.5 and SQLCoder, on the Arabic questions. Back translation helped both systems noticeably. ChatGPT 3.5 gave the best translations, and SQLCoder was stronger at generating SQL.

The work also offers a way to do non-English-to-SQL without fine-tuning on non-English questions. Released scripts let others reproduce the results or port Spider to further languages.

Key points

  • AraSpider is described as the first Arabic version of Spider. Spider covers 200 multi-table databases across 138 domains, with 10,181 questions and 5,693 unique SQL queries.
  • Translation was drafted by ChatGPT 3.5 with the schema in the prompt. The prompt said to leave quoted text untranslated and to keep outputs in question form.
  • Two native Arabic-speaking authors with IELTS ≥ 7.0 (C1) post-edited the drafts. They focused on cultural nuance rather than literal rendering.
  • The translation evaluation covered four multilingual models on 200 samples. The samples included quoted phrases, questions of 5–20 words, and both imperative and interrogative phrasings.
  • Back translation improved both ChatGPT 3.5 and SQLCoder on Arabic text-to-SQL. Code is at github.com/ahmedheakl/AraSpider, and future plans include translating WikiSQL and fine-tuning SQLCoder for Arabic.

Interesting ideas

  • I1. The translator is given the database schema alongside each question, so ambiguous words get resolved against the actual tables and columns rather than in isolation — For Mova (back): When rendering a meaning graph into another language, the generator could receive the relevant slice of the world of the text, so that word choice is grounded in known entities rather than surface strings.
  • I2. A language-specific task is handled by converting the input back into the language the downstream model knows best, which avoids any fine-tuning on the new language — For Mova (answer): Non-English questions about a fable or test could be pivoted into English before reaching the English-only readers, with the pivot step logged so the answer stays explainable.
  • I3. An LLM produces the first draft and qualified native speakers correct it, which is cheaper than full human translation and more reliable than raw machine output — For Mova (learn): This fits the LLM teacher tiers, where an LLM drafts silver annotations and a human or stronger witness edits only the disputed cases.
  • I4. The evaluation subset is built around known failure modes (quoted spans, varied lengths, commands vs. questions) instead of being sampled uniformly — For Mova (learn): Evaluation sets for the parser and readers could be stratified by targeted phenomena, so regressions on hard constructions are not hidden by easy majority cases.
  • I5. Quoted material is treated as untouchable during translation, preserving literal values that must match database content — For Mova (tokens): The tokenizer could mark quoted spans as protected literal units, so later stages keep names and values intact instead of lemmatizing or splitting them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.07448 · arXiv 2402.07448

@misc{mohamed2024araspider,
  title = {{AraSpider: Democratizing Arabic-to-SQL}},
  author = {Mohamed;, Ahmed Heakl; Youssef and Zaky, Ahmed B.},
  year = {2024},
  doi = {10.48550/arxiv.2402.07448},
  eprint = {2402.07448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.07448}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., ... & Radev, D. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. arXiv preprint arXiv:1809.08887. ↗ our card
  2. José, M. A., & Cozman, F. G. (2021). mRAT-SQL+ GAP: a Portuguese text-to-SQL transformer. In Intelligent Systems: 10th Brazilian Conference, BRACIS 2021, Virtual Event, November 29– December 3, 2021, Proceedings, Part II 10 (pp. 511-525). Springer International Publishing. find
  3. Jose, M. A., & Cozman, F. G. (2023). A multilingual translator to SQL with database schema pruning to improve self-attention. International Journal of Information Technology, 1-9. find
  4. Walter, S., Unger, C., Cimiano, P., Bär, D.: Evaluation of a layered approach to question answering over linked data. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 7650 LNCS, 362–374 (2012). https://doi.org/10.1007/978-3-642-35173-0-25 ↗
  5. Blunschi, L., Jossen, C., Kossmann, D., Mori, M., Stockinger, K.: SODA: Generating SQL for business users. Proc. VLDB Endow. 5, 932–943 (2012). https://doi.org/10.14778/2336664.2336667 ↗
  6. Li, F., Jagadish, H. V: Constructing an interactive natural language interface for relational databases. Proc. VLDB Endow. 8, 73–84 (2014). https://doi.org/10.14778/2735461.2735468 ↗
  7. Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Pengsheng Huang. 2021. Towards robustness of text-to-sql models against synonym substitution. arXiv preprint arXiv:2106.01065. ↗
  8. Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411, Online. Association for Computational Linguistics. find
  9. Song, D., Schilder, F., Smiley, C., Brew, C., Zielund, T., Bretz, H., Martin, R., Dale, C., Duprey, J., Miller, T., Harrison, J.: TR discover: A natural language interface for querying and analyzing interlinked datasets. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 9367, 21–37 (2015). https://doi.org/10.1007/978-3-319-25010-6 2 ↗
  10. Saha, D., Floratou, A., Sankaranarayanan, K., Minhas, U.F., Mittal, A.R., Özcan, F.: ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores. Proc. VLDB Endow. 9, 1209–1220 (2016). https://doi.org/10.14778/2994509.2994536 ↗
  11. Sen, J., Lei, C., Quamar, A., Özcan, F., Efthymiou, V., Dalmia, A., Stager, G., Mittal, A., Saha, D., Sankaranarayanan, K.: ATHENA++: natural language querying for complex nested SQL queries. Proc. VLDB Endow. 13, 2747–2759 (2020). https://doi.org/10.14778/3407790.3407858 ↗
  12. Baik, C., Arbor, A., Arbor, A., Arbor, A., Jagadish, H. V: Constructing Expressive Relational Queries with Dual-Specification Synthesis. 10th Annu. Conf. Innov. Data Syst. Res. (CIDR ‘20). (2020). find
  13. Dou, L., Gao, Y., Pan, M., Wang, D., Che, W., Zhan, D., & Lou, J.-G. (2023). MultiSpider: Towards benchmarking multilingual text-to-SQL semantic parsing. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 12745–12753. https://doi.org/10.1609/aaai.v37i11.26499 ↗
  14. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. find
  15. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, Vol. 2. find
  16. Oren Etzioni Ana-Maria Popescu and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pages 149–157. find
  17. Zettlemoyer, L.S., Michael, C.: Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. Proc. 21st Conf. Uncertain. Artif. Intell. UAI 2005. 658–666 (2005). find
  18. Zhong, V., Xiong, C., Socher, R.: Seq2Sql: Generating Structured Queries From Natural Language Using Reinforcement Learning. https://arxiv.org/abs/1709. 00103v7. 1–12 (2017). ↗
  19. Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3650–3656. Association for Computational Linguistics. find
  20. Anh Tuan Nguyen, Mai Hoang Dao, and Dat Quoc Nguyen. 2020. A pilot study of text-to-sql semantic parsing for Vietnamese. CoRR, abs/2010.01891. ↗
  21. Xu, X., Liu, C., Song, D.: SQLNet: Generating structured queries from natural language without reinforcement learning. https://arxiv.org/abs/1711.04436. 1–13 (2017). ↗
  22. Wang, B., Shin, R., Liu, X., Polozov, O., Richardson, M.: RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. (2019). https://doi.org/10.18653/v1/2020.acl-main.677. ↗
  23. Lyu, Q., Chakrabarti, K., Hathi, S., Kundu, S., Zhang, J., Chen, Z.: Hybrid ranking network for text-to-SQL. https://arxiv.org/abs/2008.04759 1–12 (2020). ↗
  24. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2023). Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR, abs/2308.15363. ↗ our card
  25. Pourreza, M., & Rafiei, D. (2023). DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. arXiv preprint arXiv:2304.11015. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/heakl-2024-araspider-democratizing-arabic-sql/