Elamur Research⌕
☰
home / papers / 2024 / Ar-Spider: Text-to-SQL in Arabic

Ar-Spider: Text-to-SQL in Arabic

Summary

Most text-to-SQL benchmarks are English-only. This paper builds Ar-Spider, the first cross-domain Arabic text-to-SQL dataset. It was made by translating the public training and development parts of Spider: GPT-3 produced draft Arabic questions, two professional translators corrected them, and three SQL-literate computer science graduate students checked each question against its SQL query. The authors name two problems specific to the setting. First, the database schemas stay in English while the questions are Arabic. Second, SQL keywords and operators must be matched to Arabic wording. They test LGESQL and S2SQL with the cross-lingual encoders mBERT and XLM-R. They also add Context Similarity Relationships (CSR), new edges in the question-schema graph that connect Arabic tokens to English schema items when their LASER embeddings are similar enough. The best system, LGESQL + XLM-R + CSR, reaches 66.63%, which is 7.73 points below the best English result. The work gives Arabic a standard benchmark and a simple way to link across languages.

Key points

  • Ar-Spider has 9691 questions over 166 databases. Each translator post-edited 83 databases, and the train and dev databases do not overlap.
  • Manual translation took about 70 s per question, an estimated 188 h for the whole set. GPT-3 drafts plus post-editing took about 30 s per question, roughly 80 h in total. GPT-3 was chosen over Google NMT because its wording varied more.
  • Baselines on Ar-Spider:
    • LGESQL + XLM-R scored 65.57%.
    • S2SQL + XLM-R scored 62.48%.
    • The best English result was 74.36%, from LGESQL + XLM-R on Spider.
  • CSR used a 78% similarity threshold and created more than 14000 question-schema links. Its gains were:
    • +2.33 for S2SQL + mBERT
    • +1.52 for S2SQL + XLM-R
    • +1.06 for LGESQL + XLM-R
    • +0.10 for LGESQL + mBERT
  • Unlike earlier Portuguese and Russian Spider versions, training on English and Arabic together slightly lowered Arabic test accuracy. The authors link this to Arabic's rich morphology. Arabic questions were tokenized with Stanza, and S2SQL's question dependency parsing was adapted to Arabic.

Interesting ideas

  • I1. Using machine drafts that humans then post-edit cut annotation time by more than half. It also gave more varied wording, because human translators working from scratch kept choosing the same verb. — For Mova (learn): Test sets and silver data could start as drafts from the LLM teacher tier and then be corrected by a person. This would give more varied phrasing than hand-written items and stop parsers from fitting one annotator's habits.
  • I2. CSR adds explicit, thresholded edges based on embedding similarity between question words and schema items. A graph model can then use these links, and each one can be inspected and has a score. — For Mova (world): Mention-to-ontology linking (WordNet/Wikidata entries) could be stored as named edges with a similarity score and cutoff. Every link would then be traceable and could be tuned, instead of being hidden inside an encoder.
  • I3. Every translated question was checked against its formal target by reviewers who can read that formal language. Being a fluent speaker was not enough on its own. — For Mova (answer): Gold items for maths word problems and school tests should be checked by someone who verifies that the question really implies the stored answer representation, not only that the text sounds natural.
  • I4. Adding the high-resource English data hurt a morphologically rich language here, even though it helped in earlier studies. More data should be treated as something to test, not something to assume. — For Mova (learn): Each mix of training sources should be evaluated against a control run without the added source, as a negative control. A data source would be kept only when it measurably helps.
  • I5. The dependency-aware S2SQL fell behind on Arabic, which the authors put down to encoders with limited Arabic vocabulary. Structure-based components suffer most when the underlying representations are weak. — For Mova (tree): Parser accuracy on unusual or domain-specific vocabulary should be monitored on its own. Errors there spread directly into the frame and world stages built on the tree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3605098.3636065. · arXiv 2402.15012

@misc{almohaimeed2024ar,
  title = {{Ar-Spider: Text-to-SQL in Arabic}},
  author = {Almohaimeed, Saleh and Almohaimeed, Saad and Ghanim, Mansour Al and Wang, Liqiang},
  year = {2024},
  doi = {10.1145/3605098.3636065.},
  eprint = {2402.15012},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3605098.3636065.}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. In Transactions of the Association for Computational Linguistics, Vol. 7. 597–610. find our card
  2. Daria Bakshandaeva, Oleg Somov, Ekaterina Dmitrieva, Vera Davydova, and Elena Tutubalina. 2022. PAUQ: Text-to-SQL in Russian. In Findings of the Association for Computational Linguistics: EMNLP 2022. 2355–2376. find
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (2020). find
  4. Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. LGESQL: line graph enhanced text-to-SQL model with mixed local and non-local relations. arXiv preprint arXiv:2106.01093 (2021). ↗
  5. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020). ↗
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. CoRR abs/1911.02116 (2019). arXiv:1911.02116 http://arxiv.org/abs/1911.02116 ↗
  7. Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022. Code generation using machine learning: A systematic review. Ieee Access (2022). find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805 ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). ↗
  10. Longxu Dou, Yan Gao, Mingyang Pan, Dingzirui Wang, Wanxiang Che, Dechen Zhan, and Jian-Guang Lou. 2023. MultiSpider: towards benchmarking multilingual text-to-SQL semantic parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 12745–12753. find
  11. Abdellah El Mekki, Abdelkader El Mahdaouy, Ismail Berrada, and Ahmed Khoumsi. 2022. AdaSL: an unsupervised domain adaptation framework for arabic multi-dialectal sequence labeling. Information Processing & Management 59, 4 (2022), 102964. find
  12. Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Bowen Li, Jian Sun, and Yongbin Li. 2022. S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers. arXiv preprint arXiv:2203.06958 (2022). ↗
  13. Marcelo Archanjo José and Fabio Gagliardi Cozman. 2021. mRAT-SQL+ GAP: a Portuguese text-to-SQL transformer. In Intelligent Systems: 10th Brazilian Conference, BRACIS 2021, Virtual Event, November 29–December 3, 2021, Proceedings, Part II 10. Springer, 511–525. find
  14. Muhammad Khalifa, Muhammad Abdul-Mageed, and Khaled Shaalan. 2021. Self-training pre-trained language models for zero-and few-shot multi-dialectal Arabic sequence labeling. arXiv preprint arXiv:2101.04758 (2021). ↗
  15. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. In Advances in Neural Information Processing Systems (NeurIPS). find
  16. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. arXiv preprint arXiv:2004.14519 (2020). ↗
  17. Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. arXiv preprint arXiv:1909.13293 (2019). ↗
  18. Anh Tuan Nguyen, Mai Hoang Dao, and Dat Quoc Nguyen. 2020. A pilot study of text-to-SQL semantic parsing for Vietnamese. arXiv preprint arXiv:2010.01891 (2020). ↗
  19. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. arXiv preprint arXiv:2003.00744 (2020). ↗ our card
  20. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543. find
  21. Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, et al. 2022. A survey on text-to-sql parsing: Concepts, methods, and future directions. arXiv preprint arXiv:2208.13629 (2022). ↗
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 3982–3992. find
  23. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401 (2020). ↗
  24. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). ↗
  25. Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. arXiv preprint arXiv:1909.05378 (2019). ↗ our card
  26. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 (2018). ↗ our card
  27. Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al. 2019. Sparc: Cross-domain semantic parsing in context. arXiv preprint arXiv:1906.02285 (2019). ↗ our card
  28. Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences 63, 10 (2020), 2011–2027. find
  29. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 (2017). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almohaimeed-2024-ar-spider-text-sql-arabic/