📄 Dialect2SQL: A Novel Text-to-SQL Dataset for Arabic Dialects with a Focus on Moroccan Darija

Summary

Text-to-SQL resources exist mostly for English and Chinese, so dialect speakers have almost no benchmark for querying databases in their own language. This work builds Dialect2SQL, which the authors describe as the first large-scale, cross-domain text-to-SQL dataset in an Arabic dialect: Moroccan Darija. Darija mixes Arabic, Berber, French and Spanish elements, which makes it hard for NLP systems. The authors took the BIRD training split and had GPT-4 produce draft Darija versions of its questions. Three native-speaker computer science students who know SQL then corrected the drafts under explicit conventions, and one of them did a final consistency pass. The resource keeps BIRD's hard properties, including long schemas, dirty values and multi-table queries. Code LLMs were tested on a random Darija sample, and StarCoder2-7b scored best on all metrics. Absolute scores were low, which shows the task is far from solved. The dataset gives a starting point for dialect-aware SQL assistants and for wider low-resource work.

Key points

• Dialect2SQL has 9,428 NLQ-SQL pairs over 69 databases. It is derived from BIRD's train set (BIRD itself has 12,751 pairs, 95 databases and 37 domains).
• Each database has on average 137 examples and 8 tables, with a range of 2 to 60 tables. Each record holds db_id, the English question, darija_question, the SQL and the full CREATE TABLE schema.
• The annotation conventions were:
  • Darija is written in Arabic script.
  • Names, places, companies and titles stay in English.
  • Numbers use Western Arabic digits.
  • Schemas are left untranslated.
• Turning the GPT-4 drafts into the final versions required about 23.30% edits as measured by TER.
• The baselines were run on 697 random Darija questions. StarCoder2-7b scored BLEU 0.171, SQAM 0.403 and TSED 0.224, ahead of CodeLlama-7b, StarCoder2-3b and CodeT5-2b.

Interesting ideas

• I1. An LLM writes the first draft, native experts post-edit it, and TER between draft and final measures how much the machine got wrong; this turns a vague "LLM-assisted" claim into a measurable correction rate.
• I2. Entity values such as person or company names are kept verbatim instead of transliterated, because a one-letter change would break exact matching against database values.
• I3. Script and numeral conventions are fixed before annotation, which removes spurious variation that has nothing to do with meaning.
• I4. TSED scores predictions by edit distance between syntax trees rather than by surface n-gram overlap like BLEU, giving partial credit for structurally close outputs.
• I5. The original English question is stored next to its dialect version in every record, giving aligned pairs for free.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): compute an edit rate between LLM-teacher annotations and their corrected silver versions, and use it to rank how far each teacher tier can be trusted.
• I2 (back): when translating through the meta-world, carry entity names as opaque literal anchors copied from the source rather than regenerating them, so round trips cannot corrupt them.
• I3 (tokens): normalize digit systems and similar surface variants in a documented pre-tagging step, so the tagger and lemmatizer see one canonical form.
• I4 (frames): compare predicted and gold frame graphs with a tree or graph edit distance, so evaluation shows how close a wrong frame analysis came instead of only counting exact matches.
• I5 (back): aligned question pairs like these could test whether two phrasings map to the same meaning graph, a check on translation through the meta-world.