📄 The Evolution of Darija Open Dataset: Introducing Version 2

Summary

Moroccan Darija is spoken by millions but has very few open NLP resources. Existing ones tend to be closed, rarely updated, limited to one domain, or translated only into MSA. This data paper describes version 2 of the Darija Open Dataset (DODa), a community-built Darija–English resource. It grew from roughly 10,000 entries in its 2021 release to about 100,000. It now stores entries in both Latin and Arabic script, matching how Moroccans actually write the dialect.

DODa is more than a word list. It combines:

• a lexicon tagged by part of speech and by topic
• spelling variants, gender pairs and verb-to-noun links
• verb conjugation tables across tenses
• separate files for idioms and personal names
• more than 45,000 translated sentences, with about 43,000 more still being translated by volunteers

The authors argue that translation into English lets a low-resource dialect use mature English tooling instead of building everything from scratch. There are no model experiments. The contribution is the resource itself, released on GitHub under CC BY-NC 4.0, plus a planned translation evaluation set.

Key points

• Size and coverage: DODa v2 has about 100,000 entries, ten times the first version. It covers both Arabic and Latin script and is described as the largest collaborative Darija–English project.
• Progress figures: about 56,900 words translated, about 45,400 sentences translated and about 43,000 sentences in progress, contributed by a community of 23 people.
• Content breakdown:
  • 23,108 entries tagged by syntactic role
  • about 30,000 extra or augmented entries
  • 1,245 entries grouped by semantic theme (art, education, food, etc.)
  • 493 Moroccan names
• Related resources: MADAR (city dialects, travel domain, into MSA), MDED (dictionary into MSA), a Moroccan Darija Wordnet, OMCD (offensive comments), Goud (news headlines) and AtlasIA. AtlasIA has about 45,000 sentences and 3 models on HuggingFace, and much of its data comes from DODa.
• Hosting, license and next step: the data is on GitHub under CC BY-NC 4.0, and commercial use must be negotiated separately. The next step is a dedicated evaluation set for Darija translation models.

Interesting ideas

• I1. Structured morphology inside the lexicon: the lexicon stores conjugation tables, verb-to-noun links (read → reading) and masculine/feminine pairs as explicit correspondences, so derivation is data rather than hidden rules. — For Mova (tokens): explicit nominalization and inflection link tables could let the lemmatizer map forms like "reading" or "destruction" to their verbs, so frames can be built from nominal predicates too.
• I2. Spelling variants as a first-class field: several accepted spellings are recorded for one headword instead of picking a single canonical form, which helps any system facing noisy or non-standard input. — For Mova (tokens): a variant table (color/colour, informal spellings) feeding normalization before tagging would reduce out-of-vocabulary errors without guessing.
• I3. A partner front-end for untranslated items: a separate web app shows pending sentences to non-technical volunteers, keeping a steady queue of open items flowing to annotators. — For Mova (learn): sentences where the witness parsers disagree could be queued the same way for LLM teacher tiers or human review, focusing annotation effort where silver data is weakest.
• I4. Topic labels on lexicon entries: each word carries a semantic domain (food, economy, education) next to its syntactic class, giving a cheap domain signal. — For Mova (world): domain tags on lexicon entries could prime which ontology region or frame set to prefer when building the world of a text.
• I5. A graded evaluation set: the planned benchmark is meant to span basic vocabulary up to complex sentences and idioms, so a model's weaknesses show up by category rather than as one score. — For Mova (answer): splitting school tests and fable QA into difficulty and phenomenon buckets would show exactly which reading skills fail.