📄 Moroccan Dialect -Darija- Open Dataset

Summary

Moroccan Darija is spoken by tens of millions of people but has very few computational resources and no agreed spelling. This data paper introduces DODa, the Darija Open Dataset. It is a crowd-extendable lexical resource that links Darija to English, chosen as the target because most NLP tooling is built around it. The data is hosted on GitHub under the MIT license.

DODa has more than 10K entries written in a Latin-based alphanumeric script, and each entry often appears in several spellings. Entries are grouped two ways: by part of speech and by meaning. The dataset also holds structured extras: verb conjugations, gender and number paradigms for nouns, links from verbs to their derived nouns, and idioms and proverbs. The authors set out spelling guidelines that formalise conventions already common on social media.

As a demonstration, the 1000 ILSVRC ImageNet classes were translated into Darija, and a pretrained ResNet50 was used to label web images in Darija. The paper reports no quantitative evaluation.

The resource matters because earlier datasets such as MADAR and MDED map Darija to MSA, are mostly static, or cover narrow domains.

Key points

• DODa has over 10K Darija ⇆ English entries, is published on GitHub under MIT, and is designed to keep growing through outside contributions.
• It has six part-of-speech files: verbs, nouns, adjectives, adverbs, prepositions and determiners. Semantic classes include food, animals, family, emotions and professions, and there are extra files for idioms and proverbs.
• The morphological subsets cover:
  • conjugations of 800+ verbs in past, future and imperative;
  • about 900 masculine nouns paired with feminine and plural forms;
  • 850+ verb-to-noun pairs.
• The romanisation follows a one-letter-one-sound rule:
  • digits stand for Arabic sounds (3 for ع, 7 for ح, 9 for ق, 5 or kh for خ);
  • capital letters mark emphatic consonants (T for ط, S for ص, D for ض);
  • doubled letters mark gemination, as in 7mam "pigeons" versus 7mmam "bathroom".
• The paper compares DODa with prior resources. MADAR has a travel-domain parallel corpus with 14000 Moroccan entries and a 3063-word lexicon covering 25 city dialects. MDED has 12000+ entries and 34K sentences mapped to MSA.

Interesting ideas

• I1. Rather than inventing a new orthography, the authors wrote down the informal conventions people already use online and made them explicit, so machines can separate words that humans tell apart from context. — For Mova (tokens): A similar explicit, documented table mapping informal or noisy English spellings to canonical forms would keep normalisation in the tokenizer transparent and auditable.
• I2. Each lexicon entry carries several attested spellings instead of a single canonical one, so tools learn the variation directly. — For Mova (tokens): Lemma entries in crate en could list spelling variants such as regional, informal and misspelled forms, letting lemmatisation explain which variant matched.
• I3. Derivational links, such as a verb paired with its derived noun, are stored as a separate structured file rather than left implicit in the lexicon. — For Mova (frames): An explicit verb-to-nominalisation table would let prag map phrases like "the building of the bridge" onto the same VerbNet frame and roles as the verb "build".
• I4. Each word gets both a syntactic class and a semantic class, so one resource serves grammar and topic-level grouping at once. — For Mova (world): Coarse semantic tags alongside POS could act as a cheap fallback ontology layer for entities that WordNet or Wikidata do not cover.
• I5. A pretrained model was reused in a new language simply by translating its output label set through the lexicon, with no retraining. — For Mova (back): Generation and translation through the meta-world could likewise render concept nodes into another language by swapping label lexicons, leaving the underlying graph unchanged.