📄 ABDUL: A New Approach to Build Language Models for Dialects Using Formal Language Corpora Only

Summary

North African Arabic dialects (Moroccan, Algerian, Tunisian) have little training data, no standard orthography, and many loanwords. Pronunciation and vowels also vary widely between regions. The authors ask whether a usable dialect model can be built without any dialect text. Their approach is a phoneme-like preprocessing step that strips short vowels and diacritics, keeps long vowels, and rewrites Arabic script in a simplified Latin alphabet derived from Buckwalter. Phonetically close sounds share one symbol, so formal Arabic and dialectal text end up in a common consonant-centred form. With this pipeline they pretrain a BERT model, ABDUL, from scratch on the Arabic portion of OSCAR, which is mostly MSA. They then fine-tune it for dialectal emotion classification and NER. ABDUL performs on par with or better than DarijaBERT, DziriBERT, AraBERT and mBERT, and its NER results are especially strong. The work suggests that clever normalization can stand in for scarce dialect corpora.

Key points

• Pretraining used OSCAR Arabic (about 6.1B words, roughly 84.2 GB). The setup was a WordPiece vocabulary of 30,522, 9 epochs, MLM probability 0.15, and one NVIDIA A100.
• The vocabulary size was chosen to equal the smallest baseline vocabulary. That makes it about four times smaller than mBERT's 119,547.
• Algerian NER on DzNER: ABDUL reaches an F1 of 0.6828, compared with 0.5461 for DziriBERT, the model built for that dialect. Averaged over MSA, Algerian and Moroccan NER, ABDUL scores 0.6010 F1 against 0.5721 for mBERT.
• Moroccan NER on DarNER: ABDUL (0.6557) beats DarijaBERT (0.6246) but trails mBERT (0.7192). The authors attribute this to DarNER spellings that follow Arabic roots instead of actual pronunciation.
• Algerian emotion classification (SemEval 2025 Task 11-A, about 900 examples per dialect): ABDUL places second with a macro-F1 of 0.5315, behind DziriBERT at 0.5573.
• Limitations: Latin-script Arabizi is not covered yet, and the authors say absolute scores remain low for every model tested.

Interesting ideas

• I1. Map both the standard and non-standard varieties into one shared normalized representation, so that a model trained only on the abundant standard variety can be applied to the rare one.
• I2. Remove the features that vary most between speakers (short vowels) and keep the stable skeleton (consonants, long vowels) as the backbone for subwords.
• I3. Give the new model the same vocabulary size as the smallest baseline, so that any improvement cannot be explained by extra capacity.
• I4. When the method loses on a benchmark, inspect that test set's spelling conventions; here, root-based orthography in DarNER accounted for the weaker Moroccan result.
• I5. Train exclusively on formal text and evaluate exclusively on informal text, which turns the domain gap into the thing being measured.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Before parsing, informal or regional English spellings could be converted to a canonical form, with each mapping recorded so the UD tree can be traced back to the raw tokens.
• I2 (tokens): When the lemmatizer meets an unknown or misspelled word, a reduced skeleton key could serve as a fallback lookup, with the result marked as a guess.
• I3 (learn): Comparisons between parser or tagger variants should hold vocabulary and model size constant, so a reported gain reflects the method itself.
• I4 (learn): Evaluation reports could tag test sets by annotation and spelling convention, so losses are traced to data conventions before the model is blamed.
• I5 (learn): Parsers trained on edited treebanks could be scored separately on social-media or dialect test data, giving a clear out-of-domain check.