Elamur Research⌕
☰
home / papers / 2025 / Modeling North African Dialects from Standard Languages

Modeling North African Dialects from Standard Languages

Paper · 2025 · Yassine Toughrai, Kamel Smaïli and David Langlois

Summary

North African Arabic dialects such as Algerian and Moroccan have very little text, mix freely with French, and appear in both Arabic and Latin script. The authors ask whether useful encoders for these dialects can be built without any dialect data.

They use a phoneme-oriented normalization that renders both Arabic and French in a simplified Latin form. For Arabic, short vowels and diacritics are removed, similar-sounding letters are merged, and unstable characters become a placeholder. For French, vowels are dropped. BERT-base models are then pretrained from scratch on normalized MSA (OSCAR), French, and synthetic Arabic–French code-switched text. The variants differ in WordPiece vocabulary size (20k, 30k or 40k) and in data mix.

The models are fine-tuned for dialectal NER and sentiment polarity. They match or beat dialect-pretrained and multilingual baselines. Ar+Fr_40k has the best average rank, and Ar_20k wins zero-shot MSA→Algerian NER. Even the 20k vocabulary covers dialect text almost completely. The result matters because standard-language corpora, once surface-harmonized, can stand in for missing dialect corpora.

Key points

  • Pretraining data:
    • The full Arabic part of OSCAR 22.01: about 8.7M documents and about 6.1B words.
    • 1M French documents from the same source.
    • Synthetic code-switching built from OPUS OpenSubtitles ar–fr. About 25% of Arabic content words were translated with Helsinki-NLP/opus-mt-ar-fr.
  • Setup:
    • MLM objective, BERT base, about 128GB of text, 10 epochs, sequence length 512.
    • Variants: Ar_Xk, Ar+Fr_Xk and Ar+Fr+CS (built on Ar+Fr_40k).
  • Baselines: DarijaBERT, DziriBERT, TunBERT, AraBERT v2, MARBERT, CAMeLBERT-DA and mBERT.
  • Supervised results:
    • On DzNER, Ar+Fr_20k reaches 74.60 against 67.23 for MARBERT.
    • On DarNER, mBERT leads with 72.83, while the paper's models score 70.28–71.39.
    • Average rank across the four datasets: 1.75 for Ar+Fr_40k, 7.00 for the best baseline (CAMeLBERT).
  • Zero-shot result: Ar_20k was fine-tuned on ANERCorp (MSA newswire) and tested on DzNER. It scored F1 64.16, against 61.68 for AraBERT v2 and 53.32 for MARBERT.

Interesting ideas

  • I1. The normalization keeps a consonant skeleton, merges look-alike or sound-alike letters, and uses a placeholder for unstable characters; it is applied to the pretraining corpus itself, not only at inference, so standard and variant spellings land on shared subwords.
  • I2. The synthetic code-switched text replaces only content words (nouns, adjectives, names) and leaves function words and complex verbs alone, so the sentence's grammar survives the perturbation.
  • I3. Zero-shot transfer is tested only between datasets that share one annotation scheme: train on the standard variety, test on the informal one, and the gap measures robustness to the variety.
  • I4. An out-of-vocabulary and vocabulary-overlap analysis shows that a small vocabulary already covers the target variety, which supports choosing a compact model on evidence rather than guesswork.
  • I5. Evaluation is kept clean: the checkpoint is picked on a 10% validation split, the test set is scored exactly once, and models are compared by average rank across datasets.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A similar lossy key could sit beside the exact form in the en lexicon, letting the tagger and lemmatizer find candidates for informal or misspelled English words while keeping the original string for explanation.
  • I2 (learn): The same recipe could produce content-word-substituted variants of silver sentences (for example WordNet siblings); any change in the parse would then flag an unstable parse, giving a cheap negative control.
  • I3 (tree): The perceptron parser and gparse could be trained on one English UD treebank and scored on a more informal treebank with the same UD scheme, to measure the cost of a genre or dialect shift.
  • I4 (tokens): Measuring lexicon and morphology coverage per genre (school texts, fables, word problems) would show where unknown words actually come from before the lexicon is enlarged.
  • I5 (learn): Using one-shot test scoring and cross-suite average rank when comparing parser or repair-rule versions would stop gains from being overfit to a single benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire Lorrain de Recherche en Informatique et ses ApplicationsNancy · France32 works · 4,142 citations🎓Université de LorraineNancy · France22 works · 1,314 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Yassine Toughrai, Kamel Smaïli and David Langlois. 2025. Modeling North African Dialects from Standard Languages. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.30

@inproceedings{toughrai2025modeling,
  title = {{Modeling North African Dialects from Standard Languages}},
  author = {Yassine Toughrai, Kamel Smaïli and David Langlois},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.30},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.30}
}

References

23 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julien Abadji, Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2022. Oscar 22.01: A multilingual dataset of web-scraped text. arXiv preprint arXiv:2201.06642. ↗
  2. Amine Abdaoui, Mohamed Berrimi, Mourad Oussalah, and Abdelouahab Moussaoui. 2021. DziriBERT: A pre-trained language model for the algerian dialect. arXiv preprint arXiv:2109.12346. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  4. Fahad Alotaibi and Mark Lee. 2013. Automatically developing a fine-grained arabic named entity corpus and gazetteer by utilizing wikipedia. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 392–400.
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics. Advances in Signal Processing and Artificial Intelligence, Innsbruck (Austria), Austria.
  7. Abdelhafid Dahou and Mohamed Amine Cheragui. 2023. Dzner: A large algerian named entity recognition dataset. Natural Language Processing Journal, 3:100005. ↗
  8. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the International Conference on Learning Representations (ICLR).
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805. ↗
  10. Abdellah El Mekki, Abdelkader El Mahdaouy, Ismail Berrada, and Ahmed Khoumsi. 2021. Domain adaptation for Arabic cross-domain and cross-dialect sentiment analysis from contextualized word embedding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2824–2837, Online. Association for Computational Linguistics. ↗
  11. Khalil Gaanoun, Naira Messaoudi, Ahmed Allak, and Imane Benelallam. 2023. Darijabert: a step forward in nlp for the written moroccan dialect. https://huggingface.co/SI2M-Lab/DarijaBERT. SI2M Lab (INSEA Morocco). ↗
  12. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaïli. 2015. Machine translation experiments on PADIC: A parallel Arabic DIalect corpus. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation (PACLIC 2015), pages 26–34, Shanghai, China.
  13. Lilia Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An algerian corpus and an annotation platform for opinion and emotion analysis (twifil). In Proceedings of The 12th Language Resources and Evaluation Conference (LREC), pages 1202–1210. find
  14. Hicham N Moussa and Anas Mourhir. 2023. Darnercorp: An annotated named entity recognition dataset for the moroccan dialect. Data in Brief, 48:109234. find
  15. Hatem Haddad, Mouna Boussaha, Anis Mahfoudhi, and Lamia Belguith. 2022. Tunbert: The first pre-trained bert model for tunisian arabic. https://huggingface.co/tunis-ai/TunBERT. Instadeep / Tunisia.AI. ↗
  16. Injy Hamed, Caroline Sabty, Slim Abdennadher, Ngoc Thang Vu, Thamar Solorio, and Nizar Habash. 2025. A survey of code-switched Arabic NLP: Progress, challenges, and future directions. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4561–4585, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  17. Salima Harrat, Mourad Abbas, Karima Meftouh, and Kamel Smaïli. 2013. Diacritics Restoration for Arabic Dialects. In INTERSPEECH 2013 - 14th Annual Conference of the International Speech Communication Association, Lyon, France. ISCA. find
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021), pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  19. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395, Minneapolis, Minnesota. Association for Computational Linguistics.
  20. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  21. Yassine Toughrai, Kamel Smaïli, and David Langlois. 2025. ABDUL: A new approach to build language models for dialects using formal language corpora only. In Proceedings of the 1st Workshop on Language Models for Underserved Communities (LM4UC 2025), pages 16–21, Albuquerque, New Mexico. Association for Computational Linguistics.
  22. Muhammad Khalifa, Muhammad Abdul-Mageed, and Khaled Shaalan. 2021. Self-training pre-trained language models for zero- and few-shot multi-dialectal arabic sequence labeling. Preprint, arXiv:2101.04758. ↗
  23. Amina Laggoun, Chahnez Zakaria, and Kamel Smaïli. 2025. Knowledge Distillation for Efficient Algerian Dialect Processing: Training Compact BERT Models with DziriBERT. In 7th International Conference on find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toughrai-2025-modeling-north-african-dialects-standard/