Elamur Research⌕
☰
home / papers / 2023 / Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus

Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus

Paper · 2023 · Helene Olsen, Samia Touileb and Erik Velldal

Summary

The paper asks why fine-grained Arabic dialect identification on the MADAR-26 corpus has stalled. That corpus covers 25 city dialects plus MSA with parallel sentences from the travel domain. A Multinomial Naive Bayes (MNB) model with surface features (Salameh et al., 2018) still beats pre-trained transformers on it. The authors compare three dialect-aware BERT models (AraBERT, MARBERT, CAMeLBERT) with feature-rich Naive Bayes models. They then study where each family fails and why, rather than only reporting scores. A corpus analysis comes first: sentence lengths and vocabulary, plus lexical overlap measured with the Overlap Coefficient. The error analysis that follows finds that some test sentences are identical across dialect labels, so no classifier can resolve them. It also shows that closely related dialects, above all Rabat and Fes, are confused by every approach. Some errors come from the data itself: short sentences with no dialectal markers and non-Arabic proper nouns. Each model family still gets some sentences right that the other misses. The work matters because it shows that metric comparison alone hides the real obstacles in the benchmark.

Key points

  • Data: MADAR-26 split from MADAR shared task 1, with 1600/200/200 sentences per dialect for train/dev/test and perfectly balanced classes. Sentences average about 5.6 whitespace tokens.
  • Models: AraBERTv0.2-Twitter, MARBERTv2 and CAMeLBERT-Mix, fine-tuned ALUE-style with a classifier on the CLS token, compared against Naive Bayes models. The reference state of the art is MNB at 67.9% accuracy.
  • Overlap with MSA: the Overlap Coefficient ranges from about 0.2 (Sfax, Tunis) to over 0.4 (Muscat, Riyadh).
  • Overlap between dialects: the mean pairwise value is 0.35 (SD 0.07). It is highest within regions such as the Levant and the Gulf, and many dialects have fewer than 400 tokens of their own.
  • Main findings: test-set sentences are duplicated across dialect classes, and Rabat/Fes is the most confused pair. Transformers were run with five seeds, but the error analysis uses a single run. Code is released on GitHub.

Interesting ideas

  • I1. Before training, build a pairwise lexical-overlap heatmap of the classes, grouped by known families, to predict which classes will be confused.
  • I2. Audit the test set for identical inputs that carry different gold labels, since such items set a hard ceiling on accuracy for any model.
  • I3. Compare which test items each model family uniquely gets wrong rather than only comparing aggregate scores; disjoint errors show that the approaches carry complementary signal.
  • I4. Instead of deleting duplicates, merge their labels and turn single-label classification into a multi-label task when the gold answer is genuinely ambiguous.
  • I5. Sort misclassifications into causal categories, such as too-short inputs lacking cues, foreign proper nouns and duplicates, rather than treating all errors alike.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Compute token and relation overlap between English UD treebanks or dialect-converted variants to predict where parser transfer will break, before running gparse.
  • I2 (learn): Scan silver and gold evaluation sets for duplicate sentences with conflicting trees or frames, and exclude or flag them before blaming a parser.
  • I3 (learn): When several witness parsers disagree, log which sentences each one alone gets right, to decide where voting helps and where a specific parser should be trusted.
  • I4 (frames): For speech acts, irony or role labels that are honestly ambiguous, store a set of acceptable labels and score predictions against that set.
  • I5 (answer): Tag failed school-test and fable-QA items by cause, for example missing information in the text versus a reasoning failure, so fixes go to the right stage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OsloOslo · Norway51 works · 4,155 citations🎓University of BergenBergen · Norway9 works · 583 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Helene Olsen, Samia Touileb and Erik Velldal. 2023. Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.30

@inproceedings{olsen2023arabic,
  title = {{Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus}},
  author = {Helene Olsen, Samia Touileb and Erik Velldal},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.30},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.30}
}

References

32 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nouf AlShenaifi and Aqil Azmi. 2020. Faheem at NADI shared task: Identifying the dialect of Arabic tweet. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 282–287, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. Qadi: Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
  3. Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2022. Hierarchical aggregation of dialectal data for Arabic dialect identification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4586–4596, Marseille, France. European Language Resources Association.
  4. Gabriel Bernier-colborne, Cyril Goutte, and Serge Leger. 2023. Dialect and variant identification as a multi-label classification task: A proposal based on near-duplicate analysis. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 142–151, Dubrovnik, Croatia. Association for Computational Linguistics. find
  5. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  6. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  7. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  8. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 241–245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  9. Gael de Francony, Victor Guichard, Praveen Joshi, Haithem Afli, and Abdessalam Bouchekif. 2019. Hierarchical deep learning for arabic dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, page 249–253, Florence, Italy. Association for Computational Linguistics.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Mona Diab and Nizar Habash. 2007. Arabic dialect processing tutorial. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Tutorial Abstracts, pages 5–6, Rochester, New York. Association for Computational Linguistics. find
  12. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  13. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in Arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461, Sofia, Bulgaria. Association for Computational Linguistics.
  14. Asma Etman and AA Louis Beex. 2015. Language and dialect identification: A survey. In 2015 SAI intelligent systems conference (IntelliSys), pages 220– 231. IEEE. find
  15. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, and Phillippe Langlais. 2022. Revisiting pre-trained language models and their evaluation for Arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  16. Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016. Discriminating similar languages: Evaluations and explorations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1800–1807, Portorož, Slovenia. European Language Resources Association (ELRA). find
  17. Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  18. Salima Harrat, Karima Meftouh, Mourad Abbas, Khaled-Walid Hidouci, and Kamel Smaïli. 2016. An algerian dialect: Study and resources. International Journal of Advanced Computer Science and Applications, 7:384–396. ↗
  19. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  20. Tom Lippincott, Pamela Shapiro, Kevin Duh, and Paul McNamee. 2019. Jhu system description for the madar arabic dialect identification shared task. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, page 264–268, Florence, Italy. Association for Computational Linguistics.
  21. Shervin Malmasi, Eshrag Refaee, and Mark Dras. 2016. Arabic dialect identification using a parallel multidialectal corpus. In Computational Linguistics, page 35–53, Singapore. Springer Singapore. find
  22. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  23. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  24. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  25. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171–202. ↗
  26. Marcos Zampieri, Kai North, Tommi Jauhiainen, Mariano Felice, Neha Kumari, Nishant Nair, and Yash Bangera. 2023. Language variety identification with true labels. find
  27. Helene Bøsei Olsen. 2023. Fine-grained arabic dialect identification. find
  28. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-Grained Arabic Dialect Identification. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  29. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  30. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T. Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. ArXiv:2007.05612 [cs]. ↗
  31. Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014. Improved sentence-level arabic dialect classification. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, page 110–119, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
  32. Samia Touileb. 2020. LTG-ST at NADI shared task 1: Arabic dialect identification using a stacking classifier. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 313–319, Barcelona, Spain (Online). Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/olsen-2023-arabic-dialect-identification-depth-error/