Elamur Research⌕
☰
home / papers / 2016 / Automatic Detection of Arabicized Berber and Arabic Varieties

Automatic Detection of Arabicized Berber and Arabic Varieties

Paper · 2016 · Wafia Adouane, Nasredine Semmar, Richard Johansson et al.

Summary

Standard language identifiers treat almost any Arabic-script text as Modern Standard Arabic. They cannot recognise Berber written in Arabic script (BER), and they lump regional Arabic dialects together. This paper tackles short, informal texts and separates nine classes: BER, MSA, and seven dialect groups (ALG, EGY, GUL, LEV, KUI, MOR, TUN).

Because almost no data existed, the authors built a new dataset and dialectal lexicons from scratch. They also grouped the dialects by shared linguistic features rather than by country. Three classic approaches were compared: Cavnar's n-gram profile method as the baseline, SVM, and Prediction by Partial Matching (PPM).

The best results came from combining character n-grams with lexicon words in an SVM, reaching a macro-average F-score of 92.94%. Cavnar's method struggles on texts of 140 characters or fewer, because all classes share one alphabet with similar character statistics. Remaining errors mostly fall between neighbouring dialects.

The work matters because identifying the language correctly is a prerequisite for any further processing of these under-resourced varieties.

Key points

  • Classes: Arabicized Berber (BER), MSA, and seven dialect groups: Algerian, Egyptian, Gulf, Levantine, Mesopotamian (KUI), Moroccan, Tunisian.
  • Freely available tools tried by the authors (Google Translator, Open Xerox, Translated labs) misclassify BER as MSA.
  • The new dataset and lexicons came from a Masters thesis (Adouane, 2016) and can be obtained for research from the first author.
  • The strongest setup is an SVM on character 5–6-grams plus dialectal vocabulary, giving 92.94% macro F. PPMC5 handles BER vs Arabic and MSA vs dialects well. Cavnar's method is the weak baseline on texts of 140 characters or fewer.
  • Errors cluster within the Maghrebi group and between GUL and KUI, and they are asymmetric: MOR and TUN texts are pulled toward ALG more often than the reverse. BER is rarely confused, while MSA is often mistaken for dialects.

Interesting ideas

  • I1. Character n-grams fail when every class uses the same alphabet with near-identical statistics, so adding hand-compiled lexicon features restores the discriminating signal; surface statistics need a lexical anchor for close varieties.
  • I2. Classes were defined by shared linguistic markers (such as "ch" instead of "k") rather than by political borders, making labels match what is actually observable in text.
  • I3. The confusion matrix was read for direction, not just volume: errors flow mainly toward one "attractor" class, exposing a bias rather than random noise.
  • I4. The proposed two-step scheme first picks a broad regional group, then reweights features to choose the specific variety inside it.
  • I5. Misspellings are treated as signal, not noise, since pronunciation-based spelling is consistent within a variety.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): when the tagger must separate near-identical forms (e.g., dialectal or domain spellings), a curated lexicon feature layered over character features could resolve cases that n-gram statistics blur.
  • I2 (tree): when converting between UD dialects or treebank conventions, grouping sources by observable annotation features rather than by origin could give cleaner conversion rules.
  • I3 (learn): evaluation reports could flag asymmetric confusions (e.g., one dependency label absorbing another) as targets for self-induced repair rules.
  • I4 (frames): semantic role assignment could first pick a coarse VerbNet class family and only then decide fine roles with class-specific cues.
  • I5 (tokens): nonstandard spellings could be kept as features that identify register or dialect instead of being normalised away before analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GothenburgGothenburg · Sweden34 works · 1,625 citations🏛️Commissariat à l'Énergie Atomique et aux Énergies AlternativesParis · France11 works · 416 citations

Also on the byline: Technical University of Moldova.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Wafia Adouane, Nasredine Semmar, Richard Johansson and Victoria Bobicev. 2016. Automatic Detection of Arabicized Berber and Arabic Varieties. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{adouane2016automatic,
  title = {{Automatic Detection of Arabicized Berber and Arabic Varieties}},
  author = {Wafia Adouane, Nasredine Semmar, Richard Johansson and Victoria Bobicev},
  booktitle = {Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)},
  year = {2016},
  url = {https://aclanthology.org/W16-4809/}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Salama, Houda Bouamor, Behrang Mohit and Kemal Oflazer. 2014. YouDACC: the Youtube Dialectal Arabic Comment Corpus. In the Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1246–1251, Reykjavik, Iceland.
  2. Alistair Moffat. 1990. Implementing the PPM data compression scheme. IEEE Transactions on Communications, 38(11), pages 1917–1921. ↗
  3. Abd-El-Jawad, Hassan R.S. 1992. Is Arabic a pluricentric language?. In Clyne, Michael G. Pluricentric Languages: Differing Norms in Different Nations. Contributions to the sociology of language 62. Berlin & New York: Mouton de Gruyter. pages 261–303. ↗
  4. Cyril Goutte , Serge Léger, Shervin Malmasi and Marcos Zampieri. 2016. Discriminating Similar Languages: Evaluations and Explorations. In the Proceedings of Language Resources and Evaluation (LREC). Portoroz, Slovenia. find
  5. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Machine learning in Python. Journal of Machine Learning Research, 12, pages 2825–2830. ↗
  6. Fatma Zohra Chelali, Khadidja Sadeddine and Amar Djeradi. 2015. Speaker identification system using LPC-Application on Berber language. HDSKD journal, 1(2):29–46. find
  7. Fatiha Sadat, Farnazeh Kazemi and Atefeh Farzindar. 2014. Automatic Identification of Arabic Language Varieties and Dialects in Social Media. In the Proceedings of the Second Workshop on Natural Language Processing for Social Media (SocialNLP), pages 22–27, Dublin, Ireland.
  8. Hamdi A. Qafisheh. 1977. A short reference grammar of Gulf Arabic. Tucson: University of Arizona Press. find
  9. Heba Elfardy and Mona Diab. 2013. Sentence-Level Dialect Identification in Arabic. In the Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, Sofia, Bulgaria.
  10. Heikki Palva. 2006. Encyclopedia of Arabic languages and linguistics, v.1, A-Ed.. Leiden: Brill, pages 604–613. find
  11. Houda Bouamor, Nizar Habash and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In the Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland.
  12. Houda Saâdane. 2015. Le traitement automatique de l’arabe dialectalise: aspects methodologiques et algorithmiques. PhD thesis, Université Grenoble Alpes. ↗
  13. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), pages 249–254. ↗
  14. Lameen Souag. 2004. Writing Berber Languages: a quick summary. L. Souag. Archived from http://goo.gl/ooA4uZ, Retrieved on April 8th, 2016. ↗
  15. Marcos Zampieri and Binyam Gebrekidan Gebre. 2012. Automatic Identification of Language Varieties: The Case of Portuguese. In the Proceedings of KONVENS 2012 (Main track: poster presentations), Vienna. find
  16. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy and Yassine Benajiba 2010. COLABA: Arabic dialect annotation and processing. In the Proceedings of the LREC Workshop on Semitic Language Processing, pages 66–74. find
  17. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  18. Omar F. Zaidan. 2012. Crowdsourcing Annotation for Machine Learning in Natural Language Processing Tasks. PhD thesis, Johns Hopkins University. find
  19. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1), pages 171–202. ↗
  20. Peter Behnstadt and Manfred Woidich. 2013. Dialectology. In the Oxford Handbook of Arabic Linguistics, Dialectology, pages 300–323. find
  21. Ramzi Halimouche, Hocine Teffahi and Leila Falek. 2014. Detecting Sentences Types in Berber Language. International Conference on Multimedia Computing and Systems (ICMCS), pages 197–200. find
  22. Shervin Malmasi, Eshrag Refaee and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus In the Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Bali, Indonesia. find
  23. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov , Ahmed Ali and Jörg Tiedemann. 2016. Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task. In the Proceedings of the 3rd Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (VarDial), Osaka, Japan.
  24. Victoria Bobicev. 2015. Discriminating between similar languages using ppm. In the Proceedings of the LT4VarDial Workshop, Hissar, Bulgaria. find
  25. Wafia Adouane. 2016. Automatic Detection of Under-resourced Languages: The case of Arabic Short Texts. Master’s thesis, University of Gothenburg. find
  26. William B. Cavnar and John M. Trenkle. 1994. N-gram-based text categorization. In the Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pages 161–175, Las Vegas, US. find
  27. Yassine Benajiba and Mona Diab. 2010. A web application for dialectal Arabic text annotation. In the Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Up-dates, and Prospects. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adouane-2016-automatic-detection-arabicized-berber-arabic/