Elamur Research⌕
☰
home / papers / 2025 / Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification

Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification

Paper · 2025 · Mansour Essgaer, Khamis Massud, Rabia Al Mamlook et al.

Summary

Libyan Arabic is poorly served by NLP. It has no standard spelling, it varies strongly by region, and it differs from Modern Standard Arabic, so MSA-oriented tools perform badly on it. Most dialect work has focused on Egyptian or Gulf varieties. This study treats the problem as binary classification: is a given tweet Libyan or from another Arabic dialect? The authors use the QADI corpus of Twitter sentences covering 18 dialects. First, they apply a chi-square test to decide which surface meta-features are worth keeping. Second, they compare four linear and probabilistic classifiers (Logistic Regression, Linear SVM, Multinomial NB, Bernoulli NB) on tf-idf features. Each model is tested with three word n-gram ranges, each combined with character n-grams up to length five. Multinomial NB performs best on every reported metric, including accuracy, F1, log loss, Cohen's kappa and MCC. The margins over the other models are small, but consistent. The resulting benchmark gives a simple, reproducible baseline for an under-resourced dialect. The paper also documents Libyan phonological and lexical traits, including regional synonyms and Italian loanwords.

Key points

  • Data: QADI corpus with 540,000 tweets in 18 Arabic dialects; the task is framed as Libyan vs. all other dialects.
  • Chi-square screening showed that meta-features such as email mentions and emotion indicators had no significant link to the dialect label, so they were dropped.
  • Feature grid: word n-grams (1,1), (1,2) and (1,3), each combined with character n-grams (1,5), using tf-idf weighting.
  • Best configuration was Multinomial NB with word (1,2) and char (1,5) features:
    • accuracy 85.89% and F1 0.85741;
    • log loss 4.97743, Cohen's kappa 0.71178, MCC 0.71194.
  • Competing models peaked lower: Logistic Regression at 84.41% and Linear SVM at 84.73%. ROC AUC was about 0.92 for LR and SVM and 0.93 for both NB variants.

Interesting ideas

  • I1. Mixing word n-grams with character n-grams up to five characters helps in two ways: it recovers some local word order lost by tf-idf, and it tolerates the unstable spellings of an unwritten variety. This is useful wherever orthography is noisy.
  • I2. Before modelling, each candidate surface feature is tested for statistical association with the label, and irrelevant ones are discarded. This is a cheap, explainable way to prune features.
  • I3. When accuracy gaps between models are tiny, the authors add log loss, Cohen's kappa and MCC to separate them. Chance-corrected and calibration-aware metrics show whether a lead is consistent.
  • I4. A small, fast linear or NB classifier is used as an explicit dialect or variety detector in front of heavier tools, because MSA-trained tools degrade on dialect input.
  • I5. The paper's table of regional synonyms (the same concept with different words per region) is a compact way to make variant lexicons explicit.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Character n-gram cues could back off the tagger and lemmatizer on misspelled, informal or dialectal English tokens that miss the lexicon.
  • I2 (learn): A chi-square gate could vet which tree-level or token-level features a self-induced repair rule may condition on, rejecting features that do not actually correlate with the error.
  • I3 (learn): Evaluation of witness-parser agreement and parser comparisons could report kappa or MCC alongside raw scores, so small LAS differences are not over-read.
  • I4 (tree): A lightweight register or variety detector could route input (e.g. informal social media vs. edited prose) to the right UD dialect conversion or parser model before gparse runs.
  • I5 (world): A variant-to-concept mapping could link regional or informal English synonyms to a single WordNet/ontology node, so the world model treats them as one entity type.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Sebha University · Trine University.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,485 works

Cite

Cite as Mansour Essgaer, Khamis Massud, Rabia Al Mamlook and Najah Ghmaid. 2025. Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification. arXiv:2512.04257 · International Journal of Intelligent Systems and Applications, Volume 17, Number 6, December 2025, Page 118.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5815/ijisa.2025.06.09 · arXiv 2512.04257

@misc{essgaer2025computational,
  title = {{Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification}},
  author = {Mansour Essgaer, Khamis Massud, Rabia {Al Mamlook} and Najah Ghmaid},
  year = {2025},
  doi = {10.5815/ijisa.2025.06.09},
  eprint = {2512.04257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5815/ijisa.2025.06.09}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Ghazali, R. Hamdi, and M. Barkat, “Speech rhythm variation in Arabic dialects,” in Speech Prosody, 2002, pp. 331–334. Accessed: Mar. 06, 2025. find
  2. S. Harrat, K. Meftouh, and K. Smaili, “Machine translation for Arabic dialects (survey),” Information Processing & Management, vol. 56, no. 2, pp. 262–273, 2019. find
  3. A. Abdelali, H. Mubarak, Y. Samih, S. Hassan, and K. Darwish, “Arabic Dialect Identification in the Wild,” May 15, 2020, arXiv: arXiv:2005.06557. doi: 10.48550/arXiv.2005.06557. ↗
  4. H. Bouamor et al., “The MADAR Arabic dialect corpus and lexicon,” in Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018), 2018. find
  5. M. Maamouri, A. Bies, T. Buckwalter, and W. Mekki, “The penn arabic treebank: Building a large-scale annotated arabic corpus,” in NEMLAR conference on Arabic language resources and tools, Cairo, 2004, pp. 466–467. Accessed: Mar. 06, 2025. find
  6. I. Alsarsour, E. Mohamed, R. Suwaileh, and T. Elsayed, “Dart: A large dataset of dialectal arabic tweets,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018. Accessed: Feb. 07, 2025. find
  7. S. Harrat, K. Meftouh, and K. Smaïli, “Maghrebi Arabic dialect processing: an overview,” Journal of International Science and General Applications, vol. 1, 2018, Accessed: Mar. 06, 2025. find
  8. M. Nabil, M. Aly, and A. Atiya, “Astd: Arabic sentiment tweets dataset,” in Proceedings of the 2015 conference on empirical methods in natural language processing, 2015, pp. 2515–2519. ↗
  9. H. A. Alhammi, R. A. Alfard, and A. Ramadan, “Building a twitter social media network corpus for libyan dialect,” International Journal of Computer Electrical Engineering, vol. 10, no. 1, 2018, Accessed: Feb. 18, 2025. find
  10. H. A. Elgabou and D. Kazakov, “Building dialectal Arabic corpora,” in The Proceedings of the First Workshop on Human-Informed Translation and Interpreting Technology (HiT-IT), 2017, pp. 52–57. Accessed: Feb. 07, 2025. find
  11. M. Jarrar, F. A. Zaraket, T. Hammouda, D. M. Alavi, and M. Wählisch, “Lisan: Yemeni, iraqi, libyan, and sudanese arabic dialect corpora with morphological annotations,” in 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), IEEE, 2023, pp. 1–7. Accessed: Mar. 06, 2025. find
  12. Y. Matras, “Loanwords in the world’s languages: A comparative handbook,” Language, vol. 88, no. 3, pp. 647–652, 2012. find
  13. O. Obeid, M. Salameh, H. Bouamor, and N. Habash, “ADIDA: Automatic dialect identification for Arabic,” in Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics (demonstrations), 2019, pp. 6–11. Accessed: Mar. 06, 2025. [Online]. Available: https://aclanthology.org/N19-4002/ ↗
  14. F. Biadsy and J. Hirschberg, “Using prosody and phonotactics in Arabic dialect identification.,” in Interspeech, 2009, pp. 208– 211. Accessed: Mar. 06, 2025. find
  15. F. S. Alorifi, “Automatic identification of arabic dialects using hidden markov models,” PhD Thesis, University of Pittsburgh, 2008. Accessed: Mar. 06, 2025. find
  16. M. Belgacem, G. Antoniadis, and L. Besacier, “Automatic Identification of Arabic Dialects.,” in LREC, 2010. Accessed: Mar. 06, 2025. find
  17. S. Shon, A. Ali, Y. Samih, H. Mubarak, and J. Glass, “ADI17: A fine-grained Arabic dialect identification dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 8244–8248. Accessed: Mar. 06, 2025. ↗
  18. M. Abdul-Mageed, C. Zhang, H. Bouamor, and N. Habash, “NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task,” Nov. 09, 2020, arXiv: arXiv:2010.11334. doi: 10.48550/arXiv.2010.11334. ↗
  19. M. Abdul-Mageed, C. Zhang, A. Elmadany, H. Bouamor, and N. Habash, “NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task,” Oct. 20, 2022, arXiv: arXiv:2210.09582. doi: 10.48550/arXiv.2210.09582. ↗
  20. H. A. Alhammi and K. Haddar, “Building a libyan dialect lexicon-based sentiment analysis system using semantic orientation of adjective-adverb combinations,” International Journal of Computer Theory and Engineering, vol. 12, no. 6, pp. 145–150, 2020. ↗
  21. A. S. Abdulaziz, Code switching between Tamazight and Arabic in the first Libyan Berber news broadcast: An application of Myers-Scotton’s MLF and 4M models. Portland State University, 2014. Accessed: Mar. 06, 2025. find
  22. S. Čéplö, J. Bátora, A. Benkato, J. Milička, C. Pereira, and P. Zemánek, “Mutual intelligibility of spoken Maltese, Libyan Arabic, and Tunisian Arabic functionally tested: A pilot study,” Folia Linguistica, vol. 50, no. 2, Jan. 2016, doi: 10.1515/flin-2016-0021. ↗
  23. M. Athem, M. Essgaer, K. M. S. Ahmed, and A. Agaal, “Building Polar-Oriented Libyan Dialect Corpus Using Emoji-Based Lexicon,” Sebha University Conference Proceedings, vol. 3, no. 2, Art. no. 2, Nov. 2024, doi: 10.51984/sucp.v3i2.3360. ↗
  24. M. Abuzaraida, “Classification of Arabic Comments to Detect Cyberbullying from Social Media Using Convolutional Neural Network and Meta-Learning,” Journal of Computer Science, vol. 21, pp. 622–634, Feb. 2025, doi: 10.3844/jcssp.2025.622.634. ↗
  25. H. Badr, Z. Awahida, M. Essgaer, A. Ajaal, and A. Ahessin, “Named Entity Recognition for Identifying Entities Related to Illegal Migration in Libya: An Analysis of Twitter Textual Data,” in 2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), May 2024, pp. 567–572. doi: 10.1109/MI-STA61267.2024.10599731. ↗
  26. A. Omar, M. Essgaer, and K. M. S. Ahmed, “Using Machine Learning Model To Predict Libyan Telecom Company Customer Satisfaction,” in 2022 International Conference on Engineering & MIS (ICEMIS), Jul. 2022, pp. 1–6. doi: 10.1109/ICEMIS56295.2022.9914055. ↗
  27. V. Ritt-Benmimoun, “Tunisian and Libyan Arabic dialects: common trends-recent developments-diachronic aspects,” 2017, Accessed: Mar. 06, 2025. Authors’ Profiles Mansour Essgaer, Ph.D. in Computer Science from the National University of Malaysia (Universiti Kebangsaan Malaysia) in 2016, specializing in artificial intelligence. He served as a Researcher at the National University of Malaysia's Center for Artificial Intelligence Technology from October 2011 to January 2016. Since 2017, he has been an Assistant Professor in the Faculty of Information Technology at Sebha University, Libya. His research interests include data mining, artificial intelligence, machine learning, natural language processing, and combinatorial optimization. He has published several articles in international journals and conferences, contributing to advancements in his field. Essgaer is a member of the Institute of Electrical and Electronics Engineers (IEEE) and actively participates in research and professional activities. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/essgaer-2025-computational-linguistics-meets-libyan-dialect/