Elamur Research⌕
☰
home / papers / 2019 / ArbDialectID at MADAR Shared Task 1: Language Modelling and Ensemble Learning for Fine Grained Arabic Dialect Identification

ArbDialectID at MADAR Shared Task 1: Language Modelling and Ensemble Learning for Fine Grained Arabic Dialect Identification

Paper · 2019 · Kathrein Abu Kwaik and Motaz Saad

Summary

The paper addresses Arabic Dialect Identification at the city level. Telling 26 varieties apart (25 Arab cities plus MSA) is difficult because neighbouring dialects have no fixed spelling and often look nearly identical in writing. The authors built ArbDialectID for Task 1 of the MADAR shared task, using the travel-domain MADAR corpus.

The system works in two stages:

  • Coarse stage. A first classifier assigns each sentence to one of six classes (five city dialects plus MSA).
  • Fine stage. That predicted coarse label is fed, together with a normalized sentence-length value, into a second classifier covering all 26 labels.

Both stages use weighted TF-IDF features over word n-grams, character n-grams and skip-grams. A hard-voting ensemble of Linear SVC, MultinomialNB and BernoulliNB makes the decision. Everything is built with scikit-learn.

The system took first place with a 67.32% macro F-score on the test set. This shows that carefully engineered classical features, stacked hierarchically, can still beat other entries on fine-grained dialect tasks. The code is released on GitHub.

Key points

  • Data: MADAR Corpus-6 is used for the coarse stage (BEI, CAI, DOH, RAB, TUN, MSA). Corpus-26 is used for the fine stage, with 1,600 train, 200 dev and 200 test sentences per dialect.
  • Coarse model features:
    • word 1–5-grams (weight 0.7);
    • word-boundary-aware character 2–5-grams (0.6);
    • one-skip word unigrams (0.4).
  • Coarse model result: 92.7% accuracy and 93% macro F on dev.
  • Fine model features:
    • word 1–3-grams (0.5);
    • character 2–5-grams with word boundaries (0.5);
    • character 2–4-grams without word boundaries (0.5);
    • skip-grams (0.3);
    • a length ratio (words in the sentence divided by words in the longest sentence);
    • the coarse prediction.
  • Fine model results: dev 68.7% accuracy / 69.00 F; test 67.29% accuracy / 67.32 F. The earlier MNB baseline by Salameh et al. (2018) reported 67.9% accuracy.
  • Per-dialect behaviour: North African dialects were the easiest (TRI 0.79, ALG 0.79, SFX 0.74). The most frequent confusions were between close neighbours: BAG/BAS, AMM/JER, CAI/ASW, ALE/DAM and SFX/TUN.

Interesting ideas

  • I1. Train a cheap coarse classifier first and pass its predicted label to the fine-grained model as an extra input, so the fine model can focus on distinctions inside a broad group.
  • I2. Merge feature views of different granularity under per-view weights, and adjust those weights to where differences actually lie: word level for distant varieties, character and affix level for close ones.
  • I3. Let several dissimilar classifiers (a linear SVM and two naive Bayes variants) vote by majority on discrete labels rather than relying on one model.
  • I4. Read the confusion matrix to find label pairs that are inherently close, treating such errors as signs of real ambiguity rather than plain model failure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A coarse UPOS prediction could become an input feature for the fine morphological-feature tagger, making explicit why a particular feature bundle was chosen.
  • I2 (tokens): For unknown words, the tagger could combine weighted word-context and character n-gram views, giving the affix view more weight when lexical evidence is weak.
  • I3 (learn): This matches the witness-parser approach: accept a silver label only when heterogeneous witnesses agree by hard vote, and send disputed items to a teacher tier.
  • I4 (learn): Evaluation could track known confusable pairs, such as particular deprel or frame-role pairs, separately, so that genuine regressions stand out from irreducible ambiguity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Kathrein Abu Kwaik and Motaz Saad. 2019. ArbDialectID at MADAR Shared Task 1: Language Modelling and Ensemble Learning for Fine Grained Arabic Dialect Identification. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4632

@inproceedings{kwaik2019arbdialectid,
  title = {{ArbDialectID at MADAR Shared Task 1: Language Modelling and Ensemble Learning for Fine Grained Arabic Dialect Identification}},
  author = {Kathrein Abu Kwaik and Motaz Saad},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4632},
  url = {https://doi.org/10.18653/v1/w19-4632}
}

References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kees Versteegh. 2014. The Arabic language. Edinburgh University Press. ↗
  2. Brian Bishop. 1998. A history of the Arabic language. Department of Linguistics, Brigham Young University. find
  3. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  4. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find
  5. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic Dialect Identification. Computational Linguistics, 40(1):171–202. ↗
  6. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  7. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59. Association for Computational Linguistics.
  8. Ryan Cotterell and Chris Callison-Burch. 2014. A Multi-Dialect, Multi-Genre Corpus of Informal Written Arabic. In LREC, pages 241–245.
  9. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep Models for Arabic Dialect Identification on Benchmarked Data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
  10. Heba Elfardy and Mona Diab. 2013. Sentence Level Dialect Identification in Arabic. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), Sofia, Bulgaria.
  11. Charles A. Ferguson. 1959. 15(2):325–340. Diglossia. word, Leena Lulu and Ashraf Elnagar. 2018. Automatic Arabic Dialect Classification Using Deep Learning Models. Procedia computer science, 142:262–269. find
  12. Jonathan Owens. 2015. Arabic language history and the comparative method. International Journal of Arabic Linguistics, 1(1):1–27. find
  13. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  14. Chatrine Qwaider, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. Shami: A Corpus of Levantine Arabic Dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
  15. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-Grained Arabic Dialect Identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  16. Mustafa Shah. 2008. The Arabic language. Routledge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kwaik-2019-arbdialectid-madar-shared-task-1/