Elamur Research⌕
☰
home / papers / 2019 / MICHAEL: Mining Character-level Patterns for Arabic Dialect Identification (MADAR Challenge)

MICHAEL: Mining Character-level Patterns for Arabic Dialect Identification (MADAR Challenge)

Paper · 2019 · Dhaou Ghoul and Gaël Lejeune

Summary

This paper presents MICHAEL, a system built for the MADAR shared task on travel-domain Arabic Dialect Identification (DID). The task is to give each short sentence one of 26 labels: 25 city-level dialects plus MSA. The authors work directly on raw sentences, with no tokenization, normalization or external resources. They extract character N-grams and feed them to the scikit-learn Multinomial Naive Bayes (MNB) classifier. The data is the MADAR corpus, a set of BTEC travel sentences rendered into many dialects. The paper systematically varies the minimum and maximum N-gram length. Their official submission, which used N from 1 to 3, scored 53.25% accuracy. Using character 4-grams alone, they later reached 62.17%. N-grams longer than 4 brought no further benefit, which the authors attribute to such long units encoding vocabulary more than dialect traits. A confusion analysis shows that most errors fall between dialects of the same region. The work matters because it shows that a fast, preprocessing-free character model gives a solid baseline on a fine-grained, 26-way dialect task.

Key points

  • MADAR split sizes: Train has 41,600 sentences, Dev has 5,200 and Test has 5,200. Classes are 25 city dialects plus MSA.
  • Accuracy grows with the N-gram range:
    • Unigrams alone give about 18%.
    • Raising Nmax to 2 adds about 19 pp, raising it to 3 adds about 16 pp more, and raising it to 4 adds about 6 pp.
    • Raising Nmin from 1 to 3 (with Nmax = 3) adds about 4 pp.
  • The best result is 62.17% with 4-grams only (Train+Dev → Test). The official score was 53.25% (1 ≤ N ≤ 3).
  • Test scores are on average 1.6 pp below Dev scores. Adding Dev to training helped only marginally (about +0.26 pp). A one-vs-rest SVM could not finish under the authors' compute constraints.
  • Error analysis:
    • The hardest pairs are within the Maghreb: FES–RAB and SFX–TUN.
    • MUS and SAN are most often confused with MSA.
    • Two cross-region pairs are oddly hard: BEN–RIY and KHA–MUS.

Interesting ideas

  • I1. Tuning the lower bound of the N-gram window matters, not just the upper one: removing very short character units cut noise and raised accuracy.
  • I2. The authors read the accuracy plateau beyond 4-grams as a sign that features stopped capturing variety-level style and began memorizing vocabulary.
  • I3. The confusion matrix is grouped by region, so within-family errors can be separated from surprising long-distance confusions, which are then flagged for inspection.
  • I4. A deliberately cheap, preprocessing-free model gives a reproducible floor that heavier systems must beat, and the gap between dev and test is reported explicitly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): When building suffix/prefix features for the tagger's handling of unknown words, grid-search both the minimum and maximum affix length rather than adding every short affix by default.
  • I2 (tree): A small character-N-gram classifier could tag which treebank or English variety an input resembles. Watching where its gains stop would show whether it detects genuine style or just topic words before it is trusted to choose dialect-conversion rules.
  • I3 (learn): Evaluate the parser with confusion tables grouped by deprel family. Investigate any anomalous cross-family confusions first as possible data or conversion bugs.
  • I4 (learn): Keep a trivial character-N-gram baseline as a standing control for every classification-style component, and always report dev-versus-test drift alongside headline scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sorbonne UniversitéParis · France32 works · 7,270 citations🎓Université Stendhal – Grenoble 3Saint-Martin-d'Hères · France2 works · 26 citations🔬Sens, Texte, Informatique, HistoireParis · France4 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Dhaou Ghoul and Gaël Lejeune. 2019. MICHAEL: Mining Character-level Patterns for Arabic Dialect Identification (MADAR Challenge). Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4627

@inproceedings{ghoul2019michael,
  title = {{MICHAEL: Mining Character-level Patterns for Arabic Dialect Identification (MADAR Challenge)}},
  author = {Dhaou Ghoul and Gaël Lejeune},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4627},
  url = {https://doi.org/10.18653/v1/w19-4627}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohand Tilmatine. 1999. Substrat et convergences: le berbre et l’arabe nord-africain. Estudios de dialec-tolog’ia norteafricana y andalus’i, EDNA, pages 99–120. find
  2. Nora Al-Twairesh, Hend Al-Khalifa, and Abdulmalik AlSalman. 2016. AraSenTi: Large-scale twitter-specific Arabic sentiment lexicons. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 697–705, Berlin, Germany. Association for Computational Linguistics.
  3. Cornelis Henricus Maria Versteegh. 2001. The Arabic Language. Edinburgh University Press Series. Edinburgh University Press. find
  4. Omar Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1220–1229, Portland, Oregon, USA. Association for Computational Linguistics.
  5. Muhammad Amara. 2010. Reem bassiouney: Arabic sociolinguistics. Language Policy, 9:379–381. ↗
  6. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of LREC 2018. find
  7. Omar Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  8. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR Shared Task on Arabic Fine-Grained Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  9. Davide Buscaldi, Joseph Le Roux, and Gaël Lejeune. 2018. Character-level models for polarity detection in tweets. In Atelier DEFT 2018, Rennes, France. find
  10. Wingyan Chung. 2008. Web searching in a multilingual world. Commun. ACM, 51(5):32–40. find
  11. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for arabic dialect identification on benchmarked data. In VarDial@COLING 2018.
  12. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in Arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461, Sofia, Bulgaria. Association for Computational Linguistics.
  13. Niloofar Haeri. 2003. Sacred Language, Ordinary People: Dilemmas of Culture and Politics in Egypt. Sacred Language, Ordinary People: Dilemmas of Culture and Politics in Egypt. Palgrave Macmillan. find
  14. Terence Frederic Mitchell. 1993. Pronouncing Arabic. vol. 2. Clarendon Press. find
  15. Maryam Najafian, Sameer Khurana, Suwon Shon, Ahmed Ali, and James Glass. 2018. Exploiting convolutional neural networks for phonotactic based dialect identification. In ICASSP, pages 5174–5178. IEEE. find
  16. Rabee Naser and Abualsoud Hanani. 2018. Birzeit arabic dialect identification system for the 2018 vardial challenge. In VarDial@COLING 2018.
  17. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghoul-2019-michael-mining-character-level-patterns/