Elamur Research⌕
☰
home / papers / 2017 / Une approche linguistique pour la détection des dialectes arabes (A linguistic approach for the detection of Arabic dialects)

Une approche linguistique pour la détection des dialectes arabes (A linguistic approach for the detection of Arabic dialects)

Paper · 2017 · Houda Saâdane, Damien Nouvel, Hosni Seffih et al.

Summary

The paper tackles automatic identification of which Arabic dialect a text is written in: Algerian, Tunisian, Moroccan or Egyptian. It covers both Arabic script and Arabizi, which is Arabic typed in Latin letters and often mixed with French or English. Maghrebi dialects in particular lack resources, so the authors first built their own corpora. They collected reader comments from newspaper sites and Facebook pages chosen for their regional origin, which gives each message a dialect label in advance.

Arabizi is converted to Arabic script under the CODA convention. Named entities go through finite-state transducers, and other words through dialect-specific rules and dictionaries, with a morphological analyzer filtering the candidates. Each word is then tagged by morphological analyzers and gazetteers, and a prioritized rule set merges these tags into a final label.

Compared with an n-gram statistical classifier, the dictionary-based method performs better without needing training data. However, the three Maghrebi dialects remain hard to tell apart. A further finding is that short vowels written in Arabizi, normally absent from Arabic script, are a strong cue for the dialect.

Key points

  • Corpora: four dialect corpora were assembled from region-specific sources, each split by script (Arabic vs. Latin). Comments written purely in French or English were excluded.
    • Algerian: Echorouk, Dzfoot
    • Moroccan: Yabiladi, Kifache
    • Tunisian: Aljarida, Facebook
    • Egyptian: Misr5, Facebook
  • Word-level tag set:
    • lang1: MSA, plus foreign words kept in their original spelling
    • lang2: dialectal, with DZ/TN/MA/EG subtags
    • named entity
    • other: punctuation, numbers, emoticons, sounds, URLs
    • an explicit ambiguity tag
  • Transliteration resources: Arabizi→Arabic dictionaries hold 24,442 correspondences (words, bigrams, trigrams), built by native speakers from the most frequent corpus patterns. Arabic is shown in the HSB scheme.
  • Named-entity lists: drawn from ANERGazet and GeoNames, with 7,387 persons, 73,892 locations and 22,772 organisations.
  • Main result: the dictionary approach beat the n-gram approach. The authors attribute this to training corpora that are thin compared with the lexicons. They also note corpus noise: messages filed under the wrong dialect, and English/French words identical to transliterated dialect words.

Interesting ideas

  • I1. Use where a text comes from (a regional newspaper or page) as a cheap, approximately reliable label, so a large labelled corpus appears without manual annotation.
  • I2. Merge several analyzers' outputs with decision rules ordered from most certain (punctuation, dictionary hits) to least, and keep an explicit "ambiguous" label rather than forcing a choice.
  • I3. Over-generate candidate forms with rules or transducers, then let a morphological analyzer discard those that are not real words.
  • I4. A non-standard writing variant can carry information the standard orthography drops (here, short vowels), so avoid normalizing it away before the cue has been used.
  • I5. Audit the evaluation corpus itself, for example items filed under the wrong class or cross-language false friends, because label noise can distort comparisons between methods.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): source metadata such as genre, register or the site a text came from could serve as a weak extra witness when assembling silver data, cross-checked against parser agreement.
  • I2 (tokens): the tagger/lemmatizer combination step could apply rules in confidence order and pass unresolved ambiguity downstream, keeping every decision traceable.
  • I3 (tokens): normalizing noisy English (misspellings, slang spellings) could propose variants by rule and keep only those the morphology in crate en accepts, which stays explainable.
  • I4 (tokens): before normalizing informal spellings or casing, extract the signals they carry (emphasis, dialect, register) as features for later stages.
  • I5 (learn): evaluation sets could include a routine pass that flags suspected mislabeled gold items, alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Houda Saâdane, Damien Nouvel, Hosni Seffih and Christian Fluhr. 2017. Une approche linguistique pour la détection des dialectes arabes (A linguistic approach for the detection of Arabic dialects). Actes des 24ème Conférence sur le Traitement Automatique des Langues Naturelles. Volume 2 - Articles courts.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{saadane2017une,
  title = {{Une approche linguistique pour la détection des dialectes arabes (A linguistic approach for the detection of Arabic dialects)}},
  author = {Houda Saâdane, Damien Nouvel, Hosni Seffih and Christian Fluhr},
  booktitle = {Actes des 24ème Conférence sur le Traitement Automatique des Langues Naturelles. Volume 2 - Articles courts},
  year = {2017},
  url = {https://aclanthology.org/2017.jeptalnrecital-court.30/}
}

References

20 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B ENAJIBA Y. & R OSSO P. (2007). Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In Proceedings of Workshop on Natural Language-Independant Engineering. find
  2. C OTTERELL R. & C ALLISON -B URCH C. (2014). A multi-dialect, multi-genre corpus of informal written arabic. In LREC, p. 241–245.
  3. D ARWISH K. (2013). Arabizi detection and conversion to arabic. p. 217—-224, Doha, Qatar. Association for Computational Linguistics.
  4. E LFARDY H., A L -B ADRASHINY M. & D IAB M. (2014). Aida : Identifying code switching in informal arabic text. p.9̃4 : Citeseer.
  5. E LFARDY H. & D IAB M. T. (2012). Simplified guidelines for the creation of large scale dialectal arabic annotations. In LREC, p. 371–378 : Citeseer.
  6. E LFARDY H. & D IAB M. T. (2013). Sentence level dialect identification in arabic. In ACL (2), p. 456–461.
  7. E SKANDER R., A L -B ADRASHINY M., H ABASH N. & R AMBOW O. (2014). Foreign words and the automatic processing of arabic social media text written in roman script. In Proceedings of The First Workshop on Computational Approaches to Code Switching, p. 1––12, Doha, Qatar. Association for Computational Linguistics.
  8. F LUHR C., R OSSI A., B OUCHESECHE L. & K ERDJOUDJ F. (2012). Extraction of information on activities of persons suspected of illegal activities from web open sources. p.1̃9. find
  9. H ABASH N., D IAB M. T. & R AMBOW O. (2012). Conventional orthography for dialectal arabic. In LREC, p. 711—-718, Istanbul, Turkey.
  10. H ABASH N., S OUDI A. & B UCKWALTER T. (2007). On arabic transliteration. In Arabic computational morphology, p. 15–22 : Springer. find
  11. J ARRAR M., H ABASH N., A KRA D. & Z ALMOUT N. (2014). Building a corpus for palestinian arabic : a preliminary study. In Proceedings of the EMNLP 2014 Workshop on Ar-abic Natural Langauge Processing (ANLP), p. 18—-27, Doha, Qatar. Association for Computational Linguistics.
  12. S AÂDANE H. (2015). Traitement automatique de l’arabe dialectalisé : aspects méthodologiques et algorithmiques. In PhD thesis, Université Grenoble Alpes. find
  13. S AÂDANE H. (2013). Une approche linguistique pour l’extraction des connaissances dans un texte arabe. Les Sables d’Olonne, France : TALN-Récital. find
  14. S AÂDANE H., G UIDERE M. & F LUHR C. (2013). La reconnaissance automatique des dialectes arabes à l’écrit. In colloque international «Quelle place pour la langue arabe aujourd’hui, p. 18–20. find
  15. S AÂDANE H. & H ABASH N. (2015). A conventional orthography for algerian arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, p. 69–79, Beijing, China : Association for Computational Linguistics.
  16. S AÂDANE H., R OSSI A., F LUHR C. & G UIDERE M. (2012). Transcription of arabic names into latin. In 2012 6th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications (SETIT), p. 857–866. find
  17. Y ANG Y. & E ISENSTEIN J. (2013). A log-linear model for unsupervised text normalization. In EMNLP, p. 61–72. find
  18. Z AIDAN O. & C ALLISON -B URCH C. (2014). Arabic dialect identification. p. 171–202. find
  19. Z AIDAN O. F. & C ALLISON -B URCH C. (2011). The arabic online commentary dataset : An annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies : Short Papers - Volume 2, HLT ’11, p. 37–41, Stroudsburg, PA, USA : Association for Computational Linguistics.
  20. Z RIBI I., B OUJELBANE R., M ASMOUDI A., E LLOUZE M., B ELGUITH L. H. & H ABASH N. (2014). A conventional orthography for tunisian arabic. In LREC, p. 2355—-2361, Reykjavik, Iceland.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saadane-2017-une-approche-linguistique-pour-la/