Elamur Research⌕
☰
home / papers / 2023 / ALDi: Quantifying the Arabic Level of Dialectness of Text

ALDi: Quantifying the Arabic Level of Dialectness of Text

Paper · 2023 · Amr Keleg, Sharon Goldwater and Walid Magdy

Summary

Arabic text, especially user comments and transcribed speech, often mixes Modern Standard Arabic (MSA) with Dialectal Arabic (DA). Earlier Arabic NLP handled this mostly through Dialect Identification (DI), which labels a sentence or token as either MSA or DA. The authors argue that speakers actually perceive dialectness as a gradient. They formalize this as ALDi, a continuous sentence-level variable that measures how far a sentence departs from the standard language.

To build training data, they go back to the per-sentence dialect-level labels that had been gathered for the Arabic Online Commentary (AOC) corpus but never analyzed. They convert these annotations into continuous scores, analyze them, and release the result as AOC-ALDi with canonical splits. A BERT-based regression model fine-tuned on AOC-ALDi outperforms lexicon-based and DI-based baselines. It also transfers to dialects and genres not seen in training.

Case studies on presidential speeches show that ALDi picks up shifts in style that binary DI misses. This makes ALDi useful for sociolinguistic and computational social science work.

Key points

  • AOC-ALDi contains 127,835 sentences, about 17% from news articles and 83% from reader comments. The comments come from Youm7 (Egypt), AlGhad (Jordan) and AlRiyadh (Saudi Arabia).
  • Original AOC labels are MSA, little, mixed, mostly dialectal and not Arabic, with three independent annotations per sentence.
  • In the original AOC labels, 57% of comments were tagged MSA and 23% mostly dialectal. Control sentences taken from article bodies were about 94% MSA.
  • Each mTurk HIT contained 10 comments plus 2 control sentences from article bodies, and all 12 were labeled by the same annotator.
  • Case studies cover speeches by Ben-Ali, Mubarak and 659 speeches by El-Sisi. In one El-Sisi speech, ALDi separates three levels: the prepared address, a gas-price answer, and a more dialectal human-rights answer.

Interesting ideas

  • I1. Model nonstandardness as a continuous score rather than a yes/no class, because two sentences with the same number of dialect cues can still feel differently "far" from the standard.
  • I2. Mix control items with a known expected answer into every annotation batch, so annotator reliability can be checked on the same sheet.
  • I3. Turn coarse ordinal labels from several annotators into one continuous target and train a regressor on it, keeping disagreement as graded information.
  • I4. Plot a style score sentence by sentence through a document to expose where the speaker shifts register, such as moving from a scripted speech to spontaneous replies.
  • I5. Evaluate a model trained on one genre on corpora from other genres and dialects, to show that it measures the phenomenon rather than the source.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A per-sentence nonstandardness score from the tagger could decide how aggressively to normalize spelling or slang before lemmatization, and could explain why a sentence was treated differently.
  • I2 (learn): Planting known-answer sentences among items sent to LLM teacher tiers or witness parsers gives a built-in negative control for filtering bad silver labels.
  • I3 (learn): When witness parsers disagree on an attachment, an agreement-based soft score could weight training examples instead of discarding them.
  • I4 (frames): Tracking register shifts across a text could feed speech-act and stance analysis, for example flagging where a speaker switches to a persuasive or sympathy-seeking mode.
  • I5 (tree): gparse could routinely be scored on UD treebanks from genres absent in training, to check that parsing quality is not tied to news-like text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Amr Keleg, Sharon Goldwater and Walid Magdy. 2023. ALDi: Quantifying the Arabic Level of Dialectness of Text. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.655

@inproceedings{keleg2023aldi,
  title = {{ALDi: Quantifying the Arabic Level of Dialectness of Text}},
  author = {Amr Keleg, Sharon Goldwater and Walid Magdy},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.655},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.655}
}

References

54 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maha J. Althobaiti. 2020. Automatic arabic dialect identification systems for written texts: A survey. ↗
  2. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. QADI: Arabic dialect identification in the wild. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  3. Maha J. Althobaiti. 2022. Creation of annotated country-level dialectal arabic resources: An unsupervised approach. Natural Language Engineering, 28(5):607–648. ↗
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  6. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  7. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  8. Ibrahim Abu Farha and Walid Magdy. 2022. The effect of Arabic dialect familiarity on data annotation. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 399– 408, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  9. Wafia Adouane and Simon Dobnik. 2017. Identification of languages in Algerian Arabic multilingual documents. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 1–8, Valencia, Spain. Association for Computational Linguistics.
  10. Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2022. Hierarchical aggregation of dialectal data for Arabic dialect identification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4586–4596, Marseille, France. European Language Resources Association.
  11. A. Bergman and Mona Diab. 2022. Towards responsible natural language annotation for the varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022, pages 364–371, Dublin, Ireland. Association for Computational Linguistics.
  12. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA).
  13. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  14. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  15. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective Arabic dialect identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465–1468, Doha, Qatar. Association for Computational Linguistics.
  16. Amitava Das and Björn Gambäck. 2014. Identifying languages at the word level in code-mixed Indian social media text. In Proceedings of the 11th International Conference on Natural Language Processing, pages 378–387, Goa, India. NLP Association of India. find
  17. Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315–2338, Online. Association for Computational Linguistics. ↗
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  19. Mahmoud El-Haj. 2020. Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1318–1326, Marseille, France. European Language Resources Association.
  20. Mohammed K. El-Yasin. 1985. Basic word order in classical arabic and jordanian arabic. Lingua, 65:107– 122. ↗
  21. Heba Elfardy and Mona Diab. 2012. Simplified guidelines for the creation of large scale dialectal Arabic annotations. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 371–378, Istanbul, Turkey. European Language Resources Association (ELRA).
  22. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in Arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461, Sofia, Bulgaria. Association for Computational Linguistics.
  23. Charles A Ferguson. 1959. Diglossia. word, 15(2):325– 340. find
  24. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  25. Saad Gamal-Eldin. 1968. A syntactic study of Egyptian colloquial Arabic / By Saad M. Gamal-Eldin. Janua linguarum. Series practica ; 34. Mouton, The Hague. find
  26. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of arabic dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world, pages 49–53. find
  27. Nizar Y. Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  28. Francis Heylighen and Jean-Marc Dewaele. 1999. Formality of language: definition, measurement and behavioral determinants. Interner Bericht, Center “Leo Apostel”, Vrije Universiteit Brüssel, 4. find
  29. Clive Holes. 2013. Word order and textual function in gulf arabic. In Information structure in spoken Arabic, pages 79–92. Routledge. find
  30. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Proceedings of the first Conference on Arabic Natural Language Processing, ArabicNLP, Singapore (Hybrid). Association for Computational Linguistics.
  31. Scott F. Kiesling. 2011. Linguistic Variation and Change, volume 1. Edinburgh University Press. find
  32. Klaus Krippendorff. 2004. Content Analysis, an Introduction to Its Methodology, second edition. Thousand Oaks, CA: Sage Publications. find
  33. Shibamouli Lahiri. 2016. Squinky! a corpus of sentence-level formality, informativeness, and implicature. ↗
  34. M Lahlali. 2011. The arab spring and the discourse of desperation: shifting from an authoritarian discourse to a “democratic one”. The Journ. of Arab Media and Society.—Cairo: American Univ. in Cairo, (13). find
  35. Karen McNeil. 2022. ‘we don’t speak the same language:’language choice and identity on a tunisian internet forum. International Journal of the Sociology of Language, 2022(278):51–80. ↗
  36. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas. Association for Computational Linguistics.
  37. Alejandro Mosquera and Paloma Moreda. 2012. A qualitative analysis of informality levels in web 2.0 texts: The facebook case study. In Proceedings of the LREC workshop:@ NLP can u tag# user generated content, pages 23–29. find
  38. Hamdy Mubarak. 2018. Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  39. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  40. Dilworth B. Parkinson. 1991. Searching for modern fus.h.a: Real-life formal arabic. al-’Arabiyya, 24:31– 64. find
  41. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61–74. ↗
  42. Kelly Peterson, Matt Hohensee, and Fei Xia. 2011. Email formality in the workplace: A case study on the Enron corpus. In Proceedings of the Workshop on Language in Social Media (LSM 2011), pages 86– 95, Portland, Oregon. Association for Computational Linguistics. find
  43. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics.
  44. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  45. Ahmed Salama, Houda Bouamor, Behrang Mohit, and Kemal Oflazer. 2014. YouDACC: the Youtube dialectal Arabic comment corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1246– 1251, Reykjavik, Iceland. European Language Resources Association (ELRA).
  46. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332– 1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  47. Philippa Shoemark, Debnil Sur, Luke Shrimpton, Iain Murray, and Sharon Goldwater. 2017. Aye or naw, whit dae ye hink? Scottish independence and linguistic identity on social media. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1239–1248, Valencia, Spain. Association for Computational Linguistics. find
  48. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar. Association for Computational Linguistics.
  49. Mohammad Ali Yaghan. 2008. "arabizi": A contemporary style of arabic slang. Design Issues, 24(2):39– 52. ↗
  50. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect Twitter corpus for gender, age and language variety identification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  51. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  52. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  53. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 58–67, Dublin, Ireland. Association for Computational Linguistics and Dublin City University. ↗
  54. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keleg-2023-aldi-quantifying-arabic-level-dialectness/