Elamur Research⌕
☰
home / papers / 2021 / NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task

NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task

Paper · 2021 · Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany et al.

Summary

NADI 2021 is the second round of a shared task on telling apart varieties of Arabic by where a tweet was written. It covers 21 Arab countries and 100 provinces. The main change from NADI 2020, which mixed Modern Standard Arabic (MSA) and dialectal Arabic (DA), is that the organizers split the Twitter data by variety. This produced four subtasks: country-level MSA (1.1), country-level DA (1.2), province-level MSA (2.1) and province-level DA (2.2). Locating MSA by country or province had not been attempted before. Participants received new labeled TRAIN/DEV/TEST splits and were scored on CodaLab, with macro-F1 as the official metric. Of 53 registered teams, eight submitted systems. CairoSquad came first in all four subtasks, using a MARBERT-based model. Scores were modest at the country level and very low at the province level, so fine-grained identification is feasible but far from solved. The released dataset gives the community a public benchmark for studying regional variation in both standard and colloquial written Arabic.

Key points

  • Each variety has its own dataset of 31,000 tweets, split into 21,000 TRAIN, 5,000 DEV and 5,000 TEST. The TRAIN, DEV and TEST tweets are disjoint, and class sizes are strongly imbalanced, with Egypt making up about 20% of each set.
  • Participation: 53 teams registered and 8 submitted, for 68 official submissions in total. Teams could submit at most five runs per subtask. Seven system papers were accepted.
  • Best macro-F1 scores, all by CairoSquad:
    • Subtask 1.1 (country-level MSA): 22.38%
    • Subtask 1.2 (country-level DA): 32.26%
    • Subtask 2.1 (province-level MSA): 6.43%
    • Subtask 2.2 (province-level DA): 8.60%
  • Leading systems fine-tuned Arabic Transformers:
    • CairoSquad added adapter modules and vertical attention to MARBERT.
    • CS-UM6P trained MARBERT jointly on the country and province levels through multi-task learning.
    • Phonemer used AraBERT and AraELECTRA.
    • Other teams relied on non-Transformer neural networks or linear classifiers, usually combined in ensembles.
  • The data was distributed as full tweet text rather than tweet IDs, through private download links. The dataset is public at github.com/UBC-NLP/nadi.

Interesting ideas

  • I1. Separating the standard and colloquial registers into parallel subtasks. This gives a cleaner signal for each and lets researchers ask whether even the formal variety carries regional traces. The same design would suit Ukrainian–Surzhyk or standard-versus-regional studies.
  • I2. Defining labels at two nested granularities (country and province) on the same data. This shows directly how performance drops as regions get smaller. It also invites joint training across levels, as CS-UM6P's multi-task approach did.
  • I3. Sharing actual text instead of post IDs. Every participant then works on an identical corpus, which avoids the gaps that appear when deleted tweets can no longer be retrieved. This matters for any benchmark built from social media.
  • I4. Classifying single messages rather than users. The authors stress that this separates the task from user-level geolocation. That framing is useful when only individual short texts are available.
  • I5. Reporting macro-F1 as the main metric under heavy class imbalance, with trivial baselines alongside. The baselines make clear how much real signal a system captures beyond predicting the majority class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,485 works

Cite

Cite as Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor and Nizar Habash. 2021. NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abdulmageed2021nadi,
  title = {{NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task}},
  author = {Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor and Nizar Habash},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.28/}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal Arabic irony corpus extracted from twitter. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6265–6271.
  2. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of Arabic dialects. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. find
  3. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. Samar: Subjectivity and sentiment analysis for Arabic social media. Computer Speech and Language, 28(1):20–37. ↗
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. arXiv preprint arXiv:2010.04900. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), pages 97–110, Barcelona, Spain. ↗
  6. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar. 2020c. Micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876. find
  7. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39.
  8. Rania Al-Sabbagh and Roxana Girju. 2012. YADAC: Yet another Dialectal Arabic Corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2882–2889. find
  9. Nora Al-Twairesh, Rawan Al-Matham, Nora Madi, Nada Almugren, Al-Hanouf Al-Aljmi, Shahad Alshalan, Raghad Alshalan, Nafla Alrumayyan, Shams Al-Manea, Sumayah Bawazeer, Nourah Al-Mutlaq, Nada Almanea, Waad Bin Huwaymil, Dalal Alqusair, Reem Alotaibi, Suha Al-Senaydi, and Abeer Alfutamani. 2018. SUAR: Towards building a corpus for the Saudi dialect. In Proceedings of the International Conference on Arabic Computational Linguistics (ACLing). find
  10. Hassan Alhuzali, Muhammad Abdul-Mageed, and Lyle Ungar. 2018. Enabling deep learning of emotion with first-person seed expressions. In Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, pages 25–35. find
  11. Badr AlKhamiss, Mohamed Gabr, Muhammed El-Nokrashy, and Khaled Essam. 2021. Adapting MARBERT for Improved Arabic Dialect Identification: Submission to the NADI 2021 Shared Task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  12. Maha J Althobaiti. 2020. Automatic Arabic dialect identification systems for written texts: A survey. arXiv preprint arXiv:2009.12622. ↗
  13. Maha J. Althobaiti. 2021. Country-level Arabic Dialect Identification Using Small Datasets with Integrated Machine Learning Techniques and Deep Learning Models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  14. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020a. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  15. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020b. AraELECTRA: pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516. ↗
  16. MS Badawi. 1973. Levels of contemporary Arabic in Egypt. Cairo: Dâr al Ma’ârif. find
  17. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.
  18. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic Dialect Corpus and Lexicon. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. find
  19. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
  20. Kristen Brustad. 2000. The Syntax of Spoken Arabic: A Comparative Study of Moroccan, Egyptian, Syrian, and Kuwaiti Dialects. Georgetown University Press. find
  21. Mark W. Cowell. 1964. A Reference Grammar of Syrian Arabic. Georgetown University Press, Washington, D.C. find
  22. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. COLABA: Arabic dialect annotation and processing. In LREC workshop on Semitic language processing, pages 66–74. find
  23. Mahmoud El-Haj. 2020. Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1318–1326, Marseille, France.
  24. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. CS-UM6P @ NADI’2021: BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  25. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. Aida: Identifying code switching in informal Arabic text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 94–101, Doha, Qatar. find
  26. Hassan Gadalla, Hanaa Kilany, Howaida Arram, Ashraf Yacoub, Alaa El-Habashi, Amr Shalaby, Krisjanis Karins, Everett Rowson, Robert MacIntyre, Paul Kingsbury, David Graff, and Cynthia McLemore. 1997. CALLHOME Egyptian Arabic transcripts LDC97T19. Web Download. Philadelphia: Linguistic Data Consortium. find
  27. R.S. Harrell. 1962. A Short Reference Grammar of Moroccan Arabic: With Audio CD. Georgetown classics in Arabic language and linguistics. Georgetown University Press. find
  28. Clive Holes. 2004. Modern Arabic: Structures, Functions, and Varieties. Georgetown Classics in Arabic Language and Linguistics. Georgetown University Press. find
  29. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. ↗
  30. Elsayed Issa. 2021. Country-level Arabic dialect identification using RNNs with and without linguistic features. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  31. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2016. Curras: an annotated corpus for the Palestinian Arabic dialect. Language Resources and Evaluation, pages 1–31. ↗
  32. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. A Large Scale Corpus of Gulf Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  33. Mohamed Lichouri, Mourad Abbas, Khaled Lounnas, Besma Benaziz, and Aicha Zitouni. 2021. Arabic Dialect Identification based on Weighted Concatenation of TF-IDF Transformers. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  34. Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task. In Proceedings of the third workshop on NLP for similar languages, varieties and dialects (VarDial3), pages 1–14.
  35. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015. Machine translation experiments on padic: A parallel Arabic dialect corpus. In Proceedings of the Pacific Asia Conference on Language, Information and Computation.
  36. Hamdy Mubarak and Kareem Darwish. 2014. Using Twitter to collect a multi-dialectal corpus of Arabic. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), Doha, Qatar. find
  37. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of osact4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52.
  38. Hamada Nayel, Ahmed Hassan, Mahmoud Sobhi, and Ahmed El-Sawy. 2021. Data-Driven Approach for Arabic Dialect Identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  39. Ossama Obeid, Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2019. ADIDA: Automatic dialect identification for Arabic. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 6–11, Minneapolis, Minnesota. Association for Computational Linguistics. find
  40. Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of Arabic language varieties and dialects in social media. Proceedings of SocialNLP, page 22. find
  41. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 1332–1344, Santa Fe, New Mexico, USA.
  42. Kamel Smaı̈li, Mourad Abbas, Karima Meftouh, and Salima Harrat. 2014. Building resources for Algerian Arabic dialects. In Proceedings of the Conference of the International Speech Communication Association (Interspeech). find
  43. Anshul Wadhawan. 2021. Dialect Identification in Nuanced Arabic Tweets Using Farasa Segmentation and AraBERT. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). ↗
  44. Wajdi Zaghouani and Anis Charfi. 2018. ArapTweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. find
  45. Omar F Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  46. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the vardial evaluation campaign 2017.
  47. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardzic, Nikola Ljubešić, Jörg Tiedemann, et al. 2018. Language identification and morphosyntactic tagging: The second vardial evaluation campaign. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 1–17.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mageed-2021-nadi-2021-second-nuanced-arabic/