Elamur Research⌕
☰
home / papers / 2023 / IUNADI at NADI 2023 shared task: Country-level Arabic Dialect Classification in Tweets for the Shared Task NADI 2023

IUNADI at NADI 2023 shared task: Country-level Arabic Dialect Classification in Tweets for the Shared Task NADI 2023

Paper · 2023 · Yash A. Hatekar and Muhammad S. Abdo

Summary

This system paper describes the IUNADI entry to Task 1 of NADI 2023: assigning a tweet to one of 18 Arab countries based on its dialect. Tweets are hard to classify because they lack standard spelling, stretch letters for emphasis, have irregular spacing, mix dialects with MSA, and share roots across varieties. The organisers released 23.4K labelled tweets and also allowed the NADI 2020, NADI 2021 and MADAR data. No outside data was permitted.

The authors pooled all of the allowed data and applied light Arabic normalisation. They then compared fine-tuned BERT-style encoders against several classical classifiers. Their submitted system was Arabertv2-Large, trained as a 5-fold cross-validation ensemble whose scores were aggregated. It reached a macro F1 of 70.22 on the test set and placed 13th of 16 teams.

The work's main value is as a baseline comparison. Pretrained transformers outperform feature-based models by a wide margin. Hand-crafted normalisation changes little for most models, and for some classifiers it makes results worse.

Key points

  • Data: 18K training, 1.8K development and 3.6K test tweets, covering 18 countries, plus NADI 2020/2021 and MADAR (Bouamor et al., 2018).
  • Preprocessing: diacritic removal, Hamza and Lam-Alif normalisation, Kashida removal, punctuation removal and spelling correction.
  • Development macro F1:
    • Arabertv2-Large, Arabertv2-Base and CAMeLBERT-Mix DID MADAR all scored about 0.71.
    • XGBoost scored 0.52, Random Forest 0.43, Naive Bayes 0.41, SVC 0.39 and AdaBoost 0.18.
  • Effect of preprocessing:
    • It raised Random Forest from 0.39 to 0.43.
    • It lowered Naive Bayes from 0.43 to 0.41.
    • It had no effect on XGBoost, SVC or AdaBoost.
  • Fine-tuning setup: a softmax head on [CLS], learning rate 2e-5, 3 epochs, and the best checkpoint chosen by macro F1. The official test score was F1 70.22 / accuracy 70.78.

Interesting ideas

  • I1. Measuring each normalisation step separately for each model showed that a "cleaning" step can help one learner and hurt another, so normalisation should be judged by its downstream effect rather than assumed useful.
  • I2. Picking the saved checkpoint by the official metric (macro F1) rather than by loss keeps model selection aligned with what is actually reported, which matters most when classes are unbalanced.
  • I3. Training one model per cross-validation fold and averaging their scores gives a cheap ensemble that is more robust, with every training example used for both training and validation.
  • I4. The authors propose a post-check that enforces one consistent dialect label across a whole text, aimed at errors caused by code-switching.
  • I5. Reporting precision alongside recall exposed that Naive Bayes had high precision (0.73) but low recall (0.45), a pattern that a single F1 figure would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): ablate each normaliser in crate en (case folding, punctuation, spelling fixes) against tagger and lemma accuracy instead of applying them all by default.
  • I2 (learn): choose gparse checkpoints by the target LAS or per-relation macro scores, so rare dependency labels are not hidden by frequent ones.
  • I3 (learn): fold-trained parser copies could serve as extra witnesses when agreeing on silver trees, and their disagreement could flag uncertain arcs.
  • I4 (tree): when converting UD dialects or handling mixed-register English, decide the variety once per document and flag sentences whose local guess conflicts with it.
  • I5 (answer): report precision and recall separately for the readers' answers, to tell over-cautious abstaining apart from wrong answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Yash A. Hatekar and Muhammad S. Abdo. 2023. IUNADI at NADI 2023 shared task: Country-level Arabic Dialect Classification in Tweets for the Shared Task NADI 2023. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.72

@inproceedings{hatekar2023iunadi,
  title = {{IUNADI at NADI 2023 shared task: Country-level Arabic Dialect Classification in Tweets for the Shared Task NADI 2023}},
  author = {Yash A. Hatekar and Muhammad S. Abdo},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.72},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.72}
}

References

15 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, ElMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. Nadi 2020: The first nuanced arabic dialect identification shared task. arXiv preprint arXiv:2010.11334. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. Nadi 2021: The second nuanced arabic dialect identification shared task. arXiv preprint arXiv:2103.08466. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  5. William Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  6. Gilbert Badaro, Ouijdane El Jundi, Ali Khaddaj, Ahmad Maarouf, Reine Kain, Hazem Hajj, and Wassim El-Hajj. 2018. Ema at semeval-2018 task 1: Emotion mining for arabic. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 236–244.
  7. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Dima Obeid, Salam Khalifa, Fatima Eryani, Andreas Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  8. Kareem Darwish, Hassan Sajjad, and Hamdy Mubarak. 2014. Verifiably effective arabic dialect identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1465–1468.
  9. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Youssef Benajiba. 2010. Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66–74. find
  10. Genichiro Inoue, Basel Alhafni, Nazym Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  11. Maha Jarrar, Nizar Habash, Dana Akra, and Nasser Zalmout. 2014. Building a corpus for palestinian arabic: a preliminary study. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 18–27.
  12. Maha Jarrar, Nizar Habash, Fatima Alrimawi, Dana Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the palestinian arabic dialect. Language Resources and Evaluation, 51:745–775. 668 4 ↗
  13. Luma Lulu and Ahmed Elnagar. 2018. Automatic arabic dialect classification using deep learning models. Procedia computer science, 142:262–269. find
  14. Ali Y Muaad, Harisha J Davanagere, D S Guru, Jelili B Benifa, Chanda Chola, Hani AlSalman, Abdullah Gumaei, and Moulay A Al-antari. 2022. Arabic document classification: performance investigation of preprocessing and representation techniques. Mathematical Problems in Engineering, 2022:1–16. ↗
  15. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. 669 5 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hatekar-2023-iunadi-nadi-2023-shared-task/