Elamur Research⌕
☰
home / papers / 2020 / Faheem at NADI shared task: Identifying the dialect of Arabic tweet

Faheem at NADI shared task: Identifying the dialect of Arabic tweet

Paper · 2020 · Nouf AlShenaifi and Aqil Azmi

Summary

This paper presents Faheem, a system entered in subtask 1 of the NADI 2020 shared task, which asks which of 21 Arab countries a tweet's dialect comes from. Several things make the task hard. Tweets are short. Some are written in MSA rather than dialect, dialects overlap heavily, and retweets may come from other countries. The authors used only the organizers' labeled data and built a classic pipeline with scikit-learn and NLTK. Cleaning removes URLs, user mentions, punctuation and non-Arabic tokens. Stemming and stopword removal are deliberately skipped. Features are tf-idf weighted word n-grams (n = 1–3), combined with character n-grams. Six classifiers were compared, and SVM, Multinomial Naïve Bayes and Logistic Regression were submitted. These three scored well on the Dev set (macro-F1 around 0.78–0.83) and lower under 10-fold CV on the training data (around 0.38–0.42). On the blind test set they collapsed: the best was MNB at macro-F1 ≈ 0.151 with accuracy 0.34. The work matters mainly as a cautionary baseline. It shows that lexical-feature models for fine-grained dialect ID can look strong on development data and still fail on a blind test.

Key points

  • The data come from NADI 2020: 21,000 training tweets, 4,957 Dev and 5,000 unlabeled Test, covering 21 countries. Classes are imbalanced: Egypt has about 21% of the tweets, while Sudan, Somalia, Mauritania, Bahrain and Djibouti have about 1% each.
  • The official metric is macro-averaged F1, which weights every country equally regardless of its size.
  • The submitted classifiers were:
    • MNB with Laplace smoothing alpha = 0.5;
    • linear SVM with C = 1, one-vs-rest;
    • Logistic Regression, one-vs-rest.
  • Gaussian NB, Decision Tree, Random Forest and a Keras deep model all underperformed. The authors blame the small number of tweets per dialect for the deep model's result.
  • On Dev, word n-grams beyond n = 2 brought no gain, but n = 1–3 was kept as a precaution. Official test macro-F1 was SVM 0.146, MNB 0.151 and LR 0.149.

Interesting ideas

  • I1. Variety-specific stemmers and stopword lists are not applied because the variety is unknown until classification, so any choice of normalizer would presuppose the answer.
  • I2. Word n-grams and character n-grams are combined because they capture different signals: whole words catch vocabulary unique to one dialect, while character sequences catch affix patterns.
  • I3. The same models score about 0.83 on Dev, about 0.42 under cross-validation and about 0.15 on the blind test, so the evaluation setup can make results look far better than they are.
  • I4. The authors compare their scores against a chance level of 1/21 to judge whether the system beats random assignment.
  • I5. Under heavy class imbalance, macro-F1 keeps rare classes from being hidden by dominant ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Before applying register- or variety-specific normalization (informal spellings, British vs American forms), Mova should first detect the variety, or keep normalization neutral, so that it does not erase the very cues later stages need.
  • I2 (tokens): Character-affix features could back up the lexicon when guessing the lemma and morphology of unknown or misspelled words, and the decision would stay explainable as a matched suffix.
  • I3 (learn): Mova should always report a blind held-out split next to dev and cross-validation scores, and treat a large gap between them as an alarm for leakage or domain shift before trusting any silver-data gain.
  • I4 (learn): Every evaluation could include an explicit random or majority baseline as a negative control, so that a reported score is always read against what guessing would achieve.
  • I5 (answer): Results on school tests and fable QA could be reported as macro scores per question type, so that rare reasoning kinds are not masked by frequent easy ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nouf AlShenaifi and Aqil Azmi. 2020. Faheem at NADI shared task: Identifying the dialect of Arabic tweet. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alshenaifi2020faheem,
  title = {{Faheem at NADI shared task: Identifying the dialect of Arabic tweet}},
  author = {Nouf AlShenaifi and Aqil Azmi},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.29/}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. ↗
  2. Aqil M Azmi and Eman A Aljafari. 2018. Universal web accessibility and the challenge to integrate informal Arabic users: a case study. Universal Access in the Information Society, 17(1):131–145. ↗
  3. Aqil M Azmi, Manal N Almutery, and Hatim A Aboalsamh. 2019. Real-word errors in Arabic texts: A better algorithm for detection and correction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1308–1320. ↗
  4. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, pages 53–61.
  5. H Bouamor, N Habash, M Salameh, W Zaghouani, O Rambow, D Abdulrahim, O Obeid, S Khalifa, F Eryani, A Erdmann, and K Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), pages 3387–3396. find
  6. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the 4th Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  7. Frederic J Cadora. 1976. Lexical relationships among Arabic dialects and the Swadesh list. Anthropological linguistics, 18(6):237–260. find
  8. Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple Arabic dialects. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4130–4138.
  9. Aurélien Géron. 2019. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O’Reilly Media. find
  10. Nizar Habash and Owen Rambow. 2006. MAGEAD: a morphological analyzer and generator for the Arabic dialects. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 681–688. ↗
  11. N Habash, F Eryani, S Khalifa, O Rambow, D Abdulrahim, A Erdmann, R Faraj, W Zaghouani, H Bouamor, N Zalmout, S Hassan, F Al-Shargi, S Alkhereyf, B Abdulkareem, R Eskander, M Salameh, and H Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), pages 3628–3637.
  12. Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. 2014. Mining of massive datasets. Cambridge University Press. ↗
  13. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs/0205028. find
  14. Pruthwik Mishra and Vandan Mujadia. 2019. Arabic Dialect Identification for Travel and Twitter Text. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 234–238.
  15. Pangeanic.com. 2013. Arabic content on the Internet: an unfilled gap. https://www.pangeanic.com/ knowledge_center/arabic-content-internet-unfilled-gap/. Accessed: 2020-07-05. ↗
  16. F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, and E Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  17. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  18. Bashar Talafha, Wael Farhan, Ahmed Altakrouri, and Hussein Al-Natsheh. 2019. Mawdoo3 AI at MADAR Shared Task: Arabic Tweet Dialect Identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 239–243.
  19. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshenaifi-2020-faheem-nadi-shared-task-identifying/