Elamur Research⌕
☰
home / papers / 2020 / Simple vs Oversampling-based Classification Methods for Fine Grained Arabic Dialect Identification in Twitter

Simple vs Oversampling-based Classification Methods for Fine Grained Arabic Dialect Identification in Twitter

Paper · 2020 · Mohamed Lichouri and Mourad Abbas

Summary

The paper addresses country-level Arabic dialect identification from tweets, which was subtask 1 of the NADI 2020 shared task. Every tweet must be assigned to one of 21 Arab countries. The task is hard because dialects of neighbouring countries are very close, data is scarce, and the classes are strongly imbalanced. The authors ran about 200 configurations. Each combined text preprocessing steps (punctuation removal, letter normalization, stop-word removal, Farasa lemmatization, ISRI stemming, NLTK POS tags) with TF-IDF word and character n-gram features. Two families of systems were compared:

  • plain linear classifiers (LSVC, Ridge, SGD) plus a majority-vote ensemble;
  • the same classifiers trained after rebalancing the data with ROS, SMOTE or ADASYN.

The best post-evaluation result was LSVC with random oversampling, at 18.74% macro F1 on test. The best submitted run scored 18.27%, roughly level with the average of all participants. The work matters mainly as an honest baseline. It shows that cheap n-gram models with careful preprocessing hit a low ceiling on fine-grained dialect identification with little data.

Key points

  • The NADI data contains tweets from 21 countries and 100 provinces: 21,000 train, 5,000 dev and 5,000 test sentences, about 515k words in total. Class sizes are very uneven.
  • The best features were a TF-IDF union of word 5-grams, character 5-grams and character-within-word-boundary 5-grams. Models were trained with stratified 10-fold cross-validation.
  • Effects of preprocessing:
    • Punctuation removal plus Arabic letter normalization helped (19.55% dev with SGD).
    • Removing stop words hurt.
    • Stemming and lemmatization together gave the top dev score (19.75% with SGD).
  • Removing Latin-script letters lowered scores, which suggests Latin tokens carry dialect signal.
  • Among the oversamplers, the simple ROS beat SMOTE and ADASYN. ADASYN was usually the weakest.

Interesting ideas

  • I1. Rather than assuming that heavier normalization always helps, the authors toggled each preprocessing step and found that deleting stop words and Latin tokens actually discarded useful cues.
  • I2. Duplicating minority examples outperformed interpolation-based synthetic sampling in a sparse n-gram space, where points between two vectors may not correspond to plausible text.
  • I3. Looking at length statistics, such as minimum words and characters per sentence, helped the authors choose sensible n-gram orders.
  • I4. Majority voting over several cheap linear classifiers acts as a simple, inspectable combination of witnesses, even when it does not always beat the best single model.
  • I5. The fact that Latin-script tokens inside Arabic tweets carried country signal shows that code-switching itself can be a feature.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Run ablations on tokenizer and normalization choices (e.g. dropping function words or foreign-script tokens) before treating them as harmless cleanup.
  • I2 (learn): When rebalancing rare constructions in silver training data for parsers or frame labellers, prefer reusing real examples over interpolated feature vectors.
  • I3 (learn): Use corpus length and shape statistics to set feature windows and as sanity checks when building evaluation sets.
  • I4 (learn): Compare a vote among witness parsers against the best individual parser, so that agreement-based silver data is not assumed to be better by default.
  • I5 (frames): Treat foreign-language insertions as possible pragmatic or register markers rather than noise when interpreting speaker intent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohamed Lichouri and Mourad Abbas. 2020. Simple vs Oversampling-based Classification Methods for Fine Grained Arabic Dialect Identification in Twitter. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lichouri2020simple,
  title = {{Simple vs Oversampling-based Classification Methods for Fine Grained Arabic Dialect Identification in Twitter}},
  author = {Mohamed Lichouri and Mourad Abbas},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.24/}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mourad Abbas, Mohamed Lichouri, and Abed Alhakim Freihat. 2019. St madar 2019 shared task: Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 269–273. ↗
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  3. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic dialect identification in the wild. arXiv preprint arXiv:2005.06557. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Arun Rajendran, and Lyle Ungar. 2019. Dianet: Bert and hierarchical attention multi-task learning of fine-grained dialect. arXiv preprint arXiv:1910.14243. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. The Shared Task on Nuanced Arabic Dialect Identification (NADI). In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP2020), Barcelona, Spain. find
  6. Kathrein Abu Kwaik and Motaz K Saad. 2019. Arbdialectid at madar shared task 1: Language modelling and ensemble learning for fine grained arabic dialect identification. ArbDialectID at MADAR Shared Task 1: Language Modelling and Ensemble Learning for Fine Grained Arabic Dialect Identification, (Proceedings of the Fourth Arabic Natural Language Processing Workshop).
  7. Fady Baly, Hazem Hajj, et al. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15.
  8. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  9. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
  10. Jason Brownlee. 2020. Smote for imbalanced classification with python. https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/. Accessed: 2020-06-23. ↗
  11. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197. Association for Computational Linguistics.
  12. Guillaume Lemaı̂tre, Fernando Nogueira, and Christos K. Aridas. 2017. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research, 18(17):1– 5. find
  13. Mohamed Lichouri, Mourad Abbas, Abed Alhakim Freihat, and Dhiya El Hak Megtouf. 2018. Word-level vs sentence-level language identification: Application to algerian and arabic dialects. Procedia Computer Science, 142:246–253. find
  14. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  15. Ahmad Ragab, Haitham Seelawi, Mostafa Samir, Abdelrahman Mattar, Hesham Al-Bataineh, Mohammad Zaghloul, Ahmad Mustafa, Bashar Talafha, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2019. Mawdoo3 ai at madar shared task: Arabic fine-grained dialect identification with ensemble learning. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 244–248.
  16. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  17. Rohit Walimbe. 2017. Handling imbalanced dataset in supervised learning using family of smote algorithm. https://www.datasciencecentral.com/profiles/blogs/ handling-imbalanced-data-sets-in-supervised-learning-using-family. Accessed: 2020-06-23. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lichouri-2020-simple-vs-oversampling-based-classification/