Elamur Research⌕
☰
home / papers / 2021 / An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction

An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction

Summary

Dialectal Arabic is the language of everyday online communication, but most Arabic NLP resources target Modern Standard Arabic. Systems trained on MSA perform poorly on dialects, and openly available labeled dialect data is scarce. The authors address this gap with a free Twitter corpus of about 49K tweets from five countries: Algeria, Egypt, Lebanon, Tunisia and Morocco. Each tweet is annotated for three tasks at once: dialect identification, topic classification and sentiment analysis. They describe it as the first openly available dialectal Arabic dataset that is both multi-topic and multi-dialect.

Labels were produced semi-automatically. A small manually labeled seed trained classifiers, those classifiers proposed labels for new batches, and humans corrected the mistakes before the next round. After regex-based cleaning, stemming and TF-IDF/BOW vectorisation, the authors trained classical linear and probabilistic baselines. These reach roughly 0.72–0.84 accuracy across the tasks. The main value of the work is the released data and the reference baselines that later systems can try to beat.

Key points

  • The corpus has 49,306 tweets collected with Selenium and BeautifulSoup, with no date or region filter inside each country:
    • Algeria 13,393
    • Lebanon 14,482
    • Morocco 9,965
    • Tunisia 8,044
    • Egypt 7,519
  • Topic labels are Politics, Health, Social, Sport, Economics, plus 'Other'. 'Other' covers about 75% of tweets (37,313), and the authors advise undersampling it when training.
  • Sentiment counts are 30,033 neutral, 15,385 negative and 6,792 positive. The reported total of 52,210 exceeds the tweet count, which the paper leaves unexplained.
  • Four baselines, each tuned by grid search on TF-IDF/BOW features, were tested: Naive Bayes, Linear SVC, Logistic Regression and an SGD-based classifier. Best accuracy per task:
    • dialect: about 0.76 (Linear SVC); Naive Bayes has the highest balanced accuracy, 0.79
    • topic: 0.84
    • sentiment: 0.77 (SGD)
  • Seed labeling was done in MonkeyLearn with 1,000 tweets per task. The data and code are published openly.

Interesting ideas

  • I1. Growing correction loop: labeling grows in rounds of 1,000 manual seeds, then 200 predicted and corrected, then larger batches, with the model retrained each round; humans only fix errors instead of labeling from scratch. — For Mova (learn): Silver UD trees from the witness parsers could be checked the same way, in growing batches, where corrections retrain the parser and directly measure how much human effort each round saves.
  • I2. Task-specific cleaning: the same raw text is preprocessed differently per task. Emojis are kept for sentiment but removed for dialect detection, and non-Arabic words are dropped only for dialect detection. — For Mova (tokens): The tokenizer could keep emojis, interjections and foreign spans as typed tokens and let each downstream stage decide what to ignore, rather than deleting them early.
  • I3. Keeping the 'Other' class: the authors publish the huge catch-all class instead of discarding it, and leave rebalancing to the user. — For Mova (frames): Frame and speech-act labelers should have an explicit "no frame / other" outcome kept in the data, so a reader learns to abstain instead of forcing a wrong frame.
  • I4. Reporting balanced accuracy with plain accuracy: on skewed labels, the two can rank models differently. Naive Bayes looks mediocre on accuracy but leads on balanced accuracy for dialect detection. — For Mova (learn): Evaluations of rare relations or semantic roles should always report a class-balanced score alongside overall accuracy, so frequent classes do not hide failures.
  • I5. Code-switching as annotation noise: borrowed English and French words are treated as something that degrades automatic labeling unless it is handled explicitly. — For Mova (tokens): Inputs with foreign or code-switched spans should get a language tag at token level, so the tagger and lemmatizer do not silently mis-analyze them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.11000 · arXiv 2102.11000

@misc{chairi2021open,
  title = {{An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction}},
  author = {Chairi;, ElMehdi Boujou; Hamza Chataoui; Abdellah El Mekki; Saad Benjelloun; Ikram and Berrada, Ismail},
  year = {2021},
  doi = {10.48550/arxiv.2102.11000},
  eprint = {2102.11000},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.11000}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Msda open data sets. Mohammed VI Polytechnic University (UM6P), https://msda.um6p.ma/msda_datasets 2. Abdelli, A., Guerrouf, F., Tibermacine, O., Abdelli, B.: Sentiment analysis of arabic algerian dialect using a supervised method. In: 2019 International Conference on Intelligent Systems and Advanced Computing Sciences (ISACS). pp. 1–6 (2019). https://doi.org/10.1109/ISACS48493.2019.9068897 3. Abdul-Mageed, M., Zhang, C., Bouamor, H., Habash, N.: Nadi 2020: The first nuanced arabic dialect identification shared task (2020) 4. Abu Kwaik, K., Saad, M.: ArbDialectID at MADAR shared task 1: Language modelling and ensemble learning for fine grained Arabic dialect identification. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop. pp. 254–258. Association for Computational Linguistics, Florence, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-4632, https://www.aclweb.org/ anthology/W19-4632 5. Alrefaie, M.T.: Arabic stop words (May 2016), https://github.com/mohataher/ arabic-stop-words/blob/master/list.txt 6. Baly, R., Badaro, G., El-Khoury, G., Moukalled, R., Aoun, R., Hajj, H., El-Hajj, W., Habash, N., Shaban, K.: A characterization study of Arabic Twitter data with a benchmarking for state-of-the-art opinion mining models. In: Proceedings of the Third Arabic Natural Language Processing Workshop. pp. 110–118. Association for Computational Linguistics, Valencia, Spain (Apr 2017). https://doi.org/10.18653/v1/W17-1314, https://www.aclweb.org/ anthology/W17-1314 7. Bouamor, H., Hassan, S., Habash, N.: The MADAR shared task on Arabic fine-grained dialect identification. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop. pp. 199–207. Association for Computational Linguistics, Florence, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-4622, https://www.aclweb.org/anthology/W19-4622 8. Boujou, E., Chataoui, H.: Nlp python code for arabic dialects (Nov 2020), https://github.com/Elmehdidebug/ NLP-for-dialectDetection-TopicDetection-SentimentAnalysis 9. Dahou, A., Xiong, S., Zhou, J., Haddoud, M.H., Duan, P.: Word embeddings and convolutional neural network for Arabic sentiment classification. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. pp. 2418–2427. ↗ our card
  2. The COLING 2016 Organizing Committee, Osaka, Japan (Dec 2016), https://www.aclweb.org/anthology/C16-1228 10. El Mekki, A., Alami, A., Alami, H., Khoumsi, A., Berrada, I.: Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification. In: Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020). ↗
  3. Boujou et al. 11. Elfardy, H., Al-Badrashiny, M., Diab, M.: AIDA: Identifying code switching in informal Arabic text. In: Proceedings of the First Workshop on Computational Approaches to Code Switching. pp. 94–101. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/W14-3911, https://www.aclweb.org/anthology/W14-3911 12. Elnagar, A., Einea, O., Al-Debsi, R.: Automatic text tagging of Arabic news articles using ensemble deep learning models. In: Proceedings of the 3rd International Conference on Natural Language and Speech Processing. pp. 59–66. Association for Computational Linguistics, Trento, Italy (Sep 2019), https://www.aclweb. org/anthology/W19-7409 13. Medhaffar, S., Bougares, F., Est‘eve, Y., Hadrich-Belguith, L.: Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments. In: Proceedings of the Third Arabic Natural Language Processing Workshop. pp. 55–61. Association for Computational Linguistics, Valencia, Spain (Apr 2017). https://doi.org/10.18653/v1/W17-1307, https://www.aclweb.org/ anthology/W17-1307 14. Mulki, H., Haddad, H., Gridach, M., Babaoglu, I.: Tw-StAR at SemEval-2017 task 4: Sentiment classification of Arabic tweets. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pp. 664–669. Association for Computational Linguistics, Vancouver, Canada (Aug 2017). https://doi.org/10.18653/v1/S17-2110, https://www.aclweb.org/ anthology/S17-2110 15. Nabil, M., Aly, M., Atiya, A.: ASTD: Arabic sentiment tweets dataset. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 2515–2519. Association for Computational Linguistics, Lisbon, Portugal (Sep 2015). https://doi.org/10.18653/v1/D15-1299, https://www.aclweb. org/anthology/D15-1299 16. Qwaider, C., Chatzikyriakidis, S., Dobnik, S.: Can Modern Standard Arabic approaches be used for Arabic dialects? sentiment analysis as a case study. In: Proceedings of the 3rd Workshop on Arabic Corpus Linguistics. pp. 40–50. Association for Computational Linguistics, Cardiff, United Kingdom (Jul 2019), https://www.aclweb.org/anthology/W19-5606 17. Selab, E., Guessoum, A.: Building talaa, a free general and categorized arabic corpus. In: Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 1. p. 284–291. ICAART 2015, SCITEPRESS - Science and Technology Publications, Lda, Setubal, PRT (2015). https://doi.org/10.5220/0005352102840291, https://doi.org/10.5220/ 0005352102840291 18. Shoukry, A., Rafea, A.: A hybrid approach for sentiment classification of egyptian dialect tweets. In: 2015 First International Conference on Arabic Computational Linguistics (ACLing). pp. 78–85 (2015). https://doi.org/10.1109/ACLing.2015.18 19. Talafha, B., Ali, M., Za’ter, M.E., Seelawi, H., Tuffaha, I., Samir, M., Farhan, W., Al-Natsheh, H.T.: Multi-dialect arabic bert for country-level dialect identification (2020) ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boujou-2021-open-access-nlp-dataset-arabic/