Elamur Research⌕
☰
home / papers / 2021 / AraCOVID19-SSD: Arabic COVID-19 Sentiment and Sarcasm Detection Dataset

AraCOVID19-SSD: Arabic COVID-19 Sentiment and Sarcasm Detection Dataset

Summary

COVID-19 posts on social media often carry sarcasm, and sarcasm makes automatic opinion mining harder because the literal wording hides the writer's real attitude. When this paper appeared, no Arabic resource labelled both sentiment and sarcasm for pandemic discourse. The authors used COVID-related Arabic keywords to collect tweets from mid-December 2019 to mid-December 2020. They removed retweets, near-identical texts and very short posts, which left about 300k unique tweets. From this pool, one expert annotator labelled 5,162 tweets for two tasks: sarcastic or not, and positive, negative or neutral. The annotator could open embedded links when the tweet alone was unclear. The authors then ran stratified 5-fold cross-validation with three bag-of-words classifiers and three pretrained transformers. Every model scored above 0.89 weighted F-score, and the best ones were SVM and AraBERT. The released corpus, AraCOVID19-SSD, gives researchers a jointly labelled benchmark for studying how sarcasm and polarity interact in crisis-time Arabic text, which mixes MSA and dialects.

Key points

  • Dataset size and labels: AraCOVID19-SSD has 5,162 Arabic tweets, each labelled for sarcasm (1,802 yes / 3,360 no) and sentiment (1,964 positive / 1,197 negative / 2,001 neutral).
  • Collection and filtering: tweets were gathered by keyword over one year. Filtering dropped retweets, duplicates (compared after ignoring links) and tweets with fewer than 5 Arabic words, leaving a pool of about 300k.
  • Annotation process: each tweet was labelled by a single expert because of cost, and the authors plan to annotate more of the pool gradually.
  • Models compared:
    • Bag-of-words: SVM, Random Forest, Logistic Regression.
    • Transformers: AraBERT, mBERT, XLM-Roberta.
    • Tooling: scikit-learn, Flair, Huggingface-transformers, PyTorch.
  • Results: on sarcasm, SVM reached 0.9597 F and AraBERT 0.9527. On sentiment, AraBERT reached 0.9226 and SVM 0.9224. The released text is raw; normalisation, de-elongation and diacritic removal were applied only for training the models.

Interesting ideas

  • I1. Joint labelling: every tweet carries both a sarcasm label and a polarity label, so you can measure how often sarcasm co-occurs with each sentiment instead of treating the two tasks separately. — For Mova (frames): the irony module in prag could be tested on whether a detected sarcastic speech act correctly inverts or shifts the surface polarity of the frame's evaluative role.
  • I2. Context-aware annotation: annotators could follow the tweet's links when the text alone was ambiguous, which acknowledges that sarcasm often depends on information outside the sentence. — For Mova (world): irony judgements could consult the world-of-the-text state and attached context, flagging cases where the literal claim contradicts known entities or events.
  • I3. Leakage-aware deduplication: duplicates were detected after stripping links, and very short items were dropped, so near-copies would not inflate the dataset. — For Mova (learn): silver corpora agreed by witness parsers should be deduplicated on normalised text before splitting, so the same sentence cannot appear in both training and evaluation.
  • I4. Raw release, separate normalisation: the corpus is published as raw text, and normalisation is treated as a model-side step rather than something baked into the data. — For Mova (tokens): crate en could keep the original string and apply normalisation as a reversible, logged layer, so every token can be traced back to its source characters.
  • I5. Strong bag-of-words baseline: simple bag-of-words classifiers were reported alongside transformers under stratified 5-fold weighted F, and here they matched or beat the transformers. — For Mova (learn): a bag-of-words classifier could serve as a negative control in Mova's evaluations; if it scores as well as the structured pipeline, the test probably rewards lexical cues rather than understanding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2110.01948 · arXiv 2110.01948

@misc{ameur2021aracovid19,
  title = {{AraCOVID19-SSD: Arabic COVID-19 Sentiment and Sarcasm Detection Dataset}},
  author = {Ameur, Mohamed Seghir Hadj and Aliane, Hassina},
  year = {2021},
  doi = {10.48550/arxiv.2110.01948},
  eprint = {2110.01948},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.01948}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chih-Cheng Lai, Tzu-Ping Shih, Wen-Chien Ko, Hung-Jen Tang, and Po-Ren Hsueh. Severe acute respiratory syndrome coronavirus 2 (sars-cov-2) and coronavirus disease-2019 (covid-19): The epidemic and the challenges. International journal of antimicrobial agents, 55(3):105924, 2020. find
  2. Francesco Di Gennaro, Damiano Pizzol, Claudia Marotta, Mario Antunes, Vincenzo Racalbuto, Nicola Veronese, and Lee Smith. Coronavirus diseases (covid-19) current status and future perspectives: a narrative review. International journal of environmental research and public health, 17(8):2690, 2020. find
  3. Alexander Bäuerle, Martin Teufel, Venja Musche, Benjamin Weismüller, Hannah Kohler, Madeleine Hetkamp, Nora Dörrie, Adam Schweda, and Eva-Maria Skoda. Increased generalized anxiety, depression and distress during the COVID-19 pandemic: a cross-sectional study in Germany. Journal of Public Health, 42(4):672–678, 07 2020. find
  4. Gautam Kishore Shahi and Durgesh Nandini. Fakecovid - a multilingual cross-domain fact check news dataset for covid-19. 2020. find
  5. Mohamed K. Elhadad, Kin Fun Li, and Fayez Gebali. Covid-19-fakes: A twitter (arabic/english) dataset for detecting misleading information on covid-19. In Leonard Barolli, Kin Fun Li, and Hiroyoshi Miwa, editors, Advances in Intelligent Networking and Collaborative Systems, pages 256–268, Cham, 2021. Springer International Publishing. find
  6. Sarah Alqurashi, Ahmad Alhindi, and Eisa Alanazi. Large arabic twitter dataset on covid-19, 2020. Sarcasm Sentiment Recall Precision F-score Recall Precision F-score 0.9540 0.9538 0.9069 0.91 0.9072 0.9511 0.9512 0.9511 0.8994 0.9019 0.8997 0.9597 0.9223 0.9597 0.9599 0.9243 0.9224 0.9361 0.9369 0.9360 0.9186 0.9211 0.9187 0.9416 0.9431 0.94162 0.8928 0.89592 0.8930 0.9226 0.9527 0.9535 0.9527 0.9225 0.9234 find our card
  7. Mohamed Seghir Hadj Ameur and Hassina Aliane. Aracovid19-mfh: Arabic covid-19 multi-label fake news & hate speech detection dataset. Procedia Computer Science, 189:232–241, 2021. AI in Computational Linguistics. find
  8. Mohammed Rushdi-Saleh, M Teresa Martín-Valdivia, L Alfonso Ureña-López, and José M Perea-Ortega. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology, 62(10):2045– 2054, 2011. find
  9. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal, September 2015. Association for Computational Linguistics. find our card
  10. Nora Al-Twairesh, Hend Al-Khalifa, and AbdulMalik Al-Salman. Arasenti: large-scale twitter-specific arabic sentiment lexicons. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 697–705, 2016. find
  11. Mohamed Aly and Amir Atiya. LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. find
  12. Kathrein Abu Kwaik, Stergios Chatzikyriakidis, Simon Dobnik, Motaz Saad, and Richard Johansson. An Arabic tweets sentiment analysis dataset (ATSAD) using distant supervision and self training. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 1–8, Marseille, France, May 2020. European Language Resource Association. find
  13. Jihen Karoui, Farah Banamara Zitoune, and Véronique Moriceau. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168, 2017. Arabic Computational Linguistics. find
  14. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, FIRE ’19, page 10–13, New York, NY, USA, 2019. Association for Computing Machinery. find
  15. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France, May 2020. European Language Resources Association. find our card
  16. Ibrahim Abu Farha and Walid Magdy. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, 2020. find our card
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019. find
  18. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  19. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July 2020. Association for Computational Linguistics. find
  20. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28, 1998. find
  21. Leo Breiman. Random forests. Machine Learning, 45(1):5–32, Oct 2001. find
  22. Anthony J Myles, Robert N Feudale, Yang Liu, Nathaniel A Woody, and Steven D Brown. An introduction to decision tree modeling. Journal of Chemometrics: A Journal of the Chemometrics Society, 18(6):275–285, 2004. find
  23. Raymond E Wright. Logistic regression. 1995. find
  24. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. find
  25. Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638–1649, 2018. find our card
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. find our card
  27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ameur-2021-aracovid19-ssd-arabic-covid-19/