Elamur Research⌕
☰
home / papers / 2021 / AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection Dataset

AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection Dataset

Summary

The paper addresses the lack of rich Arabic resources for detecting COVID-19 misinformation. Most existing fake news corpora use only a binary real/fake label, which cannot handle opinion posts that are neither true nor false, and they ignore how urgently a post needs verification. The authors built AraCOVID19-MFH, a corpus of 10,828 Arabic tweets that annotators labeled by hand along 10 separate dimensions. These dimensions include hate, factuality, news versus opinion, dialect, advice, morale-raising, blame, cure talk, check-worthiness and fake content.

To show the corpus is usable, they trained transformer classifiers for each label. They compared frozen and fine-tuned encoders, and plain encoders against versions further pretrained on COVID-19 tweets. Fine-tuning gave large gains. The two COVID-adapted models (AraBERT and mBERT variants) scored best on every task, with weighted F-scores above 0.92. The resource matters because one corpus supports several tasks: fake news detection, hate speech detection, dialect identification and opinion/news separation in a low-resource setting.

Key points

  • The dataset has 10,828 Arabic tweets, each manually annotated with 10 labels. Most labels take Yes/No/Can't decide, and "Worth fact-checking" adds a Maybe value. Dialect is labeled as MSA, North Africa or Middle East.
  • Five models were compared: arabert, mbert, distilbert-multi, and two COVID-adapted models, AraBERT COVID-19 and mBERT COVID-19. The COVID-adapted models were further pretrained on 1.5 million multi-dialect Arabic COVID-19 tweets and released on Hugging Face as moha/arabert_c19 and moha/mbert_ar_c19.
  • Evaluation used stratified 5-fold cross-validation per task, reporting averaged weighted F-score. The implementation used Flair, Huggingface-transformers, PyTorch and Scikit-learn.
  • Fine-tuning instead of freezing the encoder improved scores by roughly 0.1–0.5 F. The hardest frozen task was "Contains fake information", which rose from about 0.47–0.77 frozen to about 0.93–0.96 fine-tuned.
  • The closest prior work, by Firoj et al., is also multi-label but has only a few hundred examples per class. This corpus has thousands per class.

Interesting ideas

  • I1. The corpus first marks whether a tweet is verifiable at all (the Factual label) and only then asks whether it is true, so opinions are never forced into a real/fake verdict. — For Mova (frames): The speech-act layer could tag each clause as a checkable assertion or a personal stance before the world layer commits it as a fact, keeping opinions as attributed beliefs.
  • I2. Every label carries an explicit "Can't decide" value, so annotators abstain rather than guess, and that uncertainty stays visible in the data. — For Mova (learn): When witness parsers or LLM teacher tiers disagree, silver data could record an explicit abstain marker instead of a forced majority choice, letting training skip or down-weight those items.
  • I3. A graded check-worthiness label (Yes/Maybe/No) ranks items by potential harm, so limited human effort goes to the most urgent cases first. — For Mova (learn): A similar priority score could decide which sentences are sent to the expensive LLM teacher or to human review, spending budget where parser errors would hurt downstream answers most.
  • I4. The authors continue pretraining a general encoder on in-domain unlabeled text before task training, and they compare frozen and fine-tuned encoders to measure how much each step adds. — For Mova (tree): gparse could be further pretrained on raw fables, tales and school texts before parser training, with a frozen-versus-tuned ablation showing whether domain adaptation actually improves LAS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.03143 · arXiv 2105.03143

@misc{ameur2021aracovid19,
  title = {{AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection Dataset}},
  author = {Ameur, Mohamed Seghir Hadj and Aliane, Hassina},
  year = {2021},
  doi = {10.48550/arxiv.2105.03143},
  eprint = {2105.03143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.03143}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chih-Cheng Lai, Tzu-Ping Shih, Wen-Chien Ko, Hung-Jen Tang, and Po-Ren Hsueh. Severe acute respiratory syndrome coronavirus 2 (sars-cov-2) and coronavirus disease-2019 (covid-19): The epidemic and the challenges. International journal of antimicrobial agents, 55(3):105924, 2020. find
  2. Francesco Di Gennaro, Damiano Pizzol, Claudia Marotta, Mario Antunes, Vincenzo Racalbuto, Nicola Veronese, and Lee Smith. Coronavirus diseases (covid-19) current status and future perspectives: a narrative review. International journal of environmental research and public health, 17(8):2690, 2020. With Fine-tuning Baseline models Pretrained Covid-19 models 0.9858 0.8492 0.9809 0.97 0.9736 0.9809 0.9176 0.9930 0.99 0.9854 0.9774 0.9904 0.9862 0.8948 0.9793 0.9664 0.9764 0.9824 0.9712 0.8838 0.9618 0.9663 0.9618 0.97 0.9627 0.9116 0.9552 0.9409 0.9529 0.9594 0.9317 0.7682 0.9266 0.9137 0.9102 0.9281 0.9653 0.7794 0.9607 0.9476 0.9587 0.9633 0.9608 0.9967 0.9958 0.9917 0.9925 0.995 0.8383 0.9907 0.9885 0.9824 0.9763 0.9891 0.9578 0.7228 0.9417 0.9353 0.9288 0.9491 find
  3. Md Saiful Islam, Tonmoy Sarkar, Sazzad Hossain Khan, Abu-Hena Mostofa Kamal, SM Murshid Hasan, Alamgir Kabir, Dalia Yeasmin, Mohammad Ariful Islam, Kamal Ibne Amin Chowdhury, Kazi Selim Anwar, et al. Covid-19–related infodemic and its impact on public health: A global social media analysis. The American Journal of Tropical Medicine and Hygiene, 103(4):1621, 2020. find
  4. S. Hakak, W. Z. Khan, M. Imran, K. R. Choo, and M. Shoaib. Have you been a victim of covid-19-related cyber incidents? survey, taxonomy, and mitigation strategies. IEEE Access, 8:124134–124144, 2020. find
  5. Aengus Bridgman, Eric Merkley, Peter John Loewen, Taylor Owen, Derek Ruths, Lisa Teichmann, and Oleg Zhilin. The causes and consequences of covid-19 misperceptions: Understanding the role of news and social media. Harvard Kennedy School Misinformation Review, 1(3), 2020. find
  6. Limeng Cui and Dongwon Lee. CoAID: COVID-19 Healthcare Misinformation Dataset. arXiv e-prints, page arXiv:2006.00885, May 2020. ↗
  7. Gautam Kishore Shahi and Durgesh Nandini. Fakecovid - a multilingual cross-domain fact check news dataset for covid-19. 2020. find
  8. Xinyi Zhou, Apurva Mulay, Emilio Ferrara, and Reza Zafarani. ReCOVery: A Multimodal Repository for COVID-19 News Credibility Research, page 3205–3212. Association for Computing Machinery, New York, NY, USA, 2020. find
  9. Xinyi Zhou and Reza Zafarani. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Comput. Surv., 53(5), September 2020. find
  10. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. Fake news detection on social media: A data mining perspective. SIGKDD Explor. Newsl., 19(1):22–36, September 2017. find
  11. James Thorne and Andreas Vlachos. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. find our card
  12. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online, July 2020. Association for Computational Linguistics. find our card
  13. Bennett Kleinberg, Isabelle van der Vegt, and Maximilian Mozes. Measuring emotions in the covid-19 real world worry dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, 2020. find
  14. Juan M Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, and Gerardo Chowell. A large-scale covid-19 twitter chatter dataset for open scientific research–an international collaboration. arXiv preprint arXiv:2004.03688, 2020. ↗
  15. Sarah Alqurashi, Ahmad Alhindi, and Eisa Alanazi. Large arabic twitter dataset on covid-19, 2020. find our card
  16. Junaid Shuja, Eisa Alanazi, Waleed Alasmary, and Abdulaziz Alashaikh. Covid-19 open source data sets: A comprehensive survey. Applied Intelligence, pages 1–30, 2020. find
  17. Mohamed K. Elhadad, Kin Fun Li, and Fayez Gebali. Covid-19-fakes: A twitter (arabic/english) dataset for detecting misleading information on covid-19. In Leonard Barolli, Kin Fun Li, and Hiroyoshi Miwa, editors, Advances in Intelligent Networking and Collaborative Systems, pages 256–268, Cham, 2021. Springer International Publishing. find
  18. Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, and Preslav Nakov. Fighting the covid-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society, 2020. find
  19. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. Arcov19-rumors: Arabic covid-19 twitter dataset for misinformation detection, 2020. find
  20. Parth Patwa, Shivam Sharma, Srinivas PYKL, Vineeth Guptha, Gitanjali Kumari, Md Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. Fighting an infodemic: Covid-19 fake news dataset, 2020. find
  21. Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. Covidlies: Detecting covid-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, 2020. find
  22. Lama Alsudias and Paul Rayson. COVID-19 and Arabic Twitter: How can Arab world governments and public health organizations learn from social media? In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online, July 2020. Association for Computational Linguistics. find
  23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019. find
  24. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9, 2019. find our card
  25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. ↗
  26. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194–206. Springer, 2019. find our card
  27. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. find
  28. Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638–1649, 2018. find our card
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics. find our card
  30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ameur-2021-aracovid19-mfh-arabic-covid-19/