Elamur Research⌕
☰
home / papers / 2023 / PD-AR at ArAIEval Shared Task: A BERT-Centric Approach to Tackle Arabic Disinformation

PD-AR at ArAIEval Shared Task: A BERT-Centric Approach to Tackle Arabic Disinformation

Paper · 2023 · Pritam Deka and Ashwathy Revi

Summary

The paper describes a system for the ArAIEval 2023 shared Task 2, which targets disinformation in Arabic tweets. Subtask 2A is a binary decision on whether a tweet is disinformative. Subtask 2B assigns one of four disinformation types: hate speech, offensive, rumour or spam. The authors argue that Arabic's dialects, rich morphology and code-mixed tweets make it harder than English for this task. Their system cleans the tweets with regex rules and then fine-tunes AraBERTv0.2-Twitter, a BERT model pre-trained on Arabic Twitter text, using the Huggingface framework. They compare it with five other models:

  • English-only models: BERT-base and RoBERTa-base.
  • Multilingual models: XLM-RoBERTa-base and mBERT.
  • An Arabic model: CamelBERT.

Their system beats all of these and the official random and majority baselines on both subtasks. It finishes slightly behind the top team. An ablation that removes classes shows that the rare Rumor class pulls down the multiclass scores. The main lesson is that matching the pre-training domain (Twitter Arabic, not just Arabic) gives measurable gains.

Key points

  • Data size and imbalance:
    • Subtask 2A has 16,062 no-disinfo tweets and 3,929 disinfo tweets in total.
    • Subtask 2B has HS 2,180, SPAM 762, OFF 735 and Rumor only 252.
  • Preprocessing: Arabic and English punctuation is stripped. Some Arabic symbols are normalized. Repeated characters, hashtags, URLs and mentions are removed.
  • Training setup:
    • Max length 64 tokens, batch size 16, 20 epochs, learning rate 2e-5.
    • Adam optimizer with weight decay 0.01.
    • A checkpoint is saved every 200 steps, and the best one is chosen by accuracy.
    • The loss is binary cross-entropy for 2A and categorical cross-entropy for 2B.
  • Subtask 2A test results: micro-F1 0.9021 and macro-F1 0.8595. The top team (DetectiveRedasers) scored 0.9048 and 0.8626. CamelBERT reached 0.8924 micro-F1.
  • Subtask 2B and ablation:
    • Test results were macro-F1 0.7209 and micro-F1 0.8174, against 0.7541 and 0.8356 for the top team.
    • Dropping Rumor raised the scores to 0.7926 macro and 0.8505 micro.
    • Dropping SPAM instead lowered them to 0.6373 macro and 0.7574 micro.

Interesting ideas

  • I1. To show that a rare class is the real problem, the authors also dropped a different, larger class and compared the effect, so the "removing any class helps" explanation is ruled out.
  • I2. A model pre-trained on the same register as the test data (tweets) beat a model pre-trained on general text in the same language, so domain match matters beyond language match.
  • I3. Results are reported as both macro- and micro-F1 alongside random and majority baselines, which exposes how much of a score comes from frequent classes.
  • I4. Surface noise such as repeated characters, URLs and mentions is normalized away before the model sees the text, which shrinks spurious vocabulary variation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When a repair rule or data filter improves scores, run the same removal on a size-matched control slice to confirm the gain comes from the targeted phenomenon.
  • I2 (tree): For gparse, prefer encoders or silver treebanks from the target genre (e.g., fables, school texts) rather than generic English when parsing those genres.
  • I3 (learn): Evaluations of taggers or frame classifiers should always show a majority-class floor and macro scores, so rare labels like uncommon dependency relations are not hidden.
  • I4 (tokens): The en tokenizer could map such noise to explicit placeholder tokens instead of deleting it, keeping the step explainable while still normalizing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SouthamptonSouthampton · United Kingdom13 works · 2,608 citations🎓Queen's University BelfastBelfast · United Kingdom6 works · 234 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Pritam Deka and Ashwathy Revi. 2023. PD-AR at ArAIEval Shared Task: A BERT-Centric Approach to Tackle Arabic Disinformation. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.57

@inproceedings{deka2023pd,
  title = {{PD-AR at ArAIEval Shared Task: A BERT-Centric Approach to Tackle Arabic Disinformation}},
  author = {Pritam Deka and Ashwathy Revi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.57},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.57}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Al-Rawi, Abdelrahman Fakida, Kelly Grounds, et al. 2022. Investigation of covid-19 misinformation in arabic on twitter: Content analysis. Jmir Infodemiology, 2(2):e37007. ↗
  2. Maha Al-Yahya, Hend Al-Khalifa, Heyam Al-Baity, Duaa AlSaeed, and Amr Essam. 2021. Arabic fake news detection: comparative study of neural networks and transformer-based approaches. Complexity, 2021:1–10. ↗
  3. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives, 31(2):211–236. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Arabert: Transformer-based model for arabic language understanding. ↗
  6. Nsrin Ashraf, Hamada Nayel, and Mohamed Taha. 2022. Misinformation detection in arabic tweets: A case study about covid-19 vaccination. Benha Journal of Applied Sciences, 7(5):265–268. ↗
  7. Alessandro Bondielli and Francesco Marcelloni. 2019. A survey on fake news and rumour detection techniques. Information Sciences, 497:38–55. ↗
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  9. Madeleine de Cock Buning. 2018. A multi-dimensional approach to disinformation: Report of the independent high level group on fake news and online disinformation. Publications Office of the European Union. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Mohamed K Elhadad, Kin Fun Li, and Fayez Gebali. 2019. Fake news detection on social media: a systematic survey. In 2019 IEEE Pacific Rim conference on communications, computers and signal processing (PACRIM), pages 1–8. IEEE. find
  12. Khaled M Fouad, Sahar F Sabbeh, and Walaa Medhat. 2022. Arabic fake news detection using deep learning. Computers, Materials & Continua, 71(2). ↗
  13. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2020. Arcov19-rumors: Arabic covid-19 twitter dataset for misinformation detection. arXiv preprint arXiv:2010.08768. ↗
  14. Fouzi Harrag and Mohamed Khalil Djahli. 2022. Arabic fake news detection: A fact checking based deep learning approach. Transactions on Asian and Low-Resource Language Information Processing, 21(4):1– 34. ↗
  15. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  16. Linmei Hu, Siqi Wei, Ziwang Zhao, and Bin Wu. 2022a. Deep learning for fake news detection: A comprehensive survey. AI Open, 3:133–155. ↗
  17. LinMei Hu, SiQi Wei, Ziwang Zhao, and Bin Wu. 2022b. Deep learning for fake news detection: A comprehensive survey. AI Open. ↗
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  20. Muhammad F Mridha, Ashfia Jannat Keya, Md Abdul Hamid, Muhammad Mostafa Monowar, and Md Saifur Rahman. 2021. A comprehensive review on fake news detection with deep learning. IEEE Access, 9:156151–156170. ↗
  21. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  22. Ali Bou Nassif, Ashraf Elnagar, Omar Elgendy, and Yaman Afadar. 2022. Arabic fake news detection based on deep contextualized embedding models. Neural Computing and Applications, 34(18):16019–16032. ↗
  23. Rasha Obeidat, Maram Gharaibeh, Malak Abdullah, and Yara Alharahsheh. 2022. Multi-label multi-class covid-19 arabic twitter dataset with fine-grained misinformation and situational information annotations. PeerJ Computer Science, 8:e1151. ↗
  24. Ray Oshikawa, Jing Qian, and William Yang Wang. 2018. A survey on natural language processing for fake news detection. arXiv preprint arXiv:1811.00770. ↗
  25. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22–36. ↗
  26. Edson C Tandoc Jr, Zheng Wei Lim, and Richard Ling. 2018. Defining “fake news” a typology of scholarly definitions. Digital journalism, 6(2):137–153. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.
  29. Xichen Zhang and Ali A Ghorbani. 2020. An overview of online fake news: Characterization, detection, and discussion. Information Processing & Management, 57(2):102025. ↗
  30. Xinyi Zhou and Reza Zafarani. 2020. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Computing Surveys (CSUR), 53(5):1–40. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deka-2023-pd-ar-araieval-shared-task/