Elamur Research⌕
☰
home / papers / 2023 / Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detection

Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detection

Paper · 2023 · Yunze Xiao and Firoj Alam

Summary

The paper describes the Nexus team's system for the ArAIEval shared task at ArabicNLP 2023, which targets automatic detection of manipulative Arabic content. The team entered two binary subtasks. Subtask 1A asks whether a tweet or news paragraph uses a persuasion (propaganda) technique. Subtask 2A asks whether a tweet is disinformative.

Their main approach fine-tuned three Arabic pre-trained transformers, AraBERT, MarBERT and Qarib, on the organizers' splits. To cope with strongly skewed label distributions, they used class-weighted cross-entropy and a tuned dropout layer. As a comparison point, they also prompted GPT-4 in zero-shot and 5-shot settings through the LLMeBench framework.

The submitted systems placed 9th in 1A (Micro F1 0.740) and 10th in 2A (Micro F1 0.893). Fine-tuned models clearly beat GPT-4 prompting on 1A, while few-shot GPT-4 came close on 2A. The work is a compact, reproducible baseline study for low-resource propaganda and disinformation detection. It shows how genre-matched pretraining, loss weighting and regularization choices affect results on imbalanced social-media data.

Key points

  • Models and settings:
    • Models were AraBERT, MarBERT and Qarib, with text preprocessed by the AraBERT preprocessor.
    • Hyperparameters were found by randomized search: max length 128, batch size 16, 3 epochs, learning rate 4e-5, AdamW.
    • The cross-entropy loss used class weights of [1.0, 4.0].
  • The data is heavily imbalanced, and the class balance shifts between splits:
    • In 1A, about 79% of training items carry a technique, but only 65.8% of test items do.
    • In 2A, only about a fifth of tweets are disinformative.
  • Subtask 1A results:
    • The submission scored Micro F1 0.740 and Macro F1 0.693.
    • MarBERT, which was pretrained on tweets, outperformed AraBERT, reaching dev Micro F1 0.841 at dropout 0.1.
  • Subtask 2A results:
    • The submission scored Micro F1 0.893 and Macro F1 0.845.
    • Qarib with dropout 0.2 did best in later runs (0.903 / 0.869).
  • GPT-4 (version 0314, temperature 0) results:
    • On 1A it reached only about 0.60–0.61 in both zero-shot and 5-shot settings.
    • On 2A, five examples lifted it from 0.759 to 0.852 Micro F1.

Interesting ideas

  • I1. Up-weighting the minority class inside cross-entropy is a cheap way to handle skewed labels without resampling the data.
  • I2. Runs with lower dropout reached lower final training loss yet scored worse on test, so falling loss alone is a misleading stopping or selection signal.
  • I3. A model pretrained on the same genre as the task (MarBERT on tweets) beat a more general one, so domain match of the encoder can matter more than its size.
  • I4. Running the same LLM at temperature zero through a fixed benchmarking harness gives a reproducible zero-shot and few-shot reference next to fine-tuned models.
  • I5. Label shares differed noticeably between train and test, which can distort micro-averaged scores and hide weak performance on the minority class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Weight rare UD relations or rare morphological tags more heavily when training the tagger and parser, so frequent labels do not swamp them.
  • I2 (learn): Choose parser checkpoints by held-out metrics and negative controls, never by training loss.
  • I3 (tree): When choosing or continuing pretraining for the gparse encoder, match it to the target text genres, for example fables or school texts, before scaling up.
  • I4 (learn): Compare LLM teacher tiers deterministically, with fixed prompts and a few shots, so their silver labels are repeatable and can be audited.
  • I5 (answer): Report per-class or macro scores, together with class balance per split, when evaluating readers on test sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Yunze Xiao and Firoj Alam. 2023. Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detection. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.58

@inproceedings{xiao2023nexus,
  title = {{Nexus at ArAIEval Shared Task: Fine-Tuning Arabic Language Models for Propaganda and Disinformation Detection}},
  author = {Yunze Xiao and Firoj Alam},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.58},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.58}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In this paper, we report on our participation in the ArAIEval 2023 shared task, which focuses on propaganda and disinformation detection. We experimented with various transformer-based models and fine-tuned them for our specific tasks. Despite challenges such as imbalanced data, we optimized our models and achieved commendable results. Our submitted system ranked 9th and 10th in subtasks 1A and 2A, respectively, on the leaderboard. In the future, our research will take advantage of the latest Large Language Models (LLMs) such as Llama, Alpaca, Bloom and more. We plan to do more experiment with data augmentation. find
  2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  3. Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2023. Benchmarking arabic ai with large language models. find
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  5. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  6. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022b. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics.
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  8. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. In IPM, 56(5):1849–1864. ↗
  9. Zakaria Boulouard, Mariya Ouaissa, Mariyam Ouaissa, Moez Krichen, Mutiq Almutiq, and Gasmi Karim. 2022. Detecting hateful and offensive speech in arabic social media using transfer learning. Applied Sciences, 12:12823. ↗
  10. Giovanni Da San Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the 14th Workshop on Semantic Evaluation, SemEval ’20, pages 1377–1414. ↗
  11. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In EMNLP-IJCNLP, pages 5636–5646. find
  12. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2023. LLMeBench: A flexible framework for accelerating llms benchmarking. arXiv:2308.04945. ↗
  13. W.B. Demilie and A.O. Salau. 2022. Detection of fake news and hate speech for ethiopian languages: a systematic review of the approaches. J Big Data. ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  15. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. Detecting propaganda techniques in memes. In ACL-IJCNLP, ACL-IJCNLP ’21, pages 6603–6617, Online. Association for Computational Linguistics. ↗
  16. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021b. Task 6 at SemEval-2021: Detection of persuasion techniques in texts and images. In SemEval. find
  17. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. CSUR, 51(4):1–30. ↗
  18. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023a. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  19. Maram Hasanain, Ahmed El-Shangiti, Rabindra Nath Nandi, Preslav Nakov, and Firoj Alam. 2023b. QCRI at SemEval-2023 task 3: News genre, framing and persuasion techniques detection using multilingual models. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1237–1244, Toronto, Canada. Association for Computational Linguistics. find
  20. Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. A survey on truth discovery. ACM Sigkdd Explorations Newsletter, 17(2):1–16. ↗
  21. Clyde R. Miller. 1939. The Techniques of Propaganda. pages 27–29. find
  22. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  23. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection, pages 48–52.
  24. R OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.
  25. Ray Oshikawa, Jing Qian, and William Yang Wang. 2018. A survey on natural language processing for fake news detection. arXiv preprint arXiv:1811.00770. ↗
  26. Ray Oshikawa, Jing Qian, and William Yang Wang. 2020. A survey on natural language processing for fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference, LREC ’20, pages 6086–6093. find
  27. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937. Association for Computational Linguistics. Shivam Sharma, Firoj Alam, Md.
  28. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5597–5606. International Joint Conferences on Artificial Intelligence Organization. Survey Track. find
  29. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In NAACL-HLT, pages 1415– 1420.
  30. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In SemEval, pages 1425–1447. ↗
  31. Lu Zhou, Wenbo Wang, and Keke Chen. 2016. Tweet properly: Analyzing deleted tweets to understand and identify regrettable ones. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 603–612, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2023-nexus-araieval-shared-task-fine/