Elamur Research⌕
☰
home / papers / 2024 / ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content

ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content

Paper · 2024 · Maram Hasanain, Md. Arid Hasan, Fatema Ahmad et al.

Summary

This overview covers the second edition of ArAIEval, a shared task at ArabicNLP 2024 on finding propaganda in Arabic content. It had two tracks. Task 1 asked systems to locate text spans in tweets and news paragraphs and label each span with one or more of 23 persuasion techniques, framed as multilabel sequence tagging. Task 2 was a binary decision on whether a meme is propagandistic, run separately on text only, image only, and text plus image.

The tweets came from Arabic news accounts, plus a new set about events in Gaza. The news paragraphs came from AraFacts and a large in-house news archive. Annotation followed the ArPro pipeline. The task drew 45 registrations, and 14 teams submitted final runs. Most systems fine-tuned Arabic transformers such as AraBERT, MARBERT or CAMeLBERT, and vision systems mostly used ResNet. Every system beat the random baselines. Image-only detection was the weakest setting. The datasets and scorers are public, which gives a shared benchmark for fine-grained persuasion analysis in Arabic.

Key points

  • Participation: 6 teams in Task 1 and 9 in Task 2. Of the 14 final participants, 11 wrote system papers.
  • The Task 1 taxonomy has 23 techniques, taken from SemEval-2023 Task 3. The label distribution is very skewed: Loaded Language has 8,779 training spans, while Straw Man and Guilt by Association have 23 each.
  • News sources were AraFacts (kept only news-media pages) and an in-house collection of 600K articles from more than 400 outlets, covering 14 broad topics.
  • Annotation had two steps. Three annotators labelled spans independently, then two expert consolidators discussed each item and fixed the gold labels.
  • Task 2 was scored with macro F1. Best results per subtask:
    • text only (2A): 0.787 against a 0.453 baseline;
    • image only (2B): 0.711 by CLTL against 0.475;
    • multimodal (2C): 0.805 by AlexUNLP-MZ against 0.493.

Interesting ideas

  • I1. Persuasion is annotated as typed, possibly overlapping spans rather than as a label for the whole text, so one sentence can carry several rhetorical devices at different positions.
  • I2. Labels are built in two stages: several people annotate independently, then experts settle disagreements in discussion, and a span marked by only one annotator can still survive.
  • I3. The same binary question is posed on text alone, image alone and the fused input, which shows how much each channel contributes.
  • I4. Every result is reported next to a random baseline, and macro F1 is the official measure, so rare classes count as much as frequent ones.
  • I5. Candidate news text is filtered by source type, keeping only pages from news outlets among the links attached to fact-checked claims.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): rhetorical-technique tags could be attached to subtrees in the frame layer, next to speech acts and irony. This would explain which phrase does the persuading and how.
  • I2 (learn): this mirrors witness-parser agreement. Cases where the parsers disagree, or where only one witness fires, could go to a higher LLM-teacher tier for adjudication instead of being dropped.
  • I3 (learn): we could evaluate answers with only some pipeline stages enabled, for example tree without world, to measure what each explainable step actually adds.
  • I4 (learn): we could pair our negative controls with macro-averaged scores over rare UD relations or frame types, so that frequent labels cannot hide failures on rare ones.
  • I5 (world): claims linked to fact-check verdicts could serve as test material for tracking states and truth values in a text's world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Maram Hasanain, Md. Arid Hasan, Fatema Ahmad, Reem Suwaileh, Md. Rafiul Biswas, Wajdi Zaghouani and Firoj Alam. 2024. ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.44

@inproceedings{hasanain2024araieval,
  title = {{ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content}},
  author = {Maram Hasanain, Md. Arid Hasan, Fatema Ahmad, Reem Suwaileh, Md. Rafiul Biswas, Wajdi Zaghouani and Firoj Alam},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.44},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.44}
}

References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Kemal Oflazer Abrar Abir. 2024. Nullpointer at ArAIEval Shared Task: Arabic propagandist technique detection with token-to-word mapping in sequence tagging. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics. ↗
  3. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, EMNLP-IJCNLP 2019, Hong Kong, China. ↗
  4. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. ArMeme: Propagandistic content in arabic memes. arXiv: 2406.03916. ↗
  5. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Preslav Nakov, and Giovanni Da San Martino. 2022b. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, WANLP ’22, Abu Dhabi, UAE.
  6. Yasser Alhabashi, Serry Sibaee, Abdullah Alharbi, Omar Najar, and Samar Ahmed. 2024. ASOS at ArAIEval Shared Task: persuasion techniques detection: an interdisciplinary approach to identifying and counteracting manipulative strategies. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics. find
  7. Zien Sheikh Ali, Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2021. AraFacts: the first large arabic dataset of naturally occurring claims. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 231–236.
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  9. A Barrón-Cedeno et al. Overview of the CLEF–2022 CheckThat! Lab on fighting the covid-19 infodemic and fake news detection. Experimental IR Meets Multilinguality, Multimodality, and Interaction, page 495. Md. find
  10. Rafiul Biswas, Zubair Shah, and Wajdi Zaghouani. 2024. MemeMind at ArAIEval Shared Task: spotting persuasive spans in arabic text with persuasion techniques identification. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  11. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM international conference on multimodal interaction, pages 163–171. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’19, pages 4171–4186, Minneapolis, MN, USA. Association for Computational Linguistics.
  13. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 Task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. ↗
  14. Taosong Fang, Bo Xu, and Linlin Zong. 2022a. Emotion-enhanced multi-modal persuasive techniques detection using split features. In Fuzzy Systems and Data Mining VIII, pages 172–180. IOS Press. ↗
  15. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2022b. EVA: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636. ↗
  16. Mary Fouad and Julie Weeds. 2024. SussexAI at ArAIEval Shared Task: mitigating class imbalance in arabic propaganda detection. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics. ↗
  17. Ivan Habernal and Iryna Gurevych. 2015. Exploiting debate portals for semi-supervised argumentation mining in user-generated web discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2127–2137, Lisbon, Portugal. Association for Computational Linguistics. find
  18. Pengyuan Chen, Lei Zhao, Yangheran Piao, Hongwei Ding, and Xiaohui Cui. 2023. Multimodal visual-textual object graph attention network for propaganda detection in memes. Multimedia Tools and Applications, pages 1–16. ↗
  19. Abdelhamid Haouhat, Hadda Cherroun, Slimane Bellaouar, and Attia Nehar. 2024. MODOS at ArAIEval Shared Task: Multimodal propagandistic memes classification using weighted sam, clip and arabiangpt. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  20. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023a. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812. ↗
  21. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024. Can GPT-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024. find
  22. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Freihat. 2023b. ArAIEval Shared Task: Persuasion techniques and disinformation detection in Arabic text. In Proceedings of ArabicNLP 2023, pages 483–493, Singapore (Hybrid). Association for Computational Linguistics.
  23. Christopher Hidey, Elena Musi, Alyssa Hwang, Smaranda Muresan, and Kathy McKeown. 2017. Analyzing the semantic types of claims and premises in an online persuasive forum. In Proceedings of the 4th Workshop on Argument Mining. Columbia Univ., New York, NY (United States).
  24. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104.
  25. IPA. 1938. In Institute for Propaganda Analysis, editor, Propaganda Analysis. Volume I of the Publications of the Institute for Propaganda Analysis. New York, NY, USA. find
  26. Momtazul Labib, Samia Rahman, Hasan Murad, and Udoy Das. 2024. CUET_sstm at ArAIEval Shared Task: unimodal (text) propagandistic technique detection using transformer-based model. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  27. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  28. Clyde R. Miller. 1939. The Techniques of Propaganda. From “How to Detect and Analyze Propaganda,” an address given at Town Hall. The Center for learning. find
  29. Fred Morstatter, Yunqiu Shao, Aram Galstyan, and Shanika Karunasekera. 2018. From alt-right to alt-rechts: Twitter analysis of the 2017 german federal election. In Companion Proceedings of the The Web Conference 2018, pages 621–628. find
  30. Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. 2016. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM international conference on multimodal interaction, pages 284–288. find
  31. OpenAI. 2023. GPT-4 technical report. Technical report, OpenAI. ↗
  32. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2016. Multimodal analysis and prediction of persuasiveness in online social multimedia. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(3):1–25. ↗
  33. Andrew Perrin. 2015. Social media usage. Pew research center, pages 52–68. find
  34. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. SemEval-2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361, Toronto, Canada. Association for Computational Linguistics. find
  35. Md Abdur Razzaq Riyadh and Sara Nabhani. 2024. Mela at ArAIEval Shared Task: propagandistic techniques detection in arabic with a multilingual approach. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  36. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. Uzair Shah, Md.
  37. Rafiul Biswas, Marco Agus, Mowafa Househ, and Wajdi Zaghouani. 2024. MemeMind at ArAIEval Shared Task: generative augmentation and feature fusion for multimodal propaganda detection in arabic memes through advanced language and vision models. In Proceedings of The Second Arabic Natural Language Processing Conference (Arabic-NLP 2024), Bangkok. Association for Computational Linguistics. Shivam Sharma, Firoj Alam, Md.
  38. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI ’22, pages 5597–5606, Vienna, Austria. International Joint Conferences on Artificial Intelligence Organization. Survey Track. Symom Hossain Shohan, Md. find
  39. Sajjad Hossain, Ashraful Islam Paran, Shawly Ahsan, Jawad Hossain, and Mohammed Moshiul Hoque. 2024. SemanticCuet-Sync at ArAIEval Shared Task: finetuning transformer models for propagandistic technique detection and span prediction. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics. find
  40. Yeshan Wang and Ilia Markov. 2024. CLTL at ArAIEval Shared Task: multimodal propagandistic memes classification using transformer models. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics. ↗
  41. Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. 2023. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1–17. ↗
  42. Mohamed Zaytoon, Nagwa ElMakky, and Marwan Torki. 2024. AlexUNLP-MZ at ArAIEval Shared Task: contrastive learning, llm features extraction and multi-objective optimization for arabic multi-modal meme propaganda detection. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hasanain-2024-araieval-shared-task-propagandistic-techniques/