ArMeme: Propagandistic Content in Arabic Memes
Paper · 2024 · Firoj Alam, Abul Hasnat, Fatema Ahmad et al.
Summary
Memes mix pictures and text and often carry persuasive or misleading messages. Tools to detect this exist mainly for English, and before this work there was no annotated meme resource for Arabic. The authors built ArMeme, a set of about 6K Arabic memes, each manually labeled as propaganda, not-propaganda, not-meme, or other. The images were gathered from public Facebook, Instagram and Pinterest groups and from keyword-based Twitter searches. A filtering pipeline then removed near-duplicates, images with no readable text, and non-meme pictures before humans annotated what remained.
The paper benchmarks text-only models (n-grams and Arabic and multilingual transformers), image-only CNNs, an early-fusion multimodal model, and zero-shot multimodal LLMs. Fine-tuned models win overall. Qarib on OCR text gives the best weighted F1 (0.690). Zero-shot GPT-4/Gemini setups trail behind. The dataset and annotation guideline are public, giving Arabic multimodal propaganda research its first benchmark.
Key points
- Data collection and filtering: about 129.5K images were collected initially and reduced to about 6K annotated memes.
- Near-duplicates were found with fine-tuned ResNet18 features and a Euclidean distance threshold of 3.6.
- EasyOCR extracted the text, and memes with no detected text were dropped.
- An in-house meme classifier, trained on 3,935 images, removed non-memes.
- Text results: Qarib reaches 0.690 weighted F1 and AraBERT 0.666. Monolingual Arabic transformers beat mBERT and XLM-RoBERTa. The majority baseline scores 0.522.
- Image and multimodal results:
- Among image models, ResNet50 is the strongest fine-tuned CNN.
- Early fusion with ConvNeXt + AraBERT + SVM reaches 0.659 weighted F1.
- Zero-shot multimodal LLMs score lower, with Gemini clearly behind the GPT-4 variants.
- Binary ArAIEval task: on the propaganda vs not-propaganda task (3,062 memes), image-only GPT-4o scores 0.849 weighted F1. This far exceeds GPT-4 Turbo and Gemini Pro.
- Release: the dataset is available as QCRI/ArMeme on Hugging Face under CC-BY-NC-SA.
Interesting ideas
- I1. Near-duplicate removal uses a pretrained network's embeddings with a fixed distance cutoff, which catches lightly edited reposts that exact hashing misses.
- I2. Cheap automatic filters (OCR presence, a small meme/non-meme classifier) are chained before costly human labeling, so annotators only see plausible items.
- I3. The label set includes explicit escape classes ("not-meme", "other") instead of forcing every input into propaganda vs not-propaganda.
- I4. The zero-shot prompt asks the model for a short description plus a label in strict JSON, which yields a rationale and removes the need for post-processing.
- I5. Results are reported against majority and random baselines with both weighted and macro F1, showing that gains over the majority class are modest and that minority classes stay weak.
For Mova
How these ideas could serve Mova, our small open language model — its files, stages and tests:
- I1 (learn): Embedding-distance dedup of sentences or documents could keep near-copies of test items out of silver training data and stop evaluation scores from being inflated by leakage.
- I2 (learn): A similar cascade could screen candidate sentences for LLM-teacher or human review, sending only parseable, in-domain text to the expensive tiers.
- I3 (answer): Readers could get an explicit "out of scope / cannot judge" outcome, so they refuse cleanly instead of guessing when a question falls outside the text's world.
- I4 (learn): LLM teacher tiers could return a brief justification alongside each silver label in a fixed schema, making disagreements with witness parsers easy to inspect and explain.
- I5 (learn): Every evaluation could include such trivial baselines as negative controls and report macro scores, so improvements on rare constructions are not hidden by frequent ones.
Institutions
Fields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works
🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works
Cite
Cite as Firoj Alam, Abul Hasnat, Fatema Ahmad, Md. Arid Hasan and Maram Hasanain. 2024. ArMeme: Propagandistic Content in Arabic Memes. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
@inproceedings{alam2024armeme,
title = {{ArMeme: Propagandistic Content in Arabic Memes}},
author = {Firoj Alam, Abul Hasnat, Fatema Ahmad, Md. Arid Hasan and Maram Hasanain},
booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
year = {2024},
doi = {10.18653/v1/2024.emnlp-main.1173},
url = {https://doi.org/10.18653/v1/2024.emnlp-main.1173}
}References
- Shamsiah Abd Kadir, Anitawati Lokman, and T. Tsuchiya. 2016. Emotion and techniques of propaganda in YouTube videos. Indian Journal of Science and Technology, Vol (9). ↗
- Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
- Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520, St. Julian’s, Malta. Association for Computational Linguistics. ↗
- Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
- Firoj Alam, Fahim Dalvi, Shaden Shaar, Nadir Durrani, Hamdy Mubarak, Alex Nikolov, Giovanni Da San Martino, Ahmed Abdelali, Hassan Sajjad, Kareem Darwish, et al. 2021. Fighting the covid-19 infodemic in social media: a holistic perspective and a call to arms. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 913–922. ↗
- Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022b. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE.
- Firoj Alam, Ferda Ofli, Muhammad Imran, Tanvirul Alam, and Umair Qazi. 2020. Deep learning benchmarks and datasets for social media image classification for disaster response. In 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 151– 158. IEEE. find
- Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
- Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. ↗
- Sian Brooke. 2019. “condescending, rude, assholes”: Framing gender and hostility on stack overflow. In Proceedings of the Third Workshop on Abusive Language Online, pages 172–180, Florence, Italy. Association for Computational Linguistics. find
- Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022. Prompting for multimodal hateful meme classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 321–332, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
- Pengyuan Chen, Lei Zhao, Yangheran Piao, Hongwei Ding, and Xiaohui Cui. 2024. Multimodal visual-textual object graph attention network for propaganda detection in memes. Multimedia Tools and Applications, 83(12):36629–36644. ↗
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv:1911.02116. ↗
- Padraig Cunningham and Sarah Jane Delany. 2007. k-nearest neighbour classifiers. Multiple Classifier Systems, 34(8):1–17. find
- Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. Findings of the NLP4IF-2019 shared task on fine-grained propaganda detection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, NLP4IF ’19, pages 162–170, Hong Kong, China. ↗
- Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 5636–5646, Hong Kong, China. ↗
- Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A flexible framework for accelerating LLMs benchmarking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 214–222, St. Julians, Malta. Association for Computational Linguistics. ↗
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11 of AAAI ’17. ↗
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of the 2019 Conference of the NAACL, NAACL-HLT ’19, Minneapolis, MN, USA.
- Dimitar Dimitrov, Firoj Alam, Maram Hasanain, Abul Hasnat, Fabrizio Silvestri, Preslav Nakov, and Giovanni Da San Martino. 2024. Semeval-2024 task 4: Multilingual detection of persuasion techniques in memes. In Proceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
- Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021a. Detecting propaganda techniques in memes. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP ’21, pages 6603–6617. ↗
- Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021b. Task 6 at SemEval-2021: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation, SemEval ’21, Bangkok, Thailand. find
- Institute for Propaganda Analysis. 1938. In Institute for Propaganda Analysis, editor, Propaganda Analysis. Volume I of the Publications of the Institute for Propaganda Analysis, chapter 2. New York, NY. find
- Shreyansh Gandhi, Samrat Kokkula, Abon Chaudhuri, Alessandro Magnani, Theban Stanley, Behzad Ahmadi, Venkatesh Kandaswamy, Omer Ovenc, and Shie Mannor. 2020. Scalable detection of offensive and non-compliant content / logo in product images. WACV, pages 2236–2245. find
- Maria Glenski, E. Ayton, J. Mendoza, and Svitlana Volkova. 2019. Multilingual multimodal digital deception detection and disinformation spread across social platforms. ArXiv, abs/1909.05838. ↗
- Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Exploring hate speech detection in multimodal publications. In WACV, pages 1470–1478. find
- Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP ’17, pages 7–12, Copenhagen, Denmark. find
- Ivan Habernal, Patrick Pauli, and Iryna Gurevych. 2018. Adapting serious game for fallacious argumentation to German: Pitfalls, insights, and best practices. In LREC. European Language Resources Association (ELRA). find
- Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can gpt-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024, Torino, Italy. find
- Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Freihat. 2023. ArAIEval shared task: Persuasion techniques and disinformation detection in Arabic text. In Proceedings of ArabicNLP 2023, pages 483–493, Singapore (Hybrid). Association for Computational Linguistics.
- J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md. ↗
- Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. Araieval shared task: Propagandistic techniques detection in unimodal and multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
- Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. 2022. Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE transactions on pattern analysis and machine intelligence, 45(1):919–931. ↗
- Abul Hasnat, Nadiya Shvai, Assan Sanogo, Marouan Khata, Arcadi Llanza, Antoine Meicler, and Amir Nakib. 2019. Application guided image quality estimation based on classification. In 2019 IEEE International Conference on Image Processing (ICIP), pages 549–553. IEEE. find
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR ’16, pages 770–778. IEEE. ↗
- Ming Shan Hee, Shivam Sharma, Rui Cao, Palash Nandi, Preslav Nakov, Tanmoy Chakraborty, and Roy Ka-Wei Lee. 2024. Recent advances in hate speech moderation: Multimodality and the role of large models. arXiv preprint arXiv:2401.16727. ↗
- Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861. ↗
- Srecko Joksimovic, Ryan S. Baker, Jaclyn Ocumpaugh, Juan Miguel L. Andres, Ivan Tot, Elle Yuan Wang, and Shane Dawson. 2019. Automated identification of verbally abusive behaviors in online discussions. In Proceedings of the Third Workshop on Abusive Language Online, pages 36–45, Florence, Italy. Association for Computational Linguistics. find
- Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. 2019. Supervised multimodal bitransformers for classifying images and text. In Proceedings of the NeurIPS 2019 Workshop on Visually Grounded Interaction and Language, ViGIL@NeurIPS ’19. ↗
- Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In Proceedings of the Annual Conference on Neural Information Processing Systems, NeurIPS ’20. ↗
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. ↗
- Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557. ↗
- Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986. ↗
- Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the Conference on Neural Information Processing Systems, NeurIPS ’19, Vancouver, Canada.
- R OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.
- Alan Partington and Charlotte Taylor. 2017. The language of persuasion in politics: An introduction. Routledge. find
- Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023a. SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361, Toronto, Canada. Association for Computational Linguistics.
- Jakub Piskorski, Nicolas Stefanovitch, Nikolaos Nikolaidis, Giovanni Da San Martino, and Preslav Nakov. 2023b. Multilingual multifaceted understanding of online news in terms of genre, framing, and persuasion techniques. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3001–3022, Toronto, Canada. Association for Computational Linguistics. find
- Nirmalendu Prakash, Han Wang, Nguyen Khoi Hoang, Ming Shan Hee, and Roy Ka-Wei Lee. 2023. PromptMTopic: Unsupervised multimodal topic modeling of memes using large language models. In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 621–631, New York, NY, USA. Association for Computing Machinery. find
- Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain. Association for Computational Linguistics.
- Hyunjin Seo. 2014. Visual propaganda in the age of social media: An empirical analysis of Twitter images during the 2012 Israeli–Hamas conflict. Visual Communication Quarterly, 21(3). Shivam Sharma, Firoj Alam, Md. ↗
- Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI ’22, pages 5597–5606, Vienna, Austria. International Joint Conferences on Artificial Intelligence Organization. Survey Track. find
- Limor Shifman. 2013. Memes in digital culture. MIT press. ↗
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. ↗
- Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mihael Arcan, and Paul Buitelaar. 2020. Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In TRAC, pages 32–41.
- Mingxing Tan and Quoc V Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv:1905.11946. ↗
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
- Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Veronique Hoste. 2015. Detection and fine-grained classification of cyberbullying events. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 672–680, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA. find
- Svitlana Volkova, Ellyn Ayton, Dustin L. Arendt, Zhuanyi Huang, and Brian Hutchinson. 2019. Explaining multimodal deceptive news prediction models. In Proceedings of the International Conference on Web and Social Media, ICWSM ’19, Munich, Germany. find
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. arXiv:abs/1910.03771. ↗
- Ching Seh Wu and Unnathi Bhandary. 2020. Detection of hate speech in videos using machine learning. In CSCI, pages 585–590. ↗
- Peng Xu, Xiatian Zhu, and David A Clifton. 2023. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. ↗
- Prosper Nunayon Zannu, Felix Olajide Talabi, Bernice Oluwalanu Sanusi, Oluwakemi A. Adesina, Adebola Adewunmi Aderibigbe, Omowale T. Adelabu, Oloyede Oyinloye, and Samson Adedapo Bello. 2024. Influence of media literacy on the dissemination of fake news among instagram and twitter users. International Research Journal of Multidisciplinary Scope (IRJMS), 5(2):246–255. Original Article. ↗
- Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/alam-2024-armeme-propagandistic-content-arabic-memes/