Elamur Research⌕
☰
home / papers / 2024 / Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs

Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs

Summary

The paper asks whether propagandistic Arabic memes also tend to carry hate. Earlier work treated propaganda and hate in memes as separate problems, and Arabic multimodal resources are scarce. The authors take the ArAIEval-2024 propagandistic meme dataset (about 3k memes) and add two layers: a coarse hateful/not-hateful label and a fine-grained label. Hateful memes get one of eight attack types. Non-hateful memes are marked as humor, sarcasm or other. To label the data cheaply, three commercial LLMs act as independent zero-shot annotators, and GPT-4o acts as a consolidator that resolves their disagreements. Each agent is checked against a human-labelled test set. Claude 3.5 Sonnet agrees moderately with the human and is used to label the train and dev splits. In the test set, roughly a third of the propagandistic memes are hateful, so the two phenomena are linked but not the same. Text, image and fusion classifiers give baselines. Binary detection reaches about 0.70 macro-F1, while fine-grained hate classification remains very weak.

Key points

  • Base data: ArAIEval-2024 memes from Facebook, Twitter, Instagram and Pinterest, labelled about 40% propagandistic and 60% not. Text was extracted with EasyOCR, and the data is split 70/10/20.
  • Annotator agents were GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet, with GPT-4o as consolidator. The test set was labelled by a single experienced human annotator.
  • The labels are heavily skewed. Train has 212 hateful vs 1,931 not-hateful memes, and dev has 32 vs 280. In the test set, 56 of the 171 propagandistic memes are hateful.
  • Binary results:
    • AraBERT text model: accuracy ≈0.82, macro-F1 0.705.
    • ConvNeXt-tiny image model: macro-F1 0.669.
    • ConvNeXt + AraBERT fusion: macro-F1 0.709.
    • A balanced 500/500 subset with the fusion model also gives macro-F1 0.709.
  • Fine-grained results with the fusion model are poor. The eight-way hateful task scores only around 0.17–0.22, and the non-hateful task (humor vs sarcasm) reaches macro-F1 0.537.

Interesting ideas

  • I1. The annotation runs in two phases. Several models label independently, then a separate consolidator sees all their labels and makes the final call, mirroring a human annotate-then-adjudicate workflow. — For Mova (learn): The witness-parser silver pipeline could add an explicit adjudicator step for disagreeing parses, and log its reasons so each merged arc stays explainable.
  • I2. Each LLM annotator is scored against a small human-gold set before any of them is trusted to label the large training splits. This keeps the choice of annotator grounded in measured agreement. — For Mova (learn): Before promoting an LLM teacher tier to produce training data, measure it on a hand-checked held-out set, and keep that set fixed as a control.
  • I3. Labels are hierarchical: a coarse decision is made first, and the fine-grained label is chosen only within the branch that decision opens. Even non-offensive content gets subtypes such as humor vs sarcasm. — For Mova (frames): Speech-act and irony detection in prag could first decide literal vs non-literal, then pick sarcasm or humor only on that branch, which shrinks the label space at each step.
  • I4. The hate definition depends on the target: attacking a group that itself spreads hate, such as a terrorist organization, does not count. — For Mova (frames): Evaluative or hostile frames could carry the target's role and type as an explicit slot, so judgments depend on who is acted upon rather than on loaded words.
  • I5. Imbalance is checked by rerunning the model on a balanced subset. This tests whether the skew alone explains the scores. — For Mova (learn): Add a class-balanced rerun as a routine control in Mova evaluations, to separate real gains from effects of label frequency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.07246 · arXiv 2409.07246

@misc{alam2024propaganda,
  title = {{Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs}},
  author = {Alam, Firoj and Biswas, Md. Rafiul and Shah, Uzair and Zaghouani, Wajdi and Mikros, Georgios},
  year = {2024},
  doi = {10.48550/arxiv.2409.07246},
  eprint = {2409.07246},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.07246}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., Samih, Y.: Pre-Training BERT on Arabic tweets: Practical considerations (2021) find our card
  2. Alam, F., Cresci, S., Chakraborty, T., Silvestri, F., Dimitrov, D., Martino, G.D.S., Shaar, S., Firooz, H., Nakov, P.: A survey on multimodal disinformation detection. In: Proceedings of the 29th International Conference on Computational Linguistics. pp. 6625–6643 (Oct 2022) find
  3. Alam, F., Hasnat, A., Ahmed, F., Hasan, M.A., Hasanain, M.: ArMeme: Propagandistic content in arabic memes. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Dec 2024) find
  4. Alam, F., Mubarak, H., Zaghouani, W., Da San Martino, G., Nakov, P.: Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In: Proceedings of the Seventh Arabic Natural Language Processing Workshop (Dec 2022) Alam et al. find
  5. Alam, F., Mubarak, H., Zaghouani, W., Da San Martino, G., Nakov, P.: Overview of the wanlp 2022 shared task on propaganda detection in arabic. In: Proceedings of the Seventh Arabic Natural Language Processing Workshop. pp. 108–115 (2022) find
  6. Antoun, W., Baly, F., Hajj, H.: AraBERT: Transformer-based model for Arabic language understanding. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools. OSAC ’20, Marseille, France (2020) find our card
  7. Chen, J., Wu, Y., Yang, X., Liu, J., Yang, H., Wang, Z.: Multi-agent reinforcement learning for dynamic content moderation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 4536–4544 (2021) find
  8. Da San Martino, G., Barrón-Cedeño, A., Wachsmuth, H., Petrov, R., Nakov, P.: Semeval-2020 task 11: Detection of propaganda techniques in news articles. In: Proceedings of the 14th International Workshop on Semantic Evaluation. pp. 1377– 1414 (2020) find our card
  9. Da San Martino, G., Yu, S., Barrón-Cedeño, A., Petrov, R., Nakov, P.: Fine-grained analysis of propaganda in news article. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. pp. 5636–5646. ACL (Nov 2019) find
  10. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Proceedings of the International AAAI Conference on Web and Social Media. AAAI ’17, vol. 11 (2017) find our card
  11. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4171–4186. ACL (Jun 2019) find
  12. Dimitrov, D., Alam, F., Hasanain, M., Hasnat, A., Silvestri, F., Nakov, P., Da San Martino, G.: SemEval-2024 task 4: Multilingual detection of persuasion techniques in memes. In: Proceedings of the 18th International Workshop on Semantic Evaluation. pp. 2009–2026. ACL, Mexico City, Mexico (Jun 2024) find
  13. Dimitrov, D., Ali, B.B., Shaar, S., Alam, F., Silvestri, F., Firooz, H., Nakov, P., Da San Martino, G.: Detecting propaganda techniques in memes. In: Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. pp. 6603–6617. ACL-IJCNLP ’21 (2021) find
  14. Dimitrov, D., Bin Ali, B., Shaar, S., Alam, F., Silvestri, F., Firooz, H., Nakov, P., Da San Martino, G.: Semeval-2021 task 6: Detection of persuasion techniques in texts and images. In: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). pp. 70–98 (2021) find
  15. Fang, T., Xu, B., Zong, L.: Emotion-enhanced multi-modal persuasive techniques detection using split features. In: Fuzzy Systems and Data Mining VIII, pp. 172– 180. IOS Press (2022) find
  16. Fortuna, P., Nunes, S.: A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR) 51(4), 1–30 (2018) find
  17. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N.V., Wiest, O., Zhang, X.: Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024) ↗ our card
  18. Hasanain, M., Ahmed, F., Alam, F.: Large language models for propaganda span annotation. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Dec 2024) find
  19. Hasanain, M., Ahmed, F., Alam, F.: Can GPT-4 identify propaganda? annotation and detection of propaganda spans in news articles. In: Proceedings of the 2024 Propaganda to Hate: A Multimodal Analysis Joint International Conference On Computational Linguistics, Language Resources And Evaluation. LREC-COLING 2024, Torino, Italy (2024) find
  20. Hasanain, M., Alam, F., Mubarak, H., Abdaljalil, S., Zaghouani, W., Nakov, P., Da San Martino, G., Freihat, A.: ArAIEval shared task: Persuasion techniques and disinformation detection in Arabic text. In: Sawaf, H., El-Beltagy, S., Zaghouani, W., Magdy, W., Abdelali, A., Tomeh, N., Abu Farha, I., Habash, N., Khalifa, S., Keleg, A., Haddad, H., Zitouni, I., Mrini, K., Almatham, R. (eds.) Proceedings of ArabicNLP 2023. pp. 483–493. ACL, Singapore (Hybrid) (Dec 2023) find
  21. Hasanain, M., Hasan, M.A., Ahmed, F., Suwaileh, R., Biswas, M.R., Zaghouani, W., Alam, F.: ArAIEval Shared Task: Propagandistic techniques detection in unimodal and multimodal arabic content. In: Proceedings of the Second Arabic Natural Language Processing Conference. ACL, Bangkok (Aug 2024) find
  22. Hossain, E., Sharif, O., Hoque, M.M., Preum, S.M.: Deciphering hate: Identifying hateful memes and their targets. arXiv preprint arXiv:2403.10829 (2024) ↗
  23. Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D.: The hateful memes challenge: Detecting hate speech in multimodal memes. In: Advances in Neural Information Processing Systems. vol. 33, pp. 2611–2624 (2020) find our card
  24. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022) find
  25. Mathias, L., Nie, S., Mostafazadeh Davani, A., Kiela, D., Prabhakaran, V., Vidgen, B., Waseem, Z.: Findings of the WOAH 5 shared task on fine grained hateful memes detection. In: Mostafazadeh Davani, A., Kiela, D., Lambert, M., Vidgen, B., Prabhakaran, V., Waseem, Z. (eds.) Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021). pp. 201–206. ACL, Online (Aug 2021) find
  26. Mubarak, H., Abdaljalil, S., Nassar, A., Alam, F.: Detecting and identifying the reasons for deleted tweets before they are posted. Frontiers in Artificial Intelligence 6, 1219767 (2023) find
  27. OpenAI, R.: GPT-4 technical report. arXiv pp. 2303–08774 (2023) find our card
  28. Piskorski, J., Stefanovitch, N., Da San Martino, G., Nakov, P.: SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In: Proceedings of the 17th International Workshop on Semantic Evaluation. pp. 2343–2361. ACL, Toronto, Canada (Jul 2023) find
  29. Schmidt, A., Wiegand, M.: A survey on hate speech detection using natural language processing. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. pp. 1–10. ACL, Valencia, Spain (Apr 2017) find our card
  30. Shah, U., Biswas, M.R., Agus, M., Househ, M., Zaghouani, W.: MemeMind at ArAIEval shared task: Generative augmentation and feature fusion for multimodal propaganda detection in Arabic memes through advanced language and vision models. In: Proceedings of The Second Arabic Natural Language Processing Conference. pp. 467–472 (Aug 2024) find
  31. Sharma, S., Alam, F., Akhtar, M.S., Dimitrov, D., Da San Martino, G., Firooz, H., Halevy, A., Silvestri, F., Nakov, P., Chakraborty, T.: Detecting and understanding harmful memes: A survey. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. pp. 5597–5606. IJCAI (7 2022) find
  32. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alam-2024-propaganda-hate-multimodal-analysis-arabic/