Elamur Research⌕
☰
home / papers / 2025 / Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

Summary

The paper addresses automatic moderation of Arabic online content, where dialect variety and scarce resources make detection hard. The authors tackle three tasks from the ArabicNLP MAHED 2025 shared task: (1) labelling Arabic text as hope, hate or not_applicable; (2) a multi-task text setup covering twelve-way emotion classification, offensive-language detection, and a hate/not_hate decision applied only to offensive items; (3) hateful-meme detection from an image plus its extracted Arabic text. They compare frozen embedding models feeding SVM or DNN classifiers, general-purpose safety moderators, prompted base LLMs, and fine-tuned LLMs. Fine-tuned GPT-4o-mini performed best on the text tasks, and fine-tuned Gemini Flash 2.5 performed best on memes. The reported macro F1 scores are 72.1%, 57.8% and 79.6% on the three tasks, which gave the team first place overall in the challenge. The study suggests that task-specific fine-tuning of large models clearly outperforms off-the-shelf safety classifiers for Arabic moderation.

Key points

  • Data comes from MAHED 2025. Task 1 has 9,843 / 1,476 / 1,477 train/val/test texts. Task 2 has 8,515 texts. Task 3 consists of image-plus-text memes, with 606 used for testing.
  • For Task 1, a majority vote was taken over three systems: fine-tuned GPT-4o-mini, an SVM trained on Google text embeddings, and a fine-tuned Gemini Flash 2.5 model. A dedicated hope/not_applicable GPT-4o-mini model was added because hope items kept being labelled not_applicable.
  • Task 2 used three separate fine-tuned GPT-4o-mini models (3 epochs, LR multiplier 1.8), with the minority "hate" class oversampled fivefold. Test macro F1 was 51.7% for emotion and 83.1% for offensive detection.
  • On memes, the generic safety tools trailed the other systems, with OpenAI content moderator at 51.75 macro F1 and Llama 4 Guard at 63.77. Fine-tuned Gemini Flash 2.5 led at 76.49, ahead of fine-tuned Llama 3.2 11B at 73.58, while PaliGemma2 lagged.
  • With Google multimodalembedding@001 (512-dim) + SVM, averaging the image and text vectors only slightly beat image alone, and text alone was weakest. The authors suspect the image encoder already reads the meme's printed text.

Interesting ideas

  • I1. When error analysis exposes one dominant confusion pair (hope vs. not_applicable), the authors attach a narrow binary specialist for just that pair instead of retraining the whole classifier. This is a cheap, targeted fix. — For Mova (learn): when evaluation shows a recurring confusion (e.g. two frame labels), train or induce a small pairwise repair rule that fires only on that pair, keeping the fix explainable.
  • I2. The task 2 cascade (offensive first, then hate only for offensive items) means first-stage mistakes affect the second stage. Some hate metrics could not even be computed, which shows that gated pipelines need stage-aware evaluation. — For Mova (learn): score each pipeline stage both on gold input and on predicted upstream input, so errors are attributed to the step that caused them.
  • I3. One meme prompt spells out what is not hate: critique aimed at a government or policy is excluded, while blaming a people as a whole counts. The deciding factor is the identity of the target, not the tone. — For Mova (frames): when classifying speech acts or irony, make the target role (institution vs. social group) an explicit frame slot that decides the label.
  • I4. The prompt with the best macro F1 was not the one with the best macro F2, so the chosen metric changes which variant looks best when recall on the rare class matters. — For Mova (learn): report recall-weighted scores alongside F1 for rare phenomena (e.g. implicature, irony) so tuning does not quietly favour precision.
  • I5. A modality ablation (image only, text only, averaged, concatenated) showed that the text channel added almost nothing, because the image encoder already absorbed it. — For Mova (learn): ablate each witness or input source in turn as a negative control, to check whether it contributes independent evidence or merely duplicates another.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.15810 · arXiv 2508.15810

@misc{aldahoul2025detecting,
  title = {{Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models}},
  author = {AlDahoul, Nouar and Zaki, Yasir},
  year = {2025},
  doi = {10.48550/arxiv.2508.15810},
  eprint = {2508.15810},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.15810}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mahed 2025: Multimodal detection of hope and hate emotions in arabic content. https://marsadlab.github.io/mahed2025/#. Accessed: 2025-08-01. ↗
  2. 2025. MAHED 2025: First multimodal shared task on hope and hate detection in arabic content. https://marsadlab.github.io/ mahed2025/. ↗
  3. Firoj Alam, Md Rafiul Biswas, Uzair Shah, Wajdi Zaghouani, and Georgios Mikros. 2024. Propaganda to hate: A multimodal analysis of arabic memes with multi-agent llms. In International Conference on Web Information Systems Engineering, pages 380–390. Springer. find our card
  4. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. Armeme: Propagandistic content in arabic memes. arXiv preprint arXiv:2406.03916. ↗
  5. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  6. Nouar AlDahoul, Myles Joshua Toledo Tan, Harishwar Reddy Kasireddy, and Yasir Zaki. 2024. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. ↗
  7. Nouar AlDahoul, Myles Joshua Toledo Tan, Harishwar Reddy Kasireddy, and Yasir Zaki. 2024. Exploring vision language models for facial attribute recognition: Emotion, race, gender, and age. arXiv preprint arXiv:2410.24148. ↗
  8. Nouar AlDahoul, Myles Joshua Toledo Tan, Raghava Reddy Tera, Hezerul Abdul Karim, Chee How Lim, Manish Kumar Mishra, and Yasir Zaki. 2025. Multitasking vision language models for vehicle plate recognition with vehiclepaligemma. Scientific Reports, 15(1):1–15. find
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. arxiv 2020. arXiv preprint arXiv:2003.00104. ↗ our card
  10. Eliza Aspen and Spandana Singh. Content moderation trends in the MENA region: Censorship, discrimination by design, and linguistic challenges. Middle East Institute. Published August 25, 2021. find
  11. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. icompass at arabic hate speech 2022: Detect hate speech using qrnn and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180. find
  12. Google Cloud. 2023. Gemini, google’s most capable model, is now available on vertex ai. https://cloud.google.com/blog/products/ ai-machine-learning/gemini-support-on-vertex-ai. ↗
  13. Google Cloud. 2025. Get multimodal embeddings. https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/ get-multimodal-embeddings. Accessed: 2025-07-12. ↗
  14. Google for Developer. 2025. State-of-the-art text embedding via the gemini api. https://developers.googleblog.com/en/ gemini-embedding-text-model-now-available-gemini-api/. Accessed: 2025-08-03. ↗
  15. Maram Hasanain, Md Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024. Araieval shared task: propagandistic techniques detection in unimodal and multimodal arabic content. arXiv preprint arXiv:2407.04247. ↗
  16. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. find
  17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  18. Sanaa Kaddoura, Suja A Alex, Maher Itani, Safaa Henno, Asma Al-Nashash, and D Jude Hemanth. 2023. Arabic spam tweets classification using deep learning. Neural Computing and Applications, 35(23):17233– 17246. find
  19. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking. ↗
  20. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521(7553):436–444. find
  21. Hanane Mohaouchane, Asmaa Mourhir, and Nikola S Nikolov. 2019. Detecting offensive language on arabic social media using deep learning. In 2019 sixth international conference on social networks analysis, management and security (SNAMS), pages 466–471. IEEE. find
  22. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436–1457. find our card
  23. Yetunde Salami. 2024. Ai content moderation: How it works and why it matters. Accessed: 2025-07-11. find
  24. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. Alexu-aic at arabic hate speech 2022: Contrast to classify. arXiv preprint arXiv:2207.08557. ↗
  25. A Steiner, AS Pinto, M Tschannen, D Keysers, X Wang, Y Bitton, A Gritsenko, M Minderer, A Sherbondy, S Long, et al. Paligemma 2: A family of versatile vlms for transfer, 2024. URL https://arxiv.org/abs/2412.03555. ↗
  26. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  27. Tech Global Institute. 2023. Content moderation in under-resourced regions: A case for arabic and hebrew. https://techglobalinstitute.com/announcements/blog/ content-moderation-arabic-hebrew-in-under-resourced-regions/. Blog post. ↗
  28. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148. ↗
  29. Wajdi Zaghouani and Md Rafiul Biswas. 2025. Emohopespeech: An annotated dataset of emotions and hope speech in english and arabic. arXiv preprint arXiv:2505.11959. ↗
  30. Wajdi Zaghouani, Hamdy Mubarak, and Md Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15044–15055. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-detecting-hope-hate-emotion-arabic/