Elamur Research⌕
☰
home / papers / 2025 / NYUAD at MAHED Shared Task: Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

NYUAD at MAHED Shared Task: Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

Paper · 2025 · Nouar AlDahoul and Yasir Zaki

Summary

The paper describes the NYUAD system for the MAHED 2025 shared task on moderating Arabic online content. It covers three tasks: classifying Arabic text as hope, hate or neither; a multi-task setup with twelve emotions, offensive language and hate; and finding hateful Arabic memes that combine an image with its extracted text.

The authors compare four families of approaches: frozen embedding models feeding SVM or DNN classifiers, off-the-shelf safety moderators, prompted base LLMs, and fine-tuned LLMs. Fine-tuned commercial models did best. GPT-4o-mini led on the text tasks, helped by majority-vote ensembling and an extra binary classifier for hope. Gemini Flash 2.5, fine-tuned on oversampled hateful memes, led on memes. Their macro F1 scores were 72.1%, 57.8% and 79.6% on tasks 1–3, which placed them first overall.

The study also shows that generic safety tools handle Arabic meme hate poorly. Supervised adaptation to the target language and genre clearly helped.

Key points

  • Data. Task 1 has 9,843 / 1,476 / 1,477 train/val/test texts. Task 2 has 8,515 texts with three sub-tasks. Task 3 has 4,500 memes, of which 2,143 train, 312 val and 606 test were used.

  • Task 1 (hope/hate/not_applicable). Three models were combined by majority vote:

    • fine-tuned GPT-4o-mini;
    • fine-tuned Gemini Flash 2.5;
    • Google text embedding + SVM.

    A dedicated hope vs not_applicable GPT-4o-mini was then added on top. This raised macro F1 from 71.4% (ensemble) to 72.1%, which ranked second on that task.

  • Task 2 (emotion, offensive, hate). Three GPT-4o-mini models were fine-tuned, with the hate class oversampled ×5. Hate detection runs only on texts predicted offensive. Macro F1 was 51.7% for emotion and 83.1% for offensive, with 57.8% overall.

  • Task 3 (memes, local test). Fine-tuned Gemini Flash 2.5 (hate oversampled ×9) reached 76.49% macro F1, ahead of fine-tuned Llama 3.2 11B with LoRA (73.58%). Off-the-shelf moderators did poorly: Llama Guard 4 scored 63.77% and the OpenAI content moderator 51.18%.

  • Embedding modalities. With Google multimodalembedding@001 (512-dim) + SVM, averaging the image and text vectors (70.65%) barely beat image-only (70.34%). Text-only was weakest at 57.33%.

Interesting ideas

  • I1. When error analysis shows one class consistently collapsing into another (hope predicted as not_applicable), a narrow binary specialist trained just on that pair can fix it after the main ensemble has voted. This targeted repair is cheaper than retraining everything.
  • I2. Hierarchical labels can be modelled as a cascade, where the hate decision applies only to texts already judged offensive. The paper is explicit that this makes some metrics for the downstream stage undefined and lets upstream errors propagate.
  • I3. A modality ablation showed that adding text to the image embedding gave almost no gain. The likely reason is that the image encoder already reads the words printed on the meme, so the "extra" signal was redundant.
  • I4. Ready-made general safety classifiers were run as baselines and turned out weak on an under-resourced language and genre. This argues for always measuring off-the-shelf tools on in-domain data before trusting them.
  • I5. Prompts were compared with both F1 and F2. The prompt with the best F1 was not the one with the best recall-weighted F2, so the choice depends on which class matters more.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could watch its confusion matrices for recurring label swaps (e.g. two VerbNet roles or two dependency relations) and train or induce a small pairwise repair rule that runs only on the affected outputs.
  • I2 (frames): Speech-act and irony detection could be gated the same way (e.g. irony checked only for evaluative utterances), with the evaluation reporting the gate's errors separately from the conditional stage.
  • I3 (learn): When Mova combines several witnesses or feature sources, ablating each one acts as a negative control that reveals hidden overlap, so agreement between two redundant witnesses is not counted as independent evidence.
  • I4 (answer): Before relying on an external LLM or tool as an answer checker for school tests or fable QA, Mova should benchmark it on a small in-domain set rather than assume general competence transfers.
  • I5 (learn): When choosing LLM-teacher prompts for silver data, Mova could pick by a metric weighted toward the rare but important labels it most needs to recover, not only overall accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Nouar AlDahoul and Yasir Zaki. 2025. NYUAD at MAHED Shared Task: Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.76

@inproceedings{aldahoul2025nyuad,
  title = {{NYUAD at MAHED Shared Task: Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models}},
  author = {Nouar AlDahoul and Yasir Zaki},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.76},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.76}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Md Rafiul Biswas, Uzair Shah, Wajdi Zaghouani, and Georgios Mikros. 2024a. Propaganda to hate: A multimodal analysis of arabic memes with multi-agent llms. In International Conference on Web Information Systems Engineering, pages 380– 390. Springer.
  2. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024b. Armeme: Propagandistic content in arabic memes. arXiv preprint arXiv:2406.03916. ↗
  3. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detec-Accuracy % Macro Macro Macro Macro Pre-Re-F1 F2 ci-call score score sion % % % % 74.75 70.70 76.23 71.34 73.62 79.21 72.49 71.29 71.84 71.50 82.51 80.86 69.44 72.29 70.14 *Table 5: GPT-4o-mini base model performance under different prompts evaluated on validation set in task 3.* tion of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  4. Nouar AlDahoul, Myles Joshua Toledo Tan, Har-ishwar Reddy Kasireddy, and Yasir Zaki. 2024a. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123. ↗
  5. Nouar AlDahoul, Myles Joshua Toledo Tan, Harish-war Reddy Kasireddy, and Yasir Zaki. 2024b. Exploring vision language models for facial attribute recognition: Emotion, race, gender, and age. arXiv preprint arXiv:2410.24148. ↗
  6. Nouar AlDahoul, Myles Joshua Toledo Tan, Raghava Reddy Tera, Hezerul Abdul Karim, Chee How Lim, Manish Kumar Mishra, and Yasir Zaki. 2025. Multitasking vision language models for vehicle plate recognition with vehiclepaligemma. Scientific Reports, 15(1):1–15. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  8. Mohamed Aziz Bennessir, Malek Rhouma, Hatem Haddad, and Chayma Fourati. 2022. icompass at arabic hate speech 2022: Detect hate speech using qrnn and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 176–180.
  9. Jianfeng Chi, Ujjwal Karn, Hongyuan Zhan, Eric Smith, Javier Rando, Yiming Zhang, Kate Plawiak, Zacharie Delpierre Coudert, Kartikeya Upasani, and Mahesh Pasupuleti. 2024. Llama guard 3 vision: Safeguarding human-ai image understanding conversations. ↗
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  11. Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. 2024. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555. ↗
  12. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  13. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148. ↗
  14. Wajdi Zaghouani and Md Rafiul Biswas. 2025a. An annotated corpus of arabic tweets for hate speech analysis. arXiv preprint arXiv:2505.11969. ↗
  15. Maram Hasanain, Md Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024. Araieval shared task: propagandistic techniques detection in unimodal and multimodal arabic content. arXiv preprint arXiv:2407.04247. ↗
  16. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. ↗
  17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  18. Sanaa Kaddoura, Suja A Alex, Maher Itani, Safaa Henno, Asma AlNashash, and D Jude Hemanth. 2023. Arabic spam tweets classification using deep learning. Neural Computing and Applications, 35(23):17233–17246. ↗
  19. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521(7553):436–444. find
  20. Hanane Mohaouchane, Asmaa Mourhir, and Nikola S Nikolov. 2019. Detecting offensive language on arabic social media using deep learning. In 2019 sixth international conference on social networks analysis, management and security (SNAMS), pages 466–471. IEEE. find
  21. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436–1457. ↗
  22. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. Alexu-aic at arabic hate speech 2022: Contrast to classify. arXiv preprint arXiv:2207.08557. ↗
  23. Wajdi Zaghouani and Md Rafiul Biswas. 2025b. Emohopespeech: An annotated dataset of emotions and hope speech in english and arabic. arXiv preprint arXiv:2505.11959. ↗
  24. Wajdi Zaghouani, Md Rafiul Biswas, Mabrouka Bessghaier, Shimaa Ibrahim, Georgios Mikros, Abul Hasnat, and Firoj Alam. 2025. MAHED shared task: Multimodal detection of hope and hate emotions in arabic content. In Proceedings of the Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics.
  25. Wajdi Zaghouani, Hamdy Mubarak, and Md Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15044–15055.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldahoul-2025-nyuad-mahed-shared-task-detecting/