Elamur Research⌕
☰
home / papers / 2025 / YassirEA at MAHED 2025: Fusion-Based Multimodal Models for Arabic Hate Meme Detection

YassirEA at MAHED 2025: Fusion-Based Multimodal Models for Arabic Hate Meme Detection

Paper · 2025 · El Attar, Yassir

Summary

The paper describes a system for subtask 3 of the MAHED 2025 shared task: deciding whether an Arabic meme (an image with embedded Arabic text) is hateful. A meme can be hateful only through the combination of its text and image, even when each part looks harmless alone, so the author focuses on how to merge the two signals. Text is encoded with MARBERTv2 and images with CLIP-Large (ViT-L/14). On top of these encoders, four fusion heads are compared: concatenation, a small transformer, bidirectional cross-attention, and MCB bilinear pooling. An ensemble then combines their outputs. Training uses the Prop2Hate-Meme splits, in which only about one meme in ten is hateful. Several imbalance remedies were tested: inverse-frequency class weights, focal loss, heavy regularization, and targeted augmentation. Transformer fusion was the most stable single model, and the ensemble edged slightly higher, reaching macro-F1 0.75 on the gold test set and second place on the leaderboard. The work gives a reproducible baseline for Arabic multimodal classification under scarce, skewed data.

Key points

  • Training data split is 90.05% not-hate vs. 9.95% hate, no external labeled data was used, and the metric is Macro-F1.
  • Gold-set Macro-F1 by model: ensemble 0.75, transformer fusion 0.75, concatenation 0.73, cross-attention 0.68, bilinear 0.66. The ensemble's gain over transformer fusion was only about +0.005.
  • Three ensembling schemes were tried: majority vote, equal-weight probability averaging, and a weighted average favouring transformer fusion.
  • Class-weighted cross-entropy gave the most reliable improvement, while focal loss helped minority recall only marginally.
  • Augmentation (image distortions, OCR re-extraction, synonym swaps, AR→EN→AR back-translation) caused the model to fit training data perfectly while falling on dev. Code is public on GitHub (YassirELATTAR/task3-mahed2025).

Interesting ideas

  • I1. The transformer fusion treats each modality's pooled embedding as a single token tagged with a type embedding, then runs a few self-attention layers over this tiny sequence; this cheap design beat heavier interaction models on small data.
  • I2. The author plots train and dev curves for 10 epochs with early stopping turned off, which makes visible how quickly each architecture overfits instead of hiding it behind a stopping rule.
  • I3. Augmentation that looked meaning-preserving still produced perfect training scores alongside worse dev scores, likely because the model learned superficial artefacts of the augmented samples.
  • I4. Rather than a plain vote, the ensemble can tilt toward the member that proved most stable on validation, combining robustness with the best individual model.
  • I5. For heavy class skew, the simple fix of inverse-frequency loss weights beat both focal loss and augmentation in this study.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Evidence from separate sources (VerbNet role guesses, parser arcs, coreference cues) could be encoded as typed tokens and combined by a small attention layer, keeping each source's contribution inspectable.
  • I2 (learn): Running training with stopping disabled could serve as a diagnostic for new parser or tagger heads, revealing which ones memorise silver data fastest.
  • I3 (learn): Synthetic or self-play sentences should be checked against a held-out set of natural text, as a negative control showing whether gains come from artefacts of the generator.
  • I4 (tree): When several witness parsers disagree on an arc, their votes could be weighted by measured per-parser reliability instead of counted equally.
  • I5 (tokens): Rare morphological tags or lemma classes in the tagger could be handled first with class-weighted loss before trying more elaborate schemes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany94 works · 46,609 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksVision & language · 811 works

Cite

Cite as El Attar, Yassir. 2025. YassirEA at MAHED 2025: Fusion-Based Multimodal Models for Arabic Hate Meme Detection. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.82

@inproceedings{yassir2025yassirea,
  title = {{YassirEA at MAHED 2025: Fusion-Based Multimodal Models for Arabic Hate Meme Detection}},
  author = {El Attar, Yassir},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.82},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.82}
}

References

19 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Go Inoue, Muhammad Abdul-Mageed, and Mahmoud El-Haj. 2021. Camelbert: Transformer-based arabic language models. In Proceedings of WANLP. find
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Samar Al-Saqqa, Arafat Awajan, and Bassam Hammo. 2024. A survey of hate speech detection for arabic social media: Methods and datasets. Procedia Computer Science, 251:224–231. find
  4. Firoj Alam, Md Rafiul Biswas, Uzair Shah, Wajdi Zaghouani, and Georgios Mikros. 2024a. Propaganda to hate: A multimodal analysis of arabic memes with multi-agent llms. In International Conference on Web Information Systems Engineering, pages 380– 390. Springer. Firoj Alam, Abul Hasnat, Fatema Ahmad, Md.
  5. Arid Hasan, and Maram Hasanain. 2024b. ArMeme: Propagandistic content in Arabic memes. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21071–21090, Miami, Florida, USA. Association for Computational Linguistics.
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the LREC. Fariha Tahosin Boishakhi, Ponkoj Chandra Shill, and Md. ↗
  7. Golam Rabiul Alam. 2021. Multi-modal hate speech detection using machine learning. In 2021 IEEE International Conference on Big Data (Big Data), page 4496–4499. IEEE. find
  8. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2021. The hateful memes challenge: Detecting hate speech in multimodal memes. Preprint, arXiv:2005.04790. ↗
  9. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2018. Focal loss for dense object detection. Preprint, arXiv:1708.02002. ↗
  10. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Preprint, arXiv:1908.02265. ↗
  11. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2017. Hierarchical question-image co-attention for visual question answering. Preprint, arXiv:1606.00061. ↗
  12. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436–1457. ↗
  13. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020. ↗
  14. Cees Snoek, Marcel Worring, and Arnold Smeulders. 2005. Early versus late fusion in semantic video analysis. pages 399–402. find
  15. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. Preprint, arXiv:1909.11740. ↗
  16. Yeshan Wang and Ilia Markov. 2024. CLTL@multimodal hate speech event detection 2024: The winning approach to detecting multimodal hate speech and its targets. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024), pages 73–78, St. Julians, Malta. Association for Computational Linguistics. ↗
  17. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457– 468, Austin, Texas. Association for Computational Linguistics.
  18. Wajdi Zaghouani, Md Rafiul Biswas, Mabrouka Bessghaier, Shimaa Ibrahim, Georgios Mikros, Abul Hasnat, and Firoj Alam. 2025. Overview of the mahed shared task: Multimodal detection of hope and hate emotions in arabic content. In The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou, China. Association for Computational Linguistics. *Table 3: Baseline summary on the test set (accuracy and macro F1).* Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. Preprint, arXiv:2303.15343. ↗
  19. Approach Yinghui Zhang, Tailin Chen, Yuchen Zhang, and Zeyu Fu. 2024. Enhanced multimodal hate video detection via channel-wise and modality-wise fusion. In 2024 IEEE International Conference on Data Mining Workshops (ICDMW), page 183–190. IEEE. Acc (Weighted) Macro-F1 (Weighted) Acc (Focal) Macro-F1 (Focal) 0.80 0.77 0.78 0.67 0.57 0.59 0.76 0.77 0.78 0.57 0.59 0.55 Text only Image only Confidence combine *Table 4: Text-only (Weighted) – classification report (test).* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/attar-2025-yassirea-mahed-2025-fusion-based/