Elamur Research⌕
☰
home / papers / 2024 / AlexUNLP-MZ at ArAIEval Shared Task: Contrastive Learning, LLM Features Extraction and Multi-Objective Optimization for Arabic Multi-Modal Meme Propaganda Detection

AlexUNLP-MZ at ArAIEval Shared Task: Contrastive Learning, LLM Features Extraction and Multi-Objective Optimization for Arabic Multi-Modal Meme Propaganda Detection

Paper · 2024 · Mohamed Zaytoon, Nagwa El-Makky and Marwan Torki

Summary

The paper describes the winning approach to the ArAIEval 2024 shared task on detecting propaganda in Arabic memes. Each meme has an image and a caption with a binary label, and systems were ranked separately on text-only, image-only and combined input. For text, the authors compared Arabic BERT variants with BLOOM and BLOOMZ models. For images, they used ImageNet CNNs such as ResNet, ResNeXt and DenseNet. They also queried ChatGPT and Gemini with yes/no questions about each caption, for example whether it is political or offensive, and added the answers as binary input features. The multimodal model fuses the text and image encoders and trains on four objectives together: weighted cross-entropy on the text branch, the image branch and the fused output, plus an NT-Xent contrastive term. On the hidden test set the system placed first on text (Macro-F1 78.69%), third on images (65.92%) and first on multimodal input (80.51%). The results show that cheap LLM-derived features and auxiliary losses add measurable gains over a plain fused classifier.

Key points

  • Data split: 2,143 training memes (1,540 non-propaganda, 603 propaganda), 312 validation memes and 607 test memes. The classes are imbalanced.
  • Text preprocessing: English words, diacritics and HTML were removed from captions.
  • Best single-modality models on validation:
    • Text: bloomz-1b1, with Macro-F1 0.801. It beat bert-base-arabertv02-twitter (0.780) and bert-base-arabic-camelbert-da (0.761).
    • Image: resnext101_32x8d, with 0.742.
  • Fusion methods: concatenation, bilinear, gated and attention fusion were all tried. On validation, concatenation (0.825) and attention (0.824) did best. The top validation score (0.831) came from bloomz-1b1 combined with resnet101 through concatenation. The official submission used attention fusion.
  • Ablation on validation: with LLM features, per-branch losses and contrastive loss all included, Macro-F1 was 0.825. Without any of them it was 0.799.
  • Error analysis: the model fails to recognise public figures in images, and mentions of countries or personal names can push it toward the wrong label.

Interesting ideas

  • I1. Asking an LLM a fixed set of yes/no questions per input yields interpretable binary features that a small model can consume cheaply, instead of opaque embeddings.
  • I2. Each modality branch gets its own classification loss in addition to the loss on the fused output, so neither encoder becomes lazy and leans entirely on the other.
  • I3. Weighted cross-entropy is a simple way to handle a minority class without resampling the data.
  • I4. The paper reports a cumulative ablation that removes the extra components (features, auxiliary losses, contrastive term) to measure what they add.
  • I5. The failure analysis showed that names and countries act as spurious cues for the label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): An LLM teacher could answer named probes such as "is this sarcastic?" or "is this a request?" to produce explicit, inspectable signals for speech-act and irony frames.
  • I2 (tree): gparse could attach auxiliary heads (for example POS or morphology) to intermediate layers so shared representations stay useful for each sub-task.
  • I3 (tokens): Rare morphological tags or rare lemmas in the en tagger could be up-weighted in the loss so they are not swamped by frequent tags.
  • I4 (learn): Every new silver-data source or repair rule could be evaluated by switching it off, giving a negative control for its real contribution.
  • I5 (learn): Evaluation sets could include perturbed copies with entity names swapped, to check whether readers rely on surface names rather than structure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Mohamed Zaytoon, Nagwa El-Makky and Marwan Torki. 2024. AlexUNLP-MZ at ArAIEval Shared Task: Contrastive Learning, LLM Features Extraction and Multi-Objective Optimization for Arabic Multi-Modal Meme Propaganda Detection. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.53

@inproceedings{zaytoon2024alexunlp,
  title = {{AlexUNLP-MZ at ArAIEval Shared Task: Contrastive Learning, LLM Features Extraction and Multi-Objective Optimization for Arabic Multi-Modal Meme Propaganda Detection}},
  author = {Mohamed Zaytoon, Nagwa El-Makky and Marwan Torki},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.53},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.53}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A survey on multimodal disinformation detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  2. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. ArMeme: Propagandistic content in arabic memes. arXiv: 2406.03916. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  4. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting propaganda techniques in memes. arXiv preprint arXiv:2109.08013. ↗
  5. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023a. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812. ↗
  6. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can gpt-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024. find
  7. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023b. ArAIEval Shared Task: persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md.
  8. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. ArAIEval Shared Task: Propagandistic techniques detection in unimodal and multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  9. Mai Ibrahim, Marwan Torki, and Nagwa El-Makky. 2020. AlexU-BackTranslation-TL at SemEval-2020 task 12: Improving offensive language detection using data augmentation and transfer learning. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1881–1890, Barcelona (online). International Committee for Computational Linguistics. ↗
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  11. TorchVision maintainers and contributors. 2016. Torchvision: Pytorch’s computer vision library. https://github.com/pytorch/vision. ↗
  12. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. ↗
  13. Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. 2020. Pytorch metric learning. ArXiv, abs/2008.09164. ↗
  14. Joris T. Nieubuurt. 2020. Internet memes: Leaflet propaganda of the digital age. Frontiers in Psychology, 11:1709. ↗
  15. Trong Huy Phan and Kazuma Yamamoto. 2020. Resolving class imbalance in object detection with weighted cross entropy losses. arXiv preprint arXiv:2006.01413. ↗
  16. Mohammad Reza Rezaei-Dastjerdehei, Amirmohammad Mijani, and Emad Fatemizadeh. 2020. Addressing imbalance in multi-label classification using weighted cross entropy loss function. In 2020 27th National and 5th International Iranian Conference on Biomedical Engineering (ICBME), pages 333– 338. IEEE. find
  17. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. AlexU-AIC at Arabic hate speech 2022: Contrast to classify. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 200–208, Marseille, France. European Language Resources Association. ↗
  18. Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find
  19. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  20. Feng Wang and Huaping Liu. 2021. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495–2504. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaytoon-2024-alexunlp-mz-araieval-shared-task/