Elamur Research⌕
☰
home / papers / 2024 / MODOS at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM, CLIP and ArabianGPT

MODOS at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM, CLIP and ArabianGPT

Paper · 2024 · Abdelhamid Haouhat, Hadda Cherroun, Slimane Bellaouar et al.

Summary

The MODOS team addressed binary detection of propaganda in Arabic memes, which combine an image with overlaid text, as subtask 2C of the ArAIEval shared task at ArabicNLP 2024. They used the organisers' dataset of roughly 3,000 memes labelled propagandistic or not. The pipeline has four parts:

  • Image segmentation: the Segment Anything Model (SAM) automatically cuts every image into object regions.
  • Image features: each region is embedded with the CLIP image encoder.
  • Text features: the meme text is cleaned and embedded with ArabianGPT-08.
  • Fusion and classification: separate LSTM encoders bring both modalities to a common shape. A learned weighting then scores how much each modality matters before the features are merged, and an MLP trained with cross-entropy makes the final decision.

The system beat all organiser baselines (random, majority, TF-IDF n-gram SVM, ImgBert) and placed 6th, with a test macro F1 around 0.73. The work matters because Arabic propaganda research had been almost entirely text-only. It shows that region-level visual features from off-the-shelf zero-shot models can be combined with Arabic LLM embeddings for multimodal detection.

Key points

  • Data split: train 2,143 memes (603 P / 1,540 NP), dev 312 (88 / 224), test 607 (171 / 436). The classes are clearly imbalanced, at about 28% propagandistic.
  • Segmentation output: SAM produced about 68.6K visual segments for train, 10K for dev and 19.7K for test. The predicted-IoU and stability thresholds were set to 0.95 so that only clean masks were kept.
  • Test results: accuracy 0.8040 and F1 0.7348 in the results table, while the abstract reports 0.7290 macro F1. The winning team, AlexUNLP-MZ, reached F1 0.8051.
  • Baseline F1 scores: n-gram SVM 0.6476, ImgBert 0.6563, majority 0.4180 (despite 0.7183 accuracy), random 0.4923.
  • Stated weaknesses: the authors cite overfitting, class imbalance, and the absence of explicit propaganda-technique cues (such as name-calling, fear and testimonial) as limits on generalisation. Code is on GitHub (belgats/Multimodal-Propaganda-Detection).

Interesting ideas

  • I1. An image is treated as a sequence of object regions, each with its own CLIP vector, rather than as one global vector, so the classifier sees parts instead of a blur of the whole picture.
  • I2. Modality-specific encoders first map inputs of very different lengths into an equal-shaped hidden space, and only then are they combined.
  • I3. Fusion uses learned importance scores per modality rather than plain concatenation, which makes each input's contribution an inspectable weight.
  • I4. Generated components are kept only when both the quality and stability scores clear a high bar (0.95), trading coverage for cleanliness.
  • I5. The baseline suite includes trivial majority and random predictors, which expose that high accuracy can coexist with near-useless F1 on imbalanced labels.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): A text could likewise be represented as a set of individually encoded entities and states rather than a single document vector. This keeps evidence attributable to specific items in the world of the text.
  • I2 (learn): When several witness parsers or feature sources with different granularities feed one decision, normalise each into a common slot structure before merging. This prevents any single source from dominating merely through its size.
  • I3 (learn): Weighting witness parsers by learned reliability when building silver data would give an explainable record of which witness drove each agreed arc.
  • I4 (learn): Silver trees or LLM-teacher labels could be admitted only above strict agreement and confidence thresholds, keeping noisy examples out of training.
  • I5 (answer): Always report macro F1 next to majority and random controls on school tests and fable QA. Then a reader that just guesses the dominant answer type cannot look competent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Amar Telidji University of LaghouatLaghouat · Algeria8 works · 121 citations🎓University of GhardaiaGhardaïa · Algeria4 works · 19 citations🎓Ziane Achour University of DjelfaDjelfa · Algeria4 works · 19 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Abdelhamid Haouhat, Hadda Cherroun, Slimane Bellaouar and Attia Nehar. 2024. MODOS at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM, CLIP and ArabianGPT. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.48

@inproceedings{haouhat2024modos,
  title = {{MODOS at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM, CLIP and ArabianGPT}},
  author = {Abdelhamid Haouhat, Hadda Cherroun, Slimane Bellaouar and Attia Nehar},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.48},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.48}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. Armeme: Propagandistic content in arabic memes.
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Bodor Moheel Almotairy, Manal Abdullah, and Dimah Hussein Alahmadi. 2024. Dataset for detecting and characterizing arab computation propaganda on x. Data in Brief, 53:110089. find
  4. Joseph Attieh and Fadi Hassan. 2022. Pythoneers at wanlp 2022 shared task: Monolingual arabert for arabic propaganda detection and span extraction. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 534– 540.
  5. Richard Dawkins. 2006. The selfish gene. Oxford university press. find
  6. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting propaganda techniques in memes. arXiv preprint arXiv:2109.08013. ↗
  7. Abdelhamid Haouhat, Slimane Bellaouar, Attia Nehar, and Hadda Cherroun. 2023. Towards arabic multimodal dataset for sentiment analysis. In 2023 Fourth International Conference on Intelligent Data Science Technologies and Applications (IDSTA), pages 126– 133.
  8. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023a. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812. ↗
  9. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can gpt-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024, Torino, Italy. find
  10. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023b. Araieval shared task: Persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md.
  11. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. ArAIEval Shared Task: Propagandistic techniques detection in unimodal and multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  12. Akib Mohi Ud Din Khanday, Qamar Rayees Khan, and Syed Tanzeel Rabani. 2021. Detecting textual propaganda using machine learning techniques. Baghdad Science Journal, 18(1):0199. ↗
  13. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything. arXiv:2304.02643. ↗
  14. Anis Koubaa, Adel Ammar, Lahouari Ghouti, Omar Najar, and Serry Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. Preprint, arXiv:2402.15313. ↗
  15. Salima Lamsiyah, Abdelkader Mahdaouy, Hamza Alami, Ismail Berrada, and Christoph Schommer. 2023. Ul & um6p at araieval shared task: Transformer-based model for persuasion techniques and disinformation detection in arabic. In The 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (ACL), Singapore, Singapore.
  16. Clyde R Miller. 1939. The techniques of propaganda. from “how to detect and analyze propaganda,” an address given at town hall. The Center for learning. find
  17. Shubham Mittal and Preslav Nakov. 2022. Iitd at the wanlp 2022 shared task: Multilingual multi-granularity network for propaganda detection. arXiv preprint arXiv:2210.17190. ↗
  18. Olumide E Ojo, Olaronke O Adebanji, Hiram Calvo, Damian O Dieke, Olumuyiwa E Ojo, Seye E Akin-sanya, Tolulope O Abiola, and Anna Feldman. 2023. Legend at araieval shared task: Persuasion technique detection using a language-agnostic text representation model. arXiv preprint arXiv:2310.09661. ↗
  19. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR. ↗
  20. Eshrag Ali Refaee, Basem Ahmed, and Motaz Saad. 2022. Arabem at wanlp 2022 shared task: Propaganda detection in arabic tweets. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 524–528.
  21. Ahmed Samir, Abu Bakr Soliman, Mohamed Ibrahim, Laila Hesham, and Samhaa R El-Beltagy. 2022. Ngu_cnlp at wanlp 2022 shared task: Propaganda detection in arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 545–550. find
  22. Limor Shifman. 2013. Memes in digital culture. MIT press. ↗
  23. Yunze Xiao and Firoj Alam. 2023. Nexus at araieval shared task: Fine-tuning arabic language models for propaganda and disinformation detection. arXiv preprint arXiv:2311.03184. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haouhat-2024-modos-araieval-shared-task-multimodal/