Elamur Research⌕
☰
home / papers / 2024 / CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models

CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models

Paper · 2024 · Yeshan Wang and Ilia Markov

Summary

This paper describes the CLTL team's entry to the ArAIEval 2024 shared task, which asked whether Arabic memes are propagandistic. Each meme was judged three ways: from its extracted text alone (2A), from its image alone (2B), and from both together (2C). The organizers supplied one dataset of memes from Facebook, Twitter, Instagram and Pinterest. It is imbalanced, with only about 28% propaganda.

The authors did no preprocessing or feature engineering. They fine-tuned off-the-shelf models with identical hyperparameters:

  • Text: three Arabic BERT variants (MARBERT, CAMeLBERT, GigaBERT).
  • Image: two vision models (EVA, CAFormer).
  • Multimodal: text and image embeddings joined through AutoGluon's MLP fusion module, with a prediction layer on top.

The submitted systems placed 2nd on text (77.96 macro-F1), 1st on images (71.04) and 3rd on the combined task (79.80). Text carried more signal than images, and fusion added roughly 2 more F1 points. The work matters because it shows that simple late fusion of strong pretrained encoders competes at the top of a new multimodal propaganda benchmark.

Key points

  • Data: 2,143 training, 312 development and 607 test memes, with the propaganda class at about 28% in every split. The official metric is macro-F1.
  • Shared training setup:
    • Learning rate 1e-4 with cosine decay (rate 0.9)
    • Batch size 8, at most 10 epochs, AdamW
    • About 12 minutes of training per model on an NVIDIA L4, using PyTorch and AutoGluon
  • Text (2A): MARBERT, pretrained on about 1 billion Arabic tweets, led on dev (80.48 vs 79.32 CAMeLBERT and 78.66 GigaBERT). It scored 77.96 on test, 0.73 below the winner.
  • Image (2B): EVA beat CAFormer on dev (76.89 vs 75.23). Its test score of 71.04 fell about 6 points from dev but still beat the runner-up by 4.7.
  • Multimodal (2C):
    • GigaBERT + EVA scored highest on dev (82.60), even though GigaBERT was the weakest text model on its own.
    • On test it reached 79.80, against 80.51 for first place.
    • The image-only model had the most missed propaganda cases: 73 false negatives and 98 true positives.

Interesting ideas

  • I1. Every candidate model was trained with the same hyperparameters and time budget, so differences in scores reflect the models rather than tuning effort.
  • I2. The weakest standalone text encoder became the strongest once paired with the image model, so a component's solo score does not predict its value inside a combination.
  • I3. Running each modality alone and then fused shows which input source actually carries the decision (here mostly text).
  • I4. Pairing confusion matrices on the minority class with the dev-to-test drop exposes weaknesses that a single F1 number hides.
  • I5. The authors chose MARBERT because its pretraining on tweets matches short, informal social-media text, and it led on that kind of input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When comparing perceptron parser variants or gparse checkpoints, fix one training budget and configuration so the reported LAS gaps are attributable to the model itself.
  • I2 (learn): Choose witness parsers for silver-data agreement by how much they improve the agreed output, not by their individual LAS ranking.
  • I3 (answer): Ablate the answer readers by removing the world layer or the frames layer and measuring how far test scores drop, which shows which stage the correct answers actually depend on.
  • I4 (learn): Report dev/test gaps and per-class error counts (e.g. missed rare relations or speech acts) alongside headline scores to catch overfitting to the development split.
  • I5 (tokens): Train or adapt the tagger and lemmatizer on genre-matched text (e.g. fables, school tests) rather than only general news to reduce errors on those inputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Vrije Universiteit AmsterdamAmsterdam · Netherlands42 works · 3,372 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksVision & language · 657 works

Cite

Cite as Yeshan Wang and Ilia Markov. 2024. CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.51

@inproceedings{wang2024cltl,
  title = {{CLTL at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Transformer Models}},
  author = {Yeshan Wang and Ilia Markov},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.51},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.51}
}

References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hamdy Mubarak AbdelRahim Elmadany and Walid Magdy. 2018. ArSAS: An Arabic speech-act and sentiment corpus of tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. ArMeme: Propagandistic content in Arabic memes. arXiv preprint arXiv:2406.03916. ↗
  4. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  5. Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849– 1864. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Dimitar Dimitrov, Firoj Alam, Maram Hasanain, Abul Hasnat, Fabrizio Silvestri, Preslav Nakov, and Giovanni Da San Martino. 2024. SemEval-2024 task 4: Multilingual detection of persuasion techniques in memes. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 2009–2026, Mexico City, Mexico. Association for Computational Linguistics. find
  8. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. EVA: Exploring the limits of masked visual representation learning at scale. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358–19369. find
  9. Faye Margarette G. De Leon and Rachelle Ballesteros-Lintao. 2021. The rise of meme culture: Internet political memes as tools for analysing philippine propaganda. Journal of Critical Studies in Language and Literature, 2(4):1–13. ↗
  10. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can GPT-4 identify propaganda? Annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2724–2744, Torino, Italia. ELRA and ICCL. find
  11. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Freihat. 2023. ArAIEval shared task: Persuasion techniques and disinformation detection in Arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), pages 483–493, Singapore. Association for Computational Linguistics. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md.
  12. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. ArAIEval shared task: Propagandistic techniques detection in unimodal and multimodal Arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  13. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  14. Ashton Kingdon. 2021. The Meme Is the Method: Examining the Power of the Image Within Extremist Propaganda, pages 301–322. Springer International Publishing, Cham. find
  15. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics.
  16. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics. ↗
  17. Joshua Troy Nieubuurt. 2021. Internet memes: Leaflet propaganda of the digital age. Frontiers in Communication, 5. ↗
  18. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic gigaword fifth edition. Philadelphia: Linguistic Data Consortium. ↗
  19. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics.
  20. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252. ↗
  21. Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. 2021. Multimodal AutoML on structured tables with text fields. In 8th ICML Workshop on Automated Machine Learning (AutoML). find
  22. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  23. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. CoRR, abs/1912.07076. ↗
  24. Yeshan Wang and Ilia Markov. 2024a. CLTL at DIMEMEX shared task: Fine-grained detection of hate speech in memes. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the Conference of the Spanish Society for Natural Language Processing (SEPLN 2024). find
  25. Yeshan Wang and Ilia Markov. 2024b. CLTL@Multimodal hate speech event detection 2024: The winning approach to detecting multimodal hate speech and its targets. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024), pages 73–78, St. Julians, Malta. Association for Computational Linguistics. find
  26. Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. 2024. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2):896–912. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-cltl-araieval-shared-task-multimodal/