Elamur Research⌕
☰
home / papers / 2024 / ASOS at ArAIEval Shared Task: Integrating Text and Image Embeddings for Multimodal Propaganda Detection in Arabic Memes

ASOS at ArAIEval Shared Task: Integrating Text and Image Embeddings for Multimodal Propaganda Detection in Arabic Memes

Paper · 2024 · Yasser Alhabashi, Abdullah Alharbi, Samar Ahmad et al.

Summary

The paper describes a system for ArAIEval 2024 Task 2C, a binary classification task that asks whether an Arabic meme is propagandistic. Each meme has an image and its extracted text. The data comes from Facebook, Twitter, Instagram and Pinterest, and every split has far more non-propaganda than propaganda items.

The authors first compared three Arabic text encoders on the development set: MARBERT, AraBERT and QARiB. MARBERT scored best, and they kept it fixed while testing three image backbones alongside it: ResNet50, Swin-V2 and MaxViT. The two embeddings are concatenated, compressed by a dense fusion layer to 512 dimensions, and passed to a softmax classifier.

The best pairing, MARBERT with MaxViT, reached a macro F1 of 0.798 on the test set and placed second. The abstract names ResNet50 as the image model, but the results section shows MaxViT was the one actually submitted.

The work shows that a simple late-fusion design over strong pretrained encoders is competitive for multimodal propaganda detection in Arabic. It also shows that adding the visual signal clearly improves on text alone.

Key points

  • On the dev set, the text-only results were MARBERT 0.754 macro F1 (P 0.778, R 0.739), AraBERT 0.713 and QARiB 0.706.
  • With MARBERT as the text encoder, dev macro F1 was 0.753 with ResNet50, about 0.787 with Swin-V2 and 0.791 with MaxViT (P 0.838, R 0.842).
  • On the test set, MARBERT + MaxViT reached P 0.790, R 0.809 and F1 0.798, ranking 2nd in Task 2C.
  • The fusion layer concatenates a 768-d text vector with an image vector. Image vectors are 2048-d for ResNet50, 1024-d for Swin-V2 and 768-d for MaxViT, so MARBERT + ResNet50 gives 2816 dimensions. A dense layer reduces this to 512 before the softmax head.
  • Training used cross-entropy loss, Adam with a learning rate of 2e-5, batch size 16, at most 3 epochs, and dropout of 0.3 in the text model and fusion layers. Evaluation used the official macro-averaged F1.

Interesting ideas

  • I1. Instead of searching the full grid of encoder pairs, the authors chose the best text encoder alone and then varied only the image side, which keeps the search linear and cheap.
  • I2. Late fusion here means concatenating features from heterogeneous sources and passing them through one learned projection. This lets independently pretrained components cooperate without retraining either one deeply.
  • I3. The authors checked that the class imbalance looks the same in train, dev and test, and they scored with macro F1 so the minority class counts fully.
  • I4. MARBERT, pretrained on dialectal tweets, beat the alternatives on informal meme text, which suggests that matching the pretraining register to the target domain matters.
  • I5. In memes, the propagandistic meaning often comes from text and image together rather than from either one alone, which is why the second signal helped.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tune pipeline components one stage at a time (e.g. fix the tagger, then compare parsers), logging each fixed choice so the selection path stays auditable.
  • I2 (tree): gparse could fuse transformer embeddings with explicit perceptron-parser features through a small projection while keeping the symbolic features inspectable.
  • I3 (learn): Report macro scores and per-split label distributions for rare phenomena such as irony or specific speech acts, so majority labels cannot hide failures.
  • I4 (tokens): Pick or weight training treebanks by register (web/social vs. news) when Mova must handle informal input, and test on each register separately.
  • I5 (frames): Irony and implicature detection in prag should take evidence from outside the sentence (surrounding discourse, world state) as an explicit second input, not judge the sentence in isolation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdulaziz UniversityJeddah · Saudi Arabia33 works · 508 citations🎓Prince Sultan UniversityRiyadh · Saudi Arabia20 works · 120 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksVision & language · 811 works

Cite

Cite as Yasser Alhabashi, Abdullah Alharbi, Samar Ahmad, Serry Sibaee, Omer Nacar, Lahouari Ghouti and Anis Koubaa. 2024. ASOS at ArAIEval Shared Task: Integrating Text and Image Embeddings for Multimodal Propaganda Detection in Arabic Memes. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.46

@inproceedings{alhabashi2024asos,
  title = {{ASOS at ArAIEval Shared Task: Integrating Text and Image Embeddings for Multimodal Propaganda Detection in Arabic Memes}},
  author = {Yasser Alhabashi, Abdullah Alharbi, Samar Ahmad, Serry Sibaee, Omer Nacar, Lahouari Ghouti and Anis Koubaa},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.46},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.46}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Tariq Habib Afridi, Aftab Alam, Muhammad Numan Khan, Jawad Khan, and Young-Koo Lee. 2021. A multimodal memes classification: A survey and open research issues. In Innovations in Smart Cities Applications Volume 4: The Proceedings of the 5th International Conference on Smart City Applications, pages 1451–1466. Springer. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9.
  5. Sravani Boinepelli, Manish Shrivastava, and Vasudeva Varma. 2020. Sis@ iiith at semeval-2020 task 8: An overview of simple text classification methods for meme analysis. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1190–1194. find
  6. Xuxiang Deng, Yifan Liu, and Qihao Yan. 2023. Meme-integrated deep learning: A multimodal classification fusion framework to fuse meme culture into deep learning. In 2023 International Conference on Image, Algorithms and Artificial Intelligence (ICIAAI 2023), pages 130–145. Atlantis Press. find
  7. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting propaganda techniques in memes. arXiv preprint arXiv:2109.08013. ↗
  8. Yuhao Du, Muhammad Aamir Masood, and Kenneth Joseph. 2020. Understanding visual memes: An empirical analysis of text superimposed on memes shared on twitter. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 153–164. find
  9. Marc J Dupuis and Andrew Williams. 2019. The spread of disinformation on the web: An examination of memes on social networking. In 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (Smart-World/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pages 1412–1418. IEEE. find
  10. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812. ↗
  11. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can gpt-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation, LREC-COLING 2024, Torino, Italy. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md. find
  12. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. ArAIEval Shared Task: Propagandistic techniques detection in unimodal and multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
  14. Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020. A multimodal framework for the detection of hateful memes. arXiv preprint arXiv:2012.12871. ↗
  15. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022.
  16. Sofiane Ouaari, Tsegaye Misikir Tashu, and Tomáš Horváth. 2022. Multimodal feature extraction for memes sentiment classification. In 2022 IEEE 2nd Conference on Information Technology and Data Science (CITDS), pages 285–290. IEEE. find
  17. Shivam Sharma, Firoj Alam, Md Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. arXiv preprint arXiv:2205.04274. ↗
  18. Shardul Suryawanshi, Mihael Arcan, Suzanne Little, and Paul Buitelaar. 2023. Multimodal offensive meme classification with natural language inference. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 134–145. find
  19. Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. 2022. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459–479. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alhabashi-2024-asos-araieval-shared-task-integrating/