Elamur Research⌕
☰
home / papers / 2026 / Multimodal Transformer Framework for Multilingual Harmful Meme Classification

Multimodal Transformer Framework for Multilingual Harmful Meme Classification

Paper · 2026 · Charmathi Rajkumar, Malliga Subramanian and Bharathi Raja Chakravarthi

Summary

The paper tackles automatic detection of harmful memes, where offensive meaning often comes from the interplay of image and overlaid text and appears across many languages, including low-resource ones. The authors build one multimodal architecture and apply it to three tasks: misogyny, homophobia/transphobia, and hate speech, spanning eight languages. Text extracted by OCR is encoded with XLM-RoBERTa and the image with a Swin Transformer. A cross-attention layer then lets the textual vector, acting as the query, attend over visual features, and a residual connection with layer normalization feeds a small GELU classifier. They compare macro-F1 scores against systems submitted to earlier shared tasks on the same data. Their model scores highest in every language track, with especially large margins on Chinese homophobia/transphobia memes and on Bodo hate speech. The work suggests that a fairly standard pairing of multilingual text and vision transformers, joined by attention, gives a strong baseline for moderating culturally diverse content, including in under-resourced languages.

Key points

  • Architecture: xlm-roberta-base handles text and swin-base-patch4-window7-224 handles images (224×224). Both encoders are fine-tuned jointly with cross-entropy loss for 20 epochs, using AdamW, a learning rate of 2e-5, and batch size 16, on a single A100.
  • Datasets:
    • Misogyny: MDMD (Tamil, Malayalam) and CMMD (Chinese).
    • Homophobia/transphobia: a meme dataset in Hindi, English, and Chinese.
    • Hate speech: a Hindi and Bodo meme collection for identifying hateful content, released by Ghosh and colleagues (2026).
  • Macro-F1, misogyny: Chinese 0.8715, Malayalam 0.856, Tamil 0.7619.
  • Macro-F1, homophobia/transphobia: English 0.7519, Hindi 0.7825, Chinese 0.923. Hate speech: Hindi 0.7615, Bodo 0.783.
  • Limitations: Errors cluster on sarcastic memes, where the text looks neutral and the image carries the offence, and on memes that depend on cultural cues. The authors also note high compute requirements and a relatively simple fusion design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksVision & language · 657 worksBias, fairness & safety · 517 works

Cite

Cite as Charmathi Rajkumar, Malliga Subramanian and Bharathi Raja Chakravarthi. 2026. Multimodal Transformer Framework for Multilingual Harmful Meme Classification. Proceedings of the Sixth Workshop on Language Technology for Equality, Diversity, Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.ltedi-1.9

@inproceedings{rajkumar2026multimodal,
  title = {{Multimodal Transformer Framework for Multilingual Harmful Meme Classification}},
  author = {Charmathi Rajkumar, Malliga Subramanian and Bharathi Raja Chakravarthi},
  booktitle = {Proceedings of the Sixth Workshop on Language Technology for Equality, Diversity, Inclusion},
  year = {2026},
  doi = {10.18653/v1/2026.ltedi-1.9},
  url = {https://doi.org/10.18653/v1/2026.ltedi-1.9}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bharathi Raja Chakravarthi, Rahul Ponnusamy, Ping Du, Xiaojian Zhuang, Saranya Rajiakodi, Paul Buitelaar, Premjith B, Bhuvaneswari Sivagnanam, Anshid Kizhakkeparambil, and Lavanya S.K. 2025. An overview of the misogyny meme detection shared task for Chinese social media. In Proceedings of the 5th Co… find
  2. Bharathi Raja Chakravarthi, Saranya Rajiakodi, Rahul Ponnusamy, Kathiravan Pannerselvam, Anand Kumar Madasamy, Ramachandran Rajalakshmi, Hariharan LekshmiAmmal, Anshid Kizhakkeparambil, Susminu S Kumar, Bhuvaneswari Sivagnanam, and Charmathi Rajkumar. 2024. Overview of shared task on multitask meme… find
  3. Shraddha Chauhan and Abhinav Kumar. 2025. MNLP@DravidianLangTech 2025: Transformer-based multimodal framework for misogyny meme detection. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 248–253, Acoma, The Albuquerque Convention Cente… find
  4. Dongha Choi, Jung-jae Kim, and Hyunju Lee. 2024. TransferCVLM: Transferring cross-modal knowledge for vision-language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16733–16746, Miami, Florida, USA. Association for Computational Linguistics. find
  5. Md. Sajid Alam Chowdhury, Mostak Mahmud Chowdhury, Anik Mahmud Shanto, Jidan Al Abrar, and Hasan Murad. 2025. Fired_from_NLP@DravidianLangTech 2025: A multimodal approach for detecting misogynistic content in Tamil and Malayalam memes. In Proceedings of the Fifth Workshop on Speech, Vision, and Lang… find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  7. Koyel Ghosh, Mithun Das, Sumukh Patel, Nilotpal Bhandary, Alloy Das, Animesh Mukherjee, Sandip Modha, Debasis Ganguly, Utpal Garain, Sylvia Jaki, and Thomas Mandl. 2026. Overview of the hasoc track at fire 2025: Abusive meme identification — shadows behind the laughter. In Proceedings of the 17th An… find
  8. Misogynistic / Non-misogynistic Gender Stereotype Misclassification Homophobic / Non-hateful Contextual LGBTQ+ Hate Misclassification “Why are so many people in relationships?” Girl asks for a maroon stone-work mask while shopping for Perunnal. “Two ladies, this way please.” “If you are under me, th… find
  9. Md. Refaj Hossan, Nazmus Sakib, Md. Alam Miah, Jawad Hossain, and Mohammed Moshiul Hoque. 2025. CUET-NLP_Big_O@DravidianLangTech 2025: A multimodal fusion-based approach for identifying misogyny memes. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian La… find
  10. Sreeja K and Bharathi B. 2025. SSNCSE@LT-EDI-2025: Detecting misogyny memes using pretrained deep learning models. In Proceedings of the 5th Conference on Language, Data and Knowledge: Fifth Workshop on Language Technology for Equality, Diversity, Inclusion, pages 1–5, Naples, Italy. Unior Press. find
  11. Prasanna Kumar Kumaresan, Bharathi Raja Chakravarthi, Ruba Priyadharshini, Paul Buitelaar, Malliga Subramanian, and Kishore Kumar Ponnusamy. 2025. Overview of homophobia and transphobia span detection in social media comments. In Proceedings of the 5th Conference on Language, Data and Knowledge: Fif… find
  12. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  13. Durga Prasad Manukonda and Rohith Gowtham Kodali. 2025. byteSizedLLM@DravidianLangTech 2025: Multimodal misogyny meme detection in low-resource Dravidian languages using transliteration-aware XLM-RoBERTa, ResNet-50, and attention-BiLSTM. In Proceedings of the Fifth Workshop on Speech, Vision, and La… find
  14. Rahul Ponnusamy, Kathiravan Pannerselvam, Saranya Rajiakodi, Prasanna Kumar Kumaresan, Sajeetha Thavareesan, Bhuvaneswari Sivagnanam, Anshid K.A, Susminu S Kumar, Paul Buitelaar, and Bharathi Raja Chakravarthi. 2024. From laughter to inequality: Annotated dataset for misogyny detection in Tamil and… find
  15. Md. Mizanur Rahman, Jidan Al Abrar, Md. Siddikul Imam Kawser, Ariful Islam, Md. Mubasshir Naib, and Hasan Murad. 2025a. CUET’s_White_Walkers@LT-EDI 2025: Racial hoax detection in code-mixed on social media data. In Proceedings of the 5th Conference on Language, Data and Knowledge: Fifth Workshop on… find
  16. Mehreen Rahman, Faozia Fariha, Nabilah Tabassum, Samia Rahman, and Hasan Murad. 2025b. CUET_12033@LT-EDI-2025: Misogyny detection. In Proceedings of the 5th Conference on Language, Data and Knowledge: Fifth Workshop on Language Technology for Equality, Diversity, Inclusion, pages 127–132, Naples, It… find
  17. Saranya Rajiakodi, Bharathi Raja Chakravarthi, Rahul Ponnusamy, Shunmuga Priya Muthusamy Chinnan, Prasanna Kumar Kumaresan, Sathiyaraj Thangasamy, Bhuvaneswari Sivagnanam, Balasubramanian Palani, Kogilavani Shanmugavadivel, Abirami Murugappan, and Charmathi Rajkumar. 2025. Findings of the shared tas… find
  18. Saranya Rajiakodi, Shunmuga Priya Muthusamy Chinnan, Kathiravan R., Rajalakshmi andPannersel-vam, Bhuvaneshwari Sivagnanam, Jananayagan V, Charmathi Rajkumar, R Ramesh Kannan, and Bharathi Raja Chakravarthi. 2026. From Comments to Harm: A Findings Report on Abusive Tamil Text Targeting Women on Soci… find
  19. CUET_Novice@DravidianLangTech 2025: A multimodal transformer-based approach for detecting misogynistic memes in Malayalam language. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 472–477, Acoma, The Albuquerque Convention Center, Albu… find
  20. Kogilavani Shanmugavadivel, Malliga Subramanian, Mohamed Arsath H, Ramya K, and Ragav R. 2025a. TEAM_STRIKERS@DravidianLangTech2025: Misogyny meme detection in Tamil using multimodal deep learning. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Langua… find
  21. Kogilavani Shanmugavadivel, Malliga Subramanian, Pooja Sree M, Palanimurugan V, and Roshini Priya K. 2025b. InnovationEngineers@DravidianLangTech 2025: Enhanced CNN models for detecting misogyny in Tamil memes using image and text classification. In Proceedings of the Fifth Workshop on Speech, Visio… find
  22. Radhika K T and Sitara K. 2025. CVF-NITT@LT-EDI-2025:MisogynyDetection. In Proceedings of the 5th Conference on Language, Data and Knowledge: Fifth Workshop on Language Technology for Equality, Diversity, Inclusion, pages 47–53, Naples, Italy. Unior Press. find
  23. Yeshan Wang and Ilia Markov. 2024. CLTL@multimodal hate speech event detection 2024: The winning approach to detecting multimodal hate speech and its targets. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024), pa… find
  24. Pathange Omkareshwara Rao, Harish Vijay V, Ippatapu Venkata Srichandra, Neethu Mohan, and Sachin Kumar S. 2025. Code_Conquerors@DravidianLangTech 2025: Multimodal misogyny detection in Dravidian languages using vision transformer and BERT. In Proceedings of the Fifth Workshop on Speech, Vision, and… find
  25. Khadiza Sultana Sayma, Farjana Alam Tofa, Md Osama, and Ashim Dey. 2025. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rajkumar-2026-multimodal-transformer-framework-multilingual-harmful/