Elamur Research⌕
☰
home / papers / 2026 / Multi-Modal-Minds@CHiPSAL 2026: A Comparative Study of Textual, Visual and Multimodal Architecture for Nepali Meme Moderation

Multi-Modal-Minds@CHiPSAL 2026: A Comparative Study of Textual, Visual and Multimodal Architecture for Nepali Meme Moderation

Paper · 2026 · Sandesh Shrestha, Bikram K.C., Akshyat Shah et al.

Summary

This paper describes the system that the Multi-Modal-Minds team entered in the CHiPSAL 2026 shared task, which focuses on moderating Nepali memes. Moderation tools for low-resource languages like Nepali remain far behind those for English. The task has two subtasks: binary hate detection (Subtask A) and three-way sentiment classification (Subtask B). The data comes from NeMeme, with roughly 1,000 training memes per subtask. Meme text was read with EasyOCR without any image cleanup, and training images received web-style augmentations.

The authors compared four families of models:

  • text-only encoders (mBERT, XLM-RoBERTa, MuRIL)
  • image-only encoders (ResNet, ConvNeXt, ViT)
  • a grid of nine early-fusion combinations
  • the pre-aligned vision-language model SigLIP

Counter to expectations, explicit fusion did not win. ViT performed best on hate detection, and SigLIP performed best on sentiment. The authors attribute the weak fusion results to noisy OCR output, weak Nepali text representations and the small training set. Grad-CAM and modality-ablation analyses support this reading. The study suggests that OCR-free vision-language models may be a better path for moderating low-resource memes.

Key points

  • Data splits: Subtask A has 1,068 training memes (348 hate, 720 non-hate), 133 validation and 134 test. Subtask B has 1,061 training memes (341 negative, 473 neutral, 247 positive), 133 validation and 133 test.
  • Best macro F1 was 0.6278 for ViT on hate detection and 0.5481 for SigLIP on sentiment.
  • Grad-CAM showed that ViT largely ignores the Devanagari text and focuses on emojis such as "tears of joy." This explains many correct hate predictions, but it also caused errors on memes whose hatefulness depends on context.
  • Counterfactual ablation on SigLIP for hate detection found 34 image-biased and 6 text-biased cases. For sentiment, it found 12 image-biased and 17 text-biased cases, with 104 cases showing genuine multimodal synergy.
  • The image augmentation pipeline was built with albumentations. It included JPEG compression and downscaling, coarse dropout and hue/saturation shifts. The authors propose OCR-free models such as Donut and PaliGemma as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Delhi Technological UniversityNew Delhi · India4 works · 196 citations🎓Kathmandu UniversityKathmandu · Nepal7 works · 20 citations🎓IIMS CollegeKathmandu · Nepal5 works · 5 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Sandesh Shrestha, Bikram K.C., Akshyat Shah, Ashish Acharya and Rabin Thapa. 2026. Multi-Modal-Minds@CHiPSAL 2026: A Comparative Study of Textual, Visual and Multimodal Architecture for Nepali Meme Moderation. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ajzb6i9sfbx

@inproceedings{shrestha2026multi,
  title = {{Multi-Modal-Minds@CHiPSAL 2026: A Comparative Study of Textual, Visual and Multimodal Architecture for Nepali Meme Moderation}},
  author = {Sandesh Shrestha, Bikram K.C., Akshyat Shah, Ashish Acharya and Rabin Thapa},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/4ajzb6i9sfbx},
  url = {https://doi.org/10.63317/4ajzb6i9sfbx}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443. find
  2. Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  3. Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. 2020. Albumentations: Fast and flexible image augmentations. Information, 11(2). find
  4. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John P. Mc-Crae. 2020. Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaborati…
  5. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, Anand Kumar M, Thomas Mandl, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Hariharan R L, John Philip McCrae, and Elizabeth Sherly. 2021. Findings of the shared task on offensive language identification in Tamil, Malayalam, and Kannada. In Pr…
  6. Alexis Conneau et al. 2020. Unsupervised cross-lingual representation learning at scale. ACL. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL.
  8. Alexey Dosovitskiy et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.
  9. Kaiming He et al. 2016. Deep residual learning for image recognition. CVPR.
  10. Simran Khanuja et al. 2021. Muril: Multilingual representations for indian languages. EMNLP.
  11. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020a. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems, volume 33, pages 2611–2624. Curran Associates, Inc.
  12. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020b. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems, volume 33, pages 2611–2624.
  13. Zhuang Liu et al. 2022. A convnet for the 2020s. CVPR. well-being. In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (CHiPSAL). find
  14. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025a. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. find
  15. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025b. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042– 2051. find
  16. Xiaohua Zhai et al. 2023. Sigmoid loss for language image pre-training. ICCV.
  17. Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302–1308. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shrestha-2026-multi-modal-minds-chipsal-2026/