Elamur Research⌕
☰
home / papers / 2026 / eGrantha.ai@CHiPSAL 2026: Stochastic Image Captioning for Robust Hate Speech Detection in Low-Resource Nepali Memes

eGrantha.ai@CHiPSAL 2026: Stochastic Image Captioning for Robust Hate Speech Detection in Low-Resource Nepali Memes

Paper · 2026 · Anish Thapaliya

Summary

This paper targets binary hate/non-hate classification of Nepali memes (Subtask A of the CHiPSAL 2026 multimodal shared task). The task is hard because meaning comes from images, text and emojis together, and Nepali has few resources. The data are drawn from NeMeme, ENeMeme and CrisisHateMM. The author's key move is to turn the multimodal problem into plain text classification. Gemini 3 Flash writes a detailed, culturally aware English description of each meme, guided by a structured prompt, and an encoder-only classifier is then fine-tuned on these descriptions. This removes the need for OCR or Nepali-specific vision-language pretraining.

Two tricks build on the randomness of sampling at temperature 1.0:

  • Augmentation: extra captions are generated for minority-class images to reduce class imbalance.
  • Test-time augmentation (TTA): predictions over several captions per test image are averaged.

Several encoders are compared (BERT, RoBERTa, DeBERTa, HateBERT, MuRIL, ModernBERT). RoBERTa-base performed best, reaching macro F1 0.7397 and third place on the leaderboard. It clearly beat zero-shot, text-only and ViT-fusion baselines. The generated captions are released as NeMeme-CAP.

Key points

  • Data splits: 1,068 training memes (348 non-hate, 720 hate), 133 for validation and 134 for testing. One additional caption per non-hate image raised the training set to 1,416 items (696 vs 720).
  • For TTA, five captions per test image come from Gemini 2.5 Flash, 2.5 Pro, 3 Flash and 3 Pro. Class probabilities are averaged before argmax, at no extra training cost.
  • Baseline macro F1 scores:
    • zero-shot Gemini 3 Pro: 0.3728
    • text-only MuRIL / XLM-RoBERTa-base on OCR text: 0.4018
    • early fusion with ViT: 0.6199–0.6266 (best was MuRIL + ViT)
  • The best system, RoBERTa-base fine-tuned on Gemini captions, scored macro F1 0.7397 and ranked 3rd on Codabench.
  • Resources: the NeMeme-CAP caption dataset is on Hugging Face (Anish/nepali-meme-captions), and the code is on GitHub (thapaliya123/LREC-CHiPSAL-2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tribhuvan UniversityKathmandu · Nepal5 works · 15 citations

Also on the byline: Institute of Science and Technology.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 worksBias, fairness & safety · 517 worksLow-resource languages · 267 works

Cite

Cite as Anish Thapaliya. 2026. eGrantha.ai@CHiPSAL 2026: Stochastic Image Captioning for Robust Hate Speech Detection in Low-Resource Nepali Memes. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5d8j4bfmcvry

@inproceedings{thapaliya2026egrantha,
  title = {{eGrantha.ai@CHiPSAL 2026: Stochastic Image Captioning for Robust Hate Speech Detection in Low-Resource Nepali Memes}},
  author = {Anish Thapaliya},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/5d8j4bfmcvry},
  url = {https://doi.org/10.63317/5d8j4bfmcvry}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prabhat Ale, Anish Thapaliya, and Suman Paudel. 2025. MDSBots@NLU of Devanagari script languages 2025: Detection of language, hate speech, and targets using MURTweet. In Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 308–313, Abu Dhabi, UAE.… find
  2. Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  3. Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (ALW), co-located with EMNLP 2021, pages 17–25. Association for Computational Linguistics.
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR.
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
  7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern…
  8. Google DeepMind. 2025a. Gemini 3 flash: frontier intelligence built for speed. https://blog.google/products-and-platforms/ products/gemini/gemini-3-flash/. Official Google AI blog post. Accessed: 2026-03-01. ↗
  9. Google DeepMind. 2025b. Gemini 3 pro model card. Technical report, Google DeepMind. Model card for Gemini 3 Pro. Accessed: 2026-03-01. find
  10. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  11. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. MuRIL: Multilingual representations for Indian langua…
  12. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems, volume 33, pages 2611–2624. Curran Associates, Inc.
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  14. Nikita Moshkov, Botond Mathe, Attila Kertesz-Farkas, Reka Hollandi, and Peter Horvath. 2020. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Scientific reports, 10(1):5068. find
  15. Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302–1308. IEEE. find
  16. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  17. Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. 2023. Nepalibert: Pre-training of masked language model in nepali corpus. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), pages 325–330. find
  18. Kengatharaiyer Sarveswaran, Surendrabikram Thapa, Ashwini Vaidya, Tafseer Ahmed Khan, and Bal Krishna Bal. 2026. Findings of the second workshop on challenges in processing south asian languages (chipsal 2026). In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (…
  19. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and infer…
  20. A. (1) Visual and textual content — Describe all visible elements and transcribe any embedded Devanagari or English text verbatim. find
  21. Surendrabikram Thapa, Siddhant Bikram Shah, Kritesh Rauniyar, Shuvam Shiwakoti, Surabhi Adhikari, Hariram Veeramani, Kristina T. Johnson, Ali Hurriyetoglu, Hristo Tanev, and Usman Naseem. 2025a. Multimodal hate, humor, and stance event detection in marginalized sociopolitical movements. In Proceedin… find
  22. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Kengatharaiyer Sarveswaran, Bal Krishna Bal, and Usman Naseem. 2026. Multimodal hate and sentiment understanding in low-resource text-embedded images for online safety…
  23. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025b. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. find
  24. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025c. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042– 2051. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thapaliya-2026-egrantha-ai-chipsal-2026-stochastic/