Elamur Research⌕
☰
home / papers / 2026 / linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture

linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture

Paper · 2026 · Sunil Regmi, Bipesh Subedi, Saugat Singh et al.

Summary

Automated moderation of memes is difficult for low-resource, code-mixed languages like Nepali. Benchmark data is scarce, OCR support is weak, and the text often mixes Devanagari, transliterated English and native English words. For the CHiPSAL 2026 shared task, which focuses on Nepali images with embedded text, the authors developed an approach that handles both binary hate detection (Subtask A) and three-class sentiment classification (Subtask B).

Their model uses late fusion. Visual features come from CLIP ViT-B/32, and the overlaid text is encoded by NepBERTa, a RoBERTa-style model specialised for Nepali. The training sets are small, so they also borrow examples from the other subtask. A heuristic treats negative sentiment as roughly equivalent to hate and positive sentiment as roughly equivalent to non-hate. The authors acknowledge that this equivalence is approximate, since sarcasm can blur it. Training also relied on image augmentations and a class-balanced loss.

On the hidden CodaBench test set, the system reached Macro F1 of 0.8052 for hate detection and 0.6881 for sentiment. The work suggests that pairing a general vision-language encoder with a language-specific text model is a workable recipe for under-resourced meme moderation.

Key points

  • Architecture: Late fusion of the CLIP ViT-B/32 image encoder with NepBERTa, using the Rajan/nepbertaTorch subword tokenizer.
  • Cross-task augmentation:
    • Subtask A training data grew from 1,068 to 1,656 samples by adding non-neutral Subtask B items.
    • Subtask B grew from 1,061 to 2,129 by mapping hate to "Negative" and no-hate to "Positive".
    • Each subtask has a validation split of 133 samples.
  • Official results (CodaBench, hidden test labels): Macro F1 of 0.8052 on Subtask A and 0.6881 on Subtask B.
  • Image augmentations: RandomResizedCrop, RandomHorizontalFlip, rotation up to 15°, and strong ColorJitter, applied on top of CLIP-specific normalisation.
  • Limitations noted by the authors:
    • The label mapping between sentiment and hate is only an approximation, because sarcastic "positive" memes may actually be hateful.
    • Bias, robustness and interpretability are not explicitly addressed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kathmandu UniversityKathmandu · Nepal7 works · 20 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksVision & language · 811 worksBias, fairness & safety · 587 worksLow-resource languages · 287 works

Cite

Cite as Sunil Regmi, Bipesh Subedi, Saugat Singh and Suman Shrestha. 2026. linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3t7nocwwhq8r

@inproceedings{regmi2026linus,
  title = {{linus@CHiPSAL 2026: Multimodal Hate Speech and Sentiment Detection in Low-Resource Memes Using Late-Fusion Hybrid Architecture}},
  author = {Sunil Regmi, Bipesh Subedi, Saugat Singh and Suman Shrestha},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/3t7nocwwhq8r},
  url = {https://doi.org/10.63317/3t7nocwwhq8r}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Greeshma Arya, Mohammad Kamrul Hasan, Ashish Bagwari, Nurhizam Safie, Shayla Islam, Fatima Rayan Awad Ahmed, Aaishani De, Muhammad Attique Khan, and Taher M. Ghazal. 2024. Multimodal hate speech detection in memes using contrastive language-image pre-training. IEEE Access, 12:22359–22375. find
  2. Kriti Dubey, Vaishnavi Srivastava, Garima Sharma, Nonita Sharma, Deepak Sharma, Uttam Ghosh, Osama Alfarraj, and Amr Tolba. 2025. Multimodal detection of offensive content in hindi memes. ACM Transactions on Asian and Low-Resource Language Information Processing. find
  3. Ahmed El-Sayed and Omar Nasr. 2024. Aast-nlp at multimodal hate speech event detection 2024 : A multimodal approach for classification of text-embedded images based on clip and bert-based models. In CASE. find
  4. Kazi Toufique Elahi, Tasnuva Binte Rahman, Shakil Shahriar, Samir Sarker, Sajib Kumar Saha Joy, and Faisal Muhammad Shah. 2023a. Explainable multimodal sentiment analysis on bengali memes. find
  5. Kazi Toufique Elahi, Tasnuva Binte Rahman, Shakil Shahriar, Samir Sarker, Sajib Kumar Saha Joy, and Faisal Muhammad Shah. 2023b. Explainable multimodal sentiment analysis on bengali memes. In Proceedings of the International Conference on Computer and Information Technology (ICCIT). find
  6. Aishvi Guleria, Kamya Varshney, Garima Pahwa, Shreya Singhal, and Nonita Sharma. 2024. Multimodal sentiment analysis of english and hinglish memes. Multimedia Tools and Applications. find
  7. Ramesh Kannan and Ratnavel Rajalakshmi. 2022. Multimodal code-mixed Tamil troll meme classification using feature fusion. In Proceedings of the First Workshop on Multimodal Machine Learning in Low-resource Languages, pages 1– 8, IIIT Delhi, New Delhi, India. Association for Computational Linguistics… find
  8. Md. Rezaul Karim, Sumon Kanti Dey, Tanhim Islam, Md. Shajalal, and Bharathi Raja Chakravarthi. 2022. Multimodal hate speech detection from bengali memes and texts. In Proceedings of the International Conference on Speech and Language Technologies for Low-Resource Languages. find
  9. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.
  10. Kshitij Rajput, Raghav Kapoor, Kaushal Rai, and Preeti Kaur. 2022. Hate me not: Detecting hate inducing memes in code switched languages. arXiv preprint arXiv:2204.11356. — [arXiv:2204.11356](https://arxiv.org/abs/2204.11356) find
  11. Kengatharaiyer Sarveswaran, Surendrabikram Thapa, Ashwini Vaidya, Tafseer Ahmed Khan, and Bal Krishna Bal. 2026. Findings of the second workshop on challenges in processing south asian languages (chipsal 2026). In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (…
  12. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Surabhi Adhikari, Hariram Veeramani, Mehwish Nasim, and Usman Naseem. 2025. Large language models (llm) in computational social science: prospects, current state, and challenges. Social Network Analysis and Mining, 15(1):1–30. find
  13. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Kengatharaiyer Sarveswaran, Bal Krishna Bal, and Usman Naseem. 2026. Multimodal hate and sentiment understanding in low-resource text-embedded images for online safety…
  14. Aidos Konyspay, Pakizar Shamoi, Malika Ziyada, and Zhusup Smambayev. 2025. Meme similarity and emotion detection using multimodal analysis. In 2025 International Conference on Activity and Behavior Computing (ABC), page 1–10. IEEE. find
  15. Shreyash Mishra, S Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. 2023. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes. find
  16. Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302–1308. IEEE. find
  17. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, 10. Language Resource References Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-… find
  18. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025a. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. find
  19. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025b. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042– 2051. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/regmi-2026-linus-chipsal-2026-multimodal-hate/