Elamur Research⌕
☰
home / papers / 2026 / MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

Paper · 2026 · Samir Wagle, Reewaj Khanal and Abiral Adhikari

Summary

The authors build a system for the CHiPSAL 2026 shared task on Nepali memes in Devanagari script. Subtask A is binary hate detection and Subtask B is three-way sentiment classification. The data are scarce: 1,068 memes, a skewed label split, and OCR-extracted text paired with images. The proposed model encodes images with CLIP (ViT-B/32) and text with BGE-M3. Both embeddings are projected into a smaller shared space and combined through 4-head self-attention. A learnable gate then decides, for each sample, how much to rely on each modality. The authors compare eight configurations under stratified 5-fold cross-validation: text-only, image-only, text-erased images, concatenation, voting and bagging ensembles, and the hybrid model. The hybrid model gives the best macro-F1 and corrects the text baseline's bias toward the majority "Hate" class. The ablation also shows two negative results. CLIP is close to chance on Devanagari content, and ensembles lose accuracy when training folds are this small. These findings are useful for designing multimodal moderation tools in low-resource South Asian languages.

Key points

  • Data: 1,068 Devanagari memes, with hate at 67.4% (a 2.07:1 ratio). Sentiment is split roughly 45% negative, 32% neutral and 23% positive. Each fold trains on about 854 samples. The content comes from Nepali social media in 2022–2023, mostly around one election cycle.
  • Main results: for Subtask A, the hybrid model (M7) attains a macro-F1 of 0.6834, while for Subtask B it attains 0.6102. The authors report a 5.9% macro-F1 gain over the text-only baseline on Subtask A.
  • Accuracy vs. macro-F1: the text-only model (M1) has the highest accuracy (0.7239) because it over-predicts "Hate". M7 lifts Non-Hate recall from 0.59 to 0.73.
  • Fine-tuning setup: only the last 2 of 12 CLIP layers and the last 4 of 24 BGE-M3 layers are fine-tuned. Tesseract OCR introduces roughly 22% character errors.
  • Error analysis: 52% of M7's false positives are political satire. 56% of its false negatives come from culturally coded threats or code-mixed out-of-vocabulary (OOV) tokens. Code is available at github.com/Tri-Yantra-Technologies/MEME-Fusion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kathmandu UniversityKathmandu · Nepal7 works · 20 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Samir Wagle, Reewaj Khanal and Abiral Adhikari. 2026. MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2womrc9pc2b7

@inproceedings{wagle2026meme,
  title = {{MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes}},
  author = {Samir Wagle, Reewaj Khanal and Abiral Adhikari},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/2womrc9pc2b7},
  url = {https://doi.org/10.63317/2womrc9pc2b7}
}

References

11 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abiral Adhikari, Prashant Manandhar, Reewaj Khanal, Samir Wagle, Praveen Acharya, and Bal Krishna Bal. 2024. Profanity and offensiveness detection in Nepali language using bidirectional LSTM models. In Proceedings of the 21st International Conference on Natural Language Processing (ICON), pages 515–… find
  2. Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  3. Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages.
  4. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597.
  5. Gauri Kitukale, Navneet Pratap Singh, Sidharth Quamara, et al. 2025. Ddcaf: Dynamic dual cross-attention fusion framework for multimodal hate speech detection. Research Square. Preprint, Version 1. find
  6. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohsen Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. In Proceedings of the 11th Forum for Information Retrieval Evaluat… find
  7. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR.
  8. Kengatharaiyer Sarveswaran, Surendrabikram Thapa, Ashwini Vaidya, Tafseer Ahmed Khan, and Bal Krishna Bal. 2026. Findings of the second workshop on challenges in processing south asian languages (chipsal 2026). In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (…
  9. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Kengatharaiyer Sarveswaran, Bal Krishna Bal, and Usman Naseem. 2026. Multimodal hate and sentiment understanding in low-resource text-embedded images for online safety…
  10. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025a. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. find
  11. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025b. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042–2051. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wagle-2026-meme-fusion-chipsal-2026-multimodal/