Elamur Research⌕
☰
home / papers / 2026 / Team Oryu@CHiPSAL 2026: Integrating Text and Vision Transformers for Multimodal Hate Speech Detection in Memes

Team Oryu@CHiPSAL 2026: Integrating Text and Vision Transformers for Multimodal Hate Speech Detection in Memes

Paper · 2026 · Noore Tamanna Orny, Joyeta Barua Moni, Md. Abtahee Kabir et al.

Summary

The paper describes Team Oryu's system for the multimodal hate speech subtask of the CHiPSAL 2026 shared task. In memes, hatefulness often depends on how the caption and the picture interact, so models that read only text or only images miss part of the meaning. The authors work with a Nepali meme dataset in which each item has an image, its embedded text (Nepali, English or code-switched) and a binary hate/non-hate label. Their model uses late fusion. XLM-RoBERTa-base encodes the multilingual caption, and a Vision Transformer (ViT) encodes the image. The two embeddings are concatenated and passed through a fully connected classification head. Because hateful memes outnumber benign ones about two to one, training uses a class-weighted cross-entropy loss. The system reached a Macro F1 of 0.66 on the hidden test set and placed 5th on the leaderboard. The authors present it as a simple, computationally light baseline for detecting hateful memes in low-resource, code-mixed South Asian settings.

Key points

  • Data: a Nepali multimodal meme corpus (Thapa et al., 2025b,c) annotated under the CrisisHateMM guidelines. It has 1,068 training items (720 hate / 348 non-hate), 133 validation items (98 / 35) and 134 test items.
  • Architecture: late fusion of XLM-RoBERTa-base text features (768-dimensional) and ViT image features. Fusion is by concatenation, followed by a fully connected head that outputs a binary decision.
  • Preprocessing: text is tokenized to at most 128 tokens. Images are resized to 224×224, normalized and randomly flipped horizontally during training.
  • Imbalance handling: class-weighted cross-entropy loss addresses the dominance of the hate class.
  • Result: Macro F1 = 0.66, ranking 5th in the CHiPSAL 2026 multimodal hate detection leaderboard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 worksBias, fairness & safety · 517 works

Cite

Cite as Noore Tamanna Orny, Joyeta Barua Moni, Md. Abtahee Kabir and Hasan Murad. 2026. Team Oryu@CHiPSAL 2026: Integrating Text and Vision Transformers for Multimodal Hate Speech Detection in Memes. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2fb8jq6jj36y

@inproceedings{orny2026team,
  title = {{Team Oryu@CHiPSAL 2026: Integrating Text and Vision Transformers for Multimodal Hate Speech Detection in Memes}},
  author = {Noore Tamanna Orny, Joyeta Barua Moni, Md. Abtahee Kabir and Hasan Murad},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/2fb8jq6jj36y},
  url = {https://doi.org/10.63317/2fb8jq6jj36y}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
  3. Susmita Das, Arpita Dutta, Kingshuk Roy, Abir Mondal, and Arnab Mukhopadhyay. 2024. A survey on automatic online hate speech detection in low-resource languages. find
  4. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. find
  5. Sujal Maharjan, Astha Shrestha, Shuvam Thakur, and Rabin Thapa. 2025. Multimodal kathmandu@CASE 2025: Task-specific adaptation of multimodal transformers for hate, stance, and humor detection. In Proceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio-politica… find
  6. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 807–814, Madison, WI, USA. Omnipress. find
  7. Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302–1308. IEEE. find
  8. Paloma Piot and Javier Parapar. 2025. Towards efficient and explainable hate speech detection via model distillation. In European Conference on Information Retrieval, pages 376–392. Springer. find
  9. Meng Qi and Chung-Lun Wei. 2025. Lamha: Efficient multimodal malicious meme classification via lora-tuned adaptation and attentive-mlp fusion. In Proceedings of the 2025 7th International Conference on Control and Computer Vision, ICCCV ’25, page 62–68, New York, NY, USA. Association for Computing M… find
  10. Texts, pages 20–31, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  11. Surendrabikram Thapa, Shuvam Shiwakoti, Siddhant Bikram Shah, Kritesh Rauniyar, Laxmi Thapa, Surabhi Adhikari, Kristina T. Johnson, Kengatharaiyer Sarveswaran, Bal Krishna Bal, and Usman Naseem. 2026. Multimodal hate and sentiment understanding in low-resource text-embedded images for online safety…
  12. Surendrabikram Thapa, Hariram Veeramani, Liang Hu, Qi Zhang, Wei Wang, and Usman Naseem. 2025b. A multimodal prompt-based framework for analyzing code-mixed and low-resource memes. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 1913–1923. find
  13. Surendrabikram Thapa, Hariram Veeramani, Imran Razzak, Roy Ka-Wei Lee, and Usman Naseem. 2025c. Cross platform multimodal retrieval augmented distillation for code-switched content understanding. In Companion Proceedings of the ACM on Web Conference 2025, pages 2042– 2051. find
  14. Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. 2020. Visual transformers: Token-based image representation and processing for computer vision. find
  15. Kengatharaiyer Sarveswaran, Surendrabikram Thapa, Ashwini Vaidya, Tafseer Ahmed Khan, and Bal Krishna Bal. 2026. Findings of the second workshop on challenges in processing south asian languages (chipsal 2026). In Proceedings of the Second Workshop on Challenges in Processing South Asian Languages (…
  16. Surendrabikram Thapa, Siddhant Bikram Shah, Kritesh Rauniyar, Shuvam Shiwakoti, Surabhi Adhikari, Hariram Veeramani, Kristina T. Johnson, Ali Hurriyetoglu, Hristo Tanev, and Usman Naseem. 2025a. Multimodal hate, humor, and stance event detection in marginalized sociopolitical movements. In Proceedin… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/orny-2026-team-oryu-chipsal-2026-integrating/