Elamur Research⌕
☰
home / papers / 2026 / Leveraging Semi-Supervised Learning for Multimodal Hate Speech Data Annotation and Detection

Leveraging Semi-Supervised Learning for Multimodal Hate Speech Data Annotation and Detection

Summary

Automatic hate speech detection is limited by the shortage of large annotated corpora. The authors test whether semi-supervised learning (SSL) can offset this for Polish content in both audio and text. Their labeled data come from the Warsaw Multimodal Hate Speech (WMHS) Database, which was gathered through keyword searches on Facebook, YouTube, TikTok and BanBye. A much larger unlabeled pool comes from similar sources. Each modality uses a multilingual pre-trained encoder with a linear classification head. Three SSL schemes are compared on top of these encoders: Fix-match, Full-match and All-match. Each one turns the model's own predictions on unlabeled items into pseudo-labels, but they differ in how much of the unlabeled data they actually use. SSL improves results over training on labeled data alone, and combining audio and text gives the best scores. The study also compares weak augmentation strategies and checks a subset of pseudo-labels for quality. This tests whether SSL could reduce the burden of manual annotation, which is useful for low-resource languages and for moderation under the EU Digital Services Act.

Key points

  • Encoders: Wav2Vec2.0-XLSR (large, wav2vec2-large-xlsr-53) handles speech and XLM-RoBERTa (base) handles text. Each has a linear classifier on top.
  • SSL methods:
    • Fix-match keeps pseudo-labels only when prediction confidence on weakly augmented samples clears a fixed threshold.
    • Full-match adds an adaptive negative loss and an entropy meaning loss, so that every unlabeled sample contributes.
    • All-match sets a Class-Adaptive Threshold (CAT) from an EMA of model confidence, scaled by the L2 norms of the classifier weights.
  • Labeled data (WMHS): 770 Polish hate samples (about 9 hours) plus 763 further samples. The extracted text breaks off before describing the second group.
  • Best F1 scores: 0.851 for speech, 0.957 for text and 0.959 for multimodal fusion.
  • Further analyses: how varying the weak augmentation applied to the labeled set affects results, and a validation of pseudo-label quality to judge whether SSL can support data annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4un2wjkpdn2m

@inproceedings{rammohan2026leveraging,
  title = {{Leveraging Semi-Supervised Learning for Multimodal Hate Speech Data Annotation and Detection}},
  author = {Rammohan, Rathi Adarshi and Ren, Zhao and Puchała, Dominik and Świderska, Aleksandra and Küster, Dennis and Schultz, Tanja},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4un2wjkpdn2m},
  url = {https://doi.org/10.63317/4un2wjkpdn2m}
}

References

37 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, and Shinji Watanabe. 2024. On the Evaluation of Speech Foundation Models for Spoken Language Understanding. In Findings of the Association f… find
  2. Nur Heri Cahyana, Shoffan Saifullah, Yuli Fauziah, Agus Sasmito Aribowo, and Rafal Drezewski. 2022. Semi-supervised Text Annotation for Hate Speech Detection Using K-Nearest Neighbors and Term Frequency-Inverse Document Frequency. International Journal of Advanced Computer Science and Applications,… find
  3. Federica Casarosa. 2023. Out-of-court Dispute Settlement Mechanisms for Failures in Content Moderation. Journal of Intellectual Property, Information Technology and Electronic Commerce Law, 14:391. find
  4. O. Chapelle, B. Scholkopf, and A. Zien, Eds. 2009. Semi-Supervised Learning. IEEE Transactions on Neural Networks, 20(3):542–542. find
  5. Yuhao Chen, Xin Tan, Borui Zhao, Zhaowei Chen, Renjie Song, Jiajun Liang, and Xuequan Lu. 2023. Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7548–7557. find
  6. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised Cross-Lingual Representation Learning for Speech Recognition. arXiv preprint arXiv:2006.13979. — [arXiv:2006.13979](https://arxiv.org/abs/2006.13979) find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. arXiv preprint arXiv:1911.02116. find
  8. Tithi Rani Das, Tanjim Mahmud, Abubokor Ha-nip, and Mohammad Shahadat Hossain. 2025. Harmful Tweet Detection using Supervised and Semi-Supervised Learning Techniques. In 2025 International Conference on Inventive Computation Technologies (ICICT), pages 1421–1427. find
  9. Arne Dreißigacker, Philipp Müller, Anna Isenhardt, and Jonas Schemmel. 2024. Online Hate Speech Victimization: Consequences for Victims’ Feelings of Insecurity. Crime Science, 13(1):4. find
  10. Chiara Patricia Drolsbach and Nicolas Pröllochs. 2024. Content Moderation on Social Media in the EU: Insights from the DSA Transparency Database. In Proceedings of the ACM Web Conference 2024, pages 939–942. find
  11. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. arXiv preprint arXiv:2109.05322. find our card
  12. European Commission. 2024. The Digital Services Act (DSA). Official website of the European Commission. Accessed: 2025-10-14. find
  13. David Garcia, Arvid Kappas, Dennis Küster, and Frank Schweitzer. 2016. The Dynamics of Emotions in Online Interaction. Royal Society Open Science, 3(8):160059. find
  14. Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Mingyue Niu, and Jiangyan Yi. 2018. Speech Emotion Recognition Using Semi-Supervised Learning with Ladder Networks. In 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), pages 1–5. IEEE. find
  15. Jessica Lin. 2022. Leveraging World Knowledge in Implicit Hate Speech Detection. arXiv preprint arXiv:2212.14100. find
  16. Florian Ludwig, Klara Dolos, Ana Alves-Pinto, and Torsten Zesch. 2024. Unraveling the dynamics of semi-supervised hate speech detection: The impact of unlabeled data characteristics and pseudo-labeling strategies. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1974–19… find
  17. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867– 14875. find
  18. Dima Mohammed. 2025. Where the hate lies in soft hate speech: the argumentative potential in hostile public spheres. Critical Discourse Studies, pages 1–14. find
  19. Yassine Ouali, Céline Hudelot, and Myriam Tami. 2020. An Overview of Deep Semi-Supervised Learning. arXiv preprint arXiv:2006.05278. — [arXiv:2006.05278](https://arxiv.org/abs/2006.05278) find
  20. Chanho Park and Thomas Hain. 2025. Semi-Supervised Learning for Automatic Speech Recognition with Word Error Rate Estimation and Targeted Domain Data Selection. In Proceedings of Interspeech 2025. International Speech Communication Association (ISCA). find
  21. Orchid Chetia Phukan, Arun Balaji Buduru, and Rajesh Sharma. 2023. A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition. arXiv preprint arXiv:2304.11472. — [arXiv:2304.11472](https://arxiv.org/abs/2304.11472) find
  22. Dominik Puchała, Rathi Adarshi Rammohan, Dennis Küster, Tanja Schultz, and Aleksandra Świderska. 2025. The Warsaw Multimodal Hate Speech Database (WMHS): Development, initial validation, and baseline automatic detection. Manuscript submitted for publication. find
  23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of International Conference on Machine Learning. 27 pages. find our card
  24. YCAP Reddy, P Viswanath, and B Eswara Reddy. 2018. Semi-Supervised Learning: A Brief Review. International Journal of Engineering & Technology, 7(1.8):81. find
  25. Zhao Ren, Rathi Adarshi Rammohan, Kevin Scheck, Sheng Li, and Tanja Schultz. 2025. End-to-end Acoustic-linguistic Emotion and Intent Recognition Enhanced by Semi-Supervised Learning. arXiv preprint arXiv:2507.07806. — [arXiv:2507.07806](https://arxiv.org/abs/2507.07806) find
  26. Shoffan Saifullah, Rafał Dreżewski, Felix Andika Dwiyanto, Agus Sasmito Aribowo, Yuli Fauziah, and Nur Heri Cahyana. 2024. Automated Text Annotation Using a Semi-supervised Approach with Meta Vectorizer and Machine Learning Algorithms for Hate Speech Detection. Applied Sciences, 14(3):1078. find
  27. Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. Advances in Neural Information Processing Systems, 33:596–608. find
  28. Wiktor Soral, Michał Bilewicz, and Mikołaj Winiewski. 2018. Exposure to Hate Speech Increases Prejudice Through Desensitization. Aggressive Behavior, 44(2):136–146. find
  29. Wiktor Soral, Aleksandra Świderska, Dominik Puchała, and Michał Bilewicz. 2023. Desensitization to Hate Speech: Examination Using Heart Rate Measurement. Aggressive Behavior, 50(1):e22118. find
  30. Betsy Sparrow and Ljubica Chatman. 2013. Social Cognition in the Internet Age: Same As It Ever Was? Psychological Inquiry, 24(4):273–292. Publisher: Routledge _eprint: https://doi.org/10.1080/1047840X.2013.827079. — [doi:10.1080/1047840x.2013.827079](https://doi.org/10.1080/1047840x.2013.827079) ↗
  31. Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2019. End-to-end ASR: From Supervised to Semi-Supervised Learning with Modern Architectures. arXiv preprint arXiv:1911.08460. — [arXiv:1911.08460](https://arxiv.org/abs/1911.08460) find
  32. Pham Thai Hoang Tung, Nguyen Tan Viet, Ngo Tien Anh, and Phan Duy Hung. 2023. SemiMemes: A Semi-Supervised Learning Approach for Multimodal Memes Analysis. In International Conference on Computational Collective Intelligence, pages 565–577. Springer. find
  33. Dominique S. Wirz and Florin Zai. 2025. Infotainment on Social Media: How News Companies Combine Information and Entertainment in News Stories on Instagram and TikTok. Digital Journalism, 13(7):1249– 1270. Publisher: Routledge _eprint: https://doi.org/10.1080/21670811.2025.2464062. — [doi:10.1080/21670811.2025.2464062](https://doi.org/10.1080/21670811.2025.2464062) ↗
  34. Zhiyu Wu and Jinshi Cui. 2024. AllMatch: Exploiting All Unlabeled Data for Semi-Supervised Learning. arXiv preprint arXiv:2406.15763. — [arXiv:2406.15763](https://arxiv.org/abs/2406.15763) find
  35. Michał Wypych and Michał Bilewicz. 2024. Psychological Toll of Hate Speech: The Role of Acculturation Stress in the Effects of Exposure to Ethnic Slurs on Mental Health among Ukrainian Immigrants in Poland. Cultural Diversity & Ethnic Minority Psychology, 30(1):35–44. find
  36. Sheng Zhang, Min Chen, Jincai Chen, Yuan-Fang Li, Yiling Wu, Minglei Li, and Chuanbo Zhu. 2021. Combining Cross-Modal Knowledge Transfer and Semi-Supervised Learning for Speech Emotion Recognition. Knowledge-Based Systems, 229:107340. find
  37. Kamila Zochniak, Oliwia Lewicka, Zuzanna Wybrańska, and Michał Bilewicz. 2023. Homophobic Hate Speech Affects Well-Being of Highly Identified LGBT People. Journal of Language and Social Psychology, 42(4). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rammohan-2026-leveraging-semi-supervised-learning-multimodal/