A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English
Paper · 2025 · Julian Baumler, Louis Blocher, Lars-Joel Frey et al.
Summary
Most work on automatic hate speech detection treats it as a yes/no decision. Real users need more detail. Content moderators, law enforcement and reporting centres want to know the target, severity or legal relevance of a message, and one message can fit several of these categories at once. This survey follows the systematic review procedure of Vom Brocke et al. and covers 46 publications on multi-label classification of English textual hate speech. It catalogues 28 datasets that can train such models and compares them by size, source, meta-concept, label-set structure, annotation process and inter-annotator agreement (IAA). It also reviews 24 papers that propose model architectures. The datasets turn out to be very heterogeneous. The models are mostly BERT-based or RNN-based, and they are evaluated so inconsistently that no state of the art can be named. The authors list the main open problems and give ten recommendations. These include transparent dataset creation, consolidated label sets, IAA reporting, validation on existing datasets, tasks tied to criminal relevance, and trying large foundation models. The survey is a map for anyone building fine-grained, overlapping-label classifiers.
Key points
- The corpus covers 46 publications: 28 multi-label-capable English datasets and 24 papers proposing models. The authors call it the first systematic survey of multi-label hate speech classification.
- Twitter/X is the most common data source. Datasets differ widely in label set, size, meta-concept, annotation method and IAA.
- The dominant architectures are BERT and RNNs such as Bi-LSTMs with attention. None of the surveyed model papers used large foundation models like GPT-4 or Llama 3.
- Four open issues are identified:
- imbalanced training data;
- heavy reliance on crowdsourcing for annotation;
- small and sparse datasets;
- missing methodological alignment between studies.
- In a side evaluation of binary models on HateCheck, Facebook's RoBERTa Hate Speech Model performed best on all metrics except Precision, where Google's Perspective API led.
Interesting ideas
- I1. Multi-class setups assume labels exclude each other, but one text can carry several overlapping categories at once, so multi-label is the honest framing.
- I2. The adopted definition rests only on content and linguistic form, because a model cannot observe speaker intent or effect on the victim from text alone.
- I3. A shared functional test suite (HateCheck) allowed several existing models to be compared on equal terms where the scattered original evaluations could not.
- I4. Vidgen et al.'s approach grows the training set over several human-in-the-loop rounds in which experts try to fool the current model, and the survey suggests it explains the best model's strength.
- I5. Instruction-tuned foundation models can receive class definitions directly in the prompt, which helps when labelled multi-label data is sparse.
For Mova
How these ideas could serve Mova, our small open language model — its files, stages and tests:
- I1 (frames): Speech-act, irony and implicature tags on a clause should be stored as an independent set of flags rather than a single forced choice, so that co-occurring readings stay visible and explainable.
- I2 (frames): Pragmatic labels in prag could be defined strictly by evidence in the UD tree and frames, so each label can be traced to observable textual cues.
- I3 (learn): A fixed suite of targeted test cases, including negative controls, could be run on every parser or reader version so comparisons do not depend on varying datasets and metrics.
- I4 (learn): Self-play rounds in which a generator crafts sentences meant to break the current parser or frame extractor could feed hard cases into silver data and repair-rule induction.
- I5 (learn): LLM teacher tiers could be given explicit written definitions of each frame or role label, producing silver annotations for rare label combinations that existing treebanks barely cover.
Institutions
Fields, topics and concepts
📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works
🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksBias, fairness & safety · 517 works
Cite
Cite as Julian Baumler, Louis Blocher, Lars-Joel Frey, Xian Chen, Markus Bayer and Christian Reuter. 2025. A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English. arXiv:2504.08609 · ACM Transactions on Knowledge Discovery from Data (2026).
@misc{baumler2025survey,
title = {{A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English}},
author = {Julian Baumler, Louis Blocher, Lars-Joel Frey, Xian Chen, Markus Bayer and Christian Reuter},
year = {2025},
doi = {10.1145/3820778},
eprint = {2504.08609},
archivePrefix = {arXiv},
url = {https://doi.org/10.1145/3820778}
}References
- Ahmed Abbasi, Abdul Rehman Javed, Farkhund Iqbal, Natalia Kryvinska, and Zunera Jalil. 2022. Deep learning for religious and continent-based toxic content detection and classification. Scientific Reports 12, 1 (Oct. 2022), 17478. doi:10.1038/s41598-022-22523-3 ↗
- Harika Abburi, Pulkit Parikh, Niyati Chhaya, and Vasudeva Varma. 2021. Fine-Grained Multi-label Sexism Classification Using a Semi-Supervised Multi-level Neural Approach. Data Science and Engineering 6, 4 (Dec. 2021), 359–379. doi:10.1007/s41019-021-00168-y ↗
- Harika Abburi, Pulkit Parikh, Niyati Chhaya, and Vasudeva Varma. 2024. Multi-task learning neural framework for categorizing sexism. Computer Speech & Language 83 (Jan. 2024), 101535. doi:10.1016/j.csl.2023.101535 ↗
- Apoorv Aditya, Rithwik Vinod, Ashish Kumar, Ishan Bhowmik, and J. Swaminathan. 2022. Classifying Speech into Offensive and Hate Categories along with Targeted Communities using Machine Learning. In 2022 International Conference on Inventive Computation Technologies (ICICT). IEEE, Nepal, 291–295. doi:10.1109/ICICT54344.2022.9850944 ↗
- Aish Albladi, Minarul Islam, Amit Das, Maryam Bigonah, Zheng Zhang, Fatemeh Jamshidi, Mostafa Rahgouy, Nilanjana Raychawdhary, Daniela Marghitu, and Cheryl Seals. 2025. Hate Speech Detection using Large Language Models: A Comprehensive Review. IEEE Access 13 (2025), 20871–20892. doi:10.1109/ACCESS.2025.3532397 ↗
- Fatimah Alkomah and Xiaogang Ma. 2022. A Literature Review of Textual Hate Speech Detection Methods and Datasets. Information 13, 6 (May 2022), 273. doi:10.3390/info13060273 ↗
- Sultan Alshamrani, Ahmed Abusnaina, Mohammed Abuhamad, Daehun Nyang, and David Mohaisen. 2021. Hate, Obscenity, and Insults: Measuring the Exposure of Children to Inappropriate Comments in YouTube. In Companion Proceedings of the Web Conference 2021. ACM, Ljubljana Slovenia, 508–515. doi:10.1145/3442442.3452314 ↗
- Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep Learning Models for Multilingual Hate Speech Detection. doi:10.48550/ARXIV.2004.06465 Version Number: 3. ↗
- Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa-Anke, Jiaxin Pei, and Jose Camacho-Collados. 2023. SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 12590–12607. doi:10.18653/v1/2023.findings-emnlp.838 ↗
- Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic Identification and Classification of Misogynistic Language on Twitter. In Natural Language Processing and Information Systems, Max Silberztein, Faten Atigui, Elena Kornyshova, Elisabeth Métais, and Farid Meziane (Eds.). Vol. 10859. Springer International Publishing, Cham, 57–64. doi:10.1007/978-3-319-91947-8_6 Series Title: Lecture Notes in Computer Science. ↗
- Femi Emmanuel Ayo, Olusegun Folorunso, Friday Thomas Ibharalu, and Idowu Ademola Osinuga. 2020. Machine learning techniques for hate speech classification of twitter data: State-of-the-art, future challenges and research directions. Computer Science Review 38 (2020), 100311. doi:10.1016/j.cosrev.2020.100311 ↗
- Michele Banko, Brendon MacKeen, and Laurie Ray. 2020. A Unified Taxonomy of Harmful Content. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 125–137. doi:10.18653/v1/2020.alw-1.16 ↗
- Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 54–63. doi:10.18653/v1/S19-2007 ↗
- Deeshant Bhattacharjee, Avishek Paul, and Deepak Kumar. 2023. Sentiment Analysis for Hateful Content on Social Media. In 2023 International Conference on Network, Multimedia and Information Technology (NMITCON). IEEE, Bengaluru, India, 1–6. doi:10.1109/NMITCON58196.2023.10275876 ↗
- Shiladitya Bhattacharya, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Atul Kr. Ojha. 2020. Developing a Multilingual Annotated Corpus of Misogyny and Aggression. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Ritesh Kumar, Atul Kr. Ojha, Bornini Lahiri, Marcos Zampieri, Shervin Malmasi, Vanessa Murdock, and Daniel Kadar (Eds.). European Language Resources Association (ELRA), Marseille, France, 158–168. https://aclanthology.org/2020.trac-1.25 ↗
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In 34th Conference on Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., Vancouver, Canada, 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf ↗
- Pete Burnap and Matthew L Williams. 2016. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. EPJ Data Science 5, 1 (Dec. 2016), 11. doi:10.1140/epjds/s13688-016-0072-6 ↗
- Julian Bäumler, Marc-André Kaufhold, Georg Voronin, and Christian Reuter. 2024. Towards an Online Hate Speech Classification Scheme for German Law Enforcement and Reporting Centers: Insights from Research and Practice. In Mensch und Computer 2024 – Workshopband. Gesellschaft für Informatik e.V., Karlsruhe, Germany, 1–11. doi:10. 18420/muc2024-mci-ws13-124 ↗
- Julian Bäumler, Thea Riebe, Marc-André Kaufhold, and Christian Reuter. 2025. Harnessing Inter-Organizational Collaboration and Automation to Combat Online Hate Speech: A Qualitative Study with German Reporting Centers. Proc. ACM Hum.-Comput. Interact. 9, 2 (2025), CSCW093. doi:10.1145/3710991 ↗
- Julian Bäumler, Georg Voronin, and Marc-André Kaufhold. 2025. Cyber Hate Awareness: Information Types and Technologies Relevant to the Law Enforcement and Reporting Center Domain. i-com - Journal of Interactive Media 24, 1 (2025), 20 pages. doi:10.1515/icom-2024-0062 ↗
- Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for Abusive Language Detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021). Association for Computational Linguistics, Online, 17–25. doi:10.18653/v1/2021.woah-1.3 ↗
- Jithin Cheriyan, Bastin Tony Roy Savarimuthu, and Stephen Cranefield. 2021. Towards offensive language detection and reduction in four Software Engineering communities. In Evaluation and Assessment in Software Engineering. ACM, Trondheim Norway, 254–259. doi:10.1145/3463274.3463805 ↗
- Naganna Chetty and Sreejith Alathur. 2018. Hate speech review in the context of online social networks. Aggression and Violent Behavior 40 (May 2018), 108–118. doi:10.1016/j.avb.2018.05.003 ↗
- Davide Chicco, Matthijs J. Warrens, and Giuseppe Jurman. 2021. The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment. IEEE Access 9 (2021), 78368–78381. doi:10.1109/ACCESS.2021.3084050 ↗
- Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. doi:10.48550/ARXIV.1412.3555 Version Number: 1. ↗
- Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 2819–2829. doi:10.18653/v1/P19-1271 ↗
- cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. 2019. Jigsaw Unintended Bias in Toxicity Classification. https://kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification ↗
- cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, nithum, and Will Cukierski. 2017. Toxic Comment Classification Challenge. https://kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge ↗
- Mithun Das, Somnath Banerjee, and Animesh Mukherjee. 2022. Data Bootstrapping Approaches to Improve Low Resource Abusive Language Detection for Indic Languages. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media. ACM, Barcelona Spain, 32–42. doi:10.1145/3511095.3531277 ↗
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. Proceedings of the International AAAI Conference on Web and Social Media 11, 1 (May 2017), 512–515. doi:10.1609/icwsm.v11i1.14955 ↗
- Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. A Comprehensive Dataset for German Offensive Language and Conversation Analysis. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). Association for Computational Linguistics, Seattle, Washington (Hybrid), 143–153. doi:10.18653/v1/2022.woah-1.14 ↗
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 4171–4186. doi:10.18653/v1/N19-1423 _eprint: 1810.04805. ↗
- Bakwa Dunka Dirting, Gloria A. Chukwudebe, Euphemia Chioma Nwokorie, and Ikechukwu Ignatius Ayogu. 2022. Multi-Label Classification of Hate Speech Severity on Social Media using BERT Model. In 2022 IEEE Nigeria 4th International Conference on Disruptive Technologies for Sustainable Development (NIGERCON). IEEE, Lagos, Nigeria, 1–5. doi:10.1109/NIGERCON54645.2022.9803164 ↗
- Farshid Faal, Jia Yuan Yu, and Ketra A Schmitt. 2021. Domain Adaptation Multi-task Deep Neural Network for Mitigating Unintended Bias in Toxic Language Detection. In Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021), Vol. 2. Scitepress, Virtual Event, 932–940. https://www.scitepress.org/Papers/2021/102661/102661.pdf ↗
- Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroğlu, and Marco Guerini. 2021. Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 3226–3240. doi:10.18653/v1/2021.acl-long.250 ↗
- Elisabetta Fersini, Debora Nozza, Paolo Rosso, and others. 2018. Overview of the evalita 2018 task on automatic misogyny identification (ami). In CEUR workshop proceedings, Vol. 2263. CEUR-WS, Torino, Italy, 1–9. doi:10.4000/ books.aaccademia.4497
- Darja Fišer, Tomaž Erjavec, and Nikola Ljubešić. 2017. Legal Framework, Dataset and Annotation Schema for Socially Unacceptable Online Discourse Practices in Slovene. In Proceedings of the First Workshop on Abusive Language Online. Association for Computational Linguistics, Vancouver, BC, Canada, 46–51. doi:10.18653/v1/W17-3007 ↗
- Paula Fortuna and Sérgio Nunes. 2019. A Survey on Automatic Detection of Hate Speech in Text. Comput. Surveys 51, 4 (July 2019), 1–30. doi:10.1145/3232676 ↗
- Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. Proceedings of the International AAAI Conference on Web and Social Media 12, 1 (June 2018), 491–500. doi:10.1609/icwsm.v12i1.14991 ↗
- Ignio Gagliardone, Danit Gal, Thiago Alves, and Gabriela Martinez. 2015. Countering Online Hate Speech. UNESCO Publishing, Paris. find
- Isuru Gunasekara and Isar Nejadgholi. 2018. A Review of Standard Text Classification Practices for Multi-label Toxicity Identification of Online Content. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Computational Linguistics, Brussels, Belgium, 21–25. doi:10.18653/v1/W18-5103 ↗
- Shrey Gupta, Pratyush Priyadarshi, and Manish Gupta. 2023. Hateful Comment Detection and Hate Target Type Prediction for Video Comments. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. ACM, Birmingham United Kingdom, 3923–3927. doi:10.1145/3583780.3615260 ↗
- Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 3309–3326. doi:10.18653/v1/2022.acl-long.234 ↗
- Ming Shan Hee, Shivam Sharma, Rui Cao, Palash Nandi, Preslav Nakov, Tanmoy Chakraborty, and Roy Ka-Wei Lee. 2024. Recent Advances in Online Hate Speech Moderation: Multimodality and the Role of Large Models. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA, 4407–4419. doi:10.18653/v1/2024.findings-emnlp.254 ↗
- Paulo Cezar De Q. Hermida and Eulanda M. Dos Santos. 2023. Detecting hate speech in memes: a review. Artificial Intelligence Review 56, 11 (Nov. 2023), 12833–12851. doi:10.1007/s10462-023-10459-7 ↗
- Sepp Hochreiter. 1998. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 06, 02 (April 1998), 107–116. doi:10.1142/S0218488598000094 ↗
- Muhammad Husnain, Adnan Khalid, and Numan Shafi. 2021. A Novel Preprocessing Technique for Toxic Comment Classification. In 2021 International Conference on Artificial Intelligence (ICAI). IEEE, Islamabad, Pakistan, 22–27. doi:10.1109/ICAI52203.2021.9445252 ↗
- Mai Ibrahim, Marwan Torki, and Nagwa El-Makky. 2018. Imbalanced Toxic Comments Classification Using Data Augmentation and Deep Learning. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, Orlando, FL, 875–878. doi:10.1109/ICMLA.2018.00141 ↗
- Md Saroar Jahan and Mourad Oussalah. 2023. A systematic review of hate speech automatic detection using natural language processing. Neurocomputing 546 (Aug. 2023), 126232. doi:10.1016/j.neucom.2023.126232 ↗
- Laszlo A. Jeni, Jeffrey F. Cohn, and Fernando De La Torre. 2013. Facing Imbalanced Data–Recommendations for the Use of Performance Metrics. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, Geneva, Switzerland, 245–251. doi:10.1109/ACII.2013.47 ↗
- Marc-André Kaufhold, Markus Bayer, Julian Bäumler, Christian Reuter, Stefan Stieglitz, Ali Sercan Basyurt, Milad Mirbabaie, Christoph Fuchss, and Kaan Eyilmez. 2023. CYLENCE: Strategies and Tools for Cross-Media Reporting, Detection, and Treatment of Cyberbullying and Hatespeech in Law Enforcement Agencies. In Mensch und Computer 2023 - Workshopband. Gesellschaft für Informatik e.V., Rapperswil, Switzerland, 1–8. doi:10.18420/MUC2023-MCI-WS01-211 ↗
- Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc., Long Beach, CA, USA, 1–9. https://proceedings.neurips.cc/paper_files/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf ↗
- Teo Keipi, Matti Näsi, Atte Oksanen, and Pekka Räsänen. 2016. Online Hate and Harmful Content: Cross-national perspectives (1 ed.). Routledge, London. doi:10.4324/9781315628370 ↗
- Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, Joe Hoover, Aida Azatian, Alyzeh Hussain, Austin Lara, Gabriel Cardenas, Adam Omary, Christina Park, Xin Wang, Clarisa Wijaya, Yong Zhang, Beth Meyerowitz, and Morteza Dehghani. 2022. Introducing the Gab Hate Corpus: defining and applying hate-based rhetoric to social media posts at scale. Language Resources and Evaluation 56, 1 (March 2022), 79–108. doi:10.1007/s10579-021-09569-x ↗
- Petra Kralj Novak, Teresa Scantamburlo, Andraž Pelicon, Matteo Cinelli, Igor Mozetič, and Fabiana Zollo. 2022. Handling Disagreement in Hate Speech Modelling. In Information Processing and Management of Uncertainty in Knowledge-Based Systems, Davide Ciucci, Inés Couso, Jesús Medina, Dominik Ślęzak, Davide Petturiti, Bernadette Bouchon-Meunier, and Ronald R. Yager (Eds.), Vol. 1602. Springer International Publishing, Cham, 681–695. doi:10. 1007/978-3-031-08974-9_54 Series Title: Communications in Computer and Information Science. find
- Ritesh Kumar, Shyam Ratan, Siddharth Singh, Enakshi Nandi, Laishram Niranjana Devi, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Akanksha Bansal. 2023. A multilingual, multimodal dataset of aggression and bias: the ComMA dataset. Language Resources and Evaluation 58 (Nov. 2023), 757–837. doi:10.1007/s10579-023-09696-7 ↗
- Salla-Maaria Laaksonen. 2023. The Datafication of Hate Speech. In Challenges and perspectives of hate speech research, Christian Strippel, Sünje Paasch-Colberg, Martin Emmer, and Joachim Trebbe (Eds.). Digital Communication, Vol. 12. Böhland & Schremmer Verlag, Berlin, 301–317. https://www.ssoar.info/ssoar/handle/document/86422 ISBN: 9783945681121 Publisher: [object Object]. ↗
- Laura Leets. 2002. Experiencing Hate Speech: Perceptions and Responses to Anti-Semitism and Antigay Speech. Journal of Social Issues 58, 2 (Jan. 2002), 341–361. doi:10.1111/1540-4560.00264 ↗
- Amanda Lenhart, Michele Ybarra, Kathryn Zickuhr, and Myeshia Price-Feeney. 2016. Online harassment, digital abuse, and cyberstalking in America. Research Report. Data and Society Research Institute. Center for Innovative Public Research. https://www.datasociety.net/pubs/oh/Online_Harassment_2016.pdf ↗
- Han Liu, Pete Burnap, Wafa Alorainy, and Matthew L. Williams. 2019. Fuzzy Multi-task Learning for Hate Speech Type Identification. In The World Wide Web Conference. ACM, San Francisco CA USA, 3006–3012. doi:10.1145/3308558. 3313546 ↗
- Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PLOS ONE 14, 8 (Aug. 2019), e0221152. doi:10.1371/journal.pone.0221152 ↗
- Thomas Mandl, Sandip Modha, Anand Kumar M, and Bharathi Raja Chakravarthi. 2020. Overview of the HASOC Track at FIRE 2020: Hate Speech and Offensive Language Identification in Tamil, Malayalam, Hindi, English and German. In Forum for Information Retrieval Evaluation. ACM, Hyderabad India, 29–32. doi:10.1145/3441501.3441517 ↗
- Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages. In Proceedings of the 11th Forum for Information Retrieval Evaluation. ACM, Kolkata India, 14–17. doi:10.1145/3368567.3368584 ↗
- Zainab Mansur, Nazlia Omar, and Sabrina Tiun. 2023. Twitter Hate Speech Detection: A Systematic Review of Methods, Taxonomy Analysis, Challenges, and Opportunities. IEEE Access 11 (2023), 16226–16249. doi:10.1109/ ACCESS.2023.3239375 ↗
- Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection. Proceedings of the AAAI Conference on Artificial Intelligence 35, 17 (May 2021), 14867–14875. doi:10.1609/aaai.v35i17.17745 ↗
- Ahmed Cherif Mazari, Nesrine Boudoukhani, and Abdelhamid Djeffal. 2024. BERT-based ensemble learning for multi-aspect hate speech detection. Cluster Computing 27, 1 (Feb. 2024), 325–339. doi:10.1007/s10586-022-03956-x ↗
- Sudhanshu Mishra, Shivangi Prasad, and Shubhanshu Mishra. 2021. Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media. SN Computer Science 2, 2 (April 2021), 72. doi:10.1007/s42979-021-00455-5 ↗
- Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2022. ETHOS: a multi-label hate speech detection dataset. Complex & Intelligent Systems 8, 6 (Dec. 2022), 4663–4678. doi:10.1007/s40747-021-00608-2 ↗
- Rachna Narula and Poonam Chaudhary. 2025. A comprehensive review on detection of hate speech for multi-lingual data. Social Network Analysis and Mining 14, 1 (Jan. 2025), 244. doi:10.1007/s13278-024-01401-y ↗
- Laura Beth Nielsen. 2002. Subtle, Pervasive, Harmful: Racist and Sexist Remarks in Public as Hate Speech. Journal of Social Issues 58, 2 (Jan. 2002), 265–280. doi:10.1111/1540-4560.00260 ↗
- Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and Multi-Aspect Hate Speech Analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 4674–4683. doi:10.18653/v1/D19-1474 ↗
- Kadir Bulut Ozler, Kate Kenski, Steve Rains, Yotam Shmargad, Kevin Coe, and Steven Bethard. 2020. Fine-tuning for multi-domain and multi-label uncivil language detection. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 28–33. doi:10.18653/v1/2020.alw-1.4 ↗
- Sünje Paasch-Colberg, Christian Strippel, Joachim Trebbe, and Martin Emmer. 2021. From Insult to Hate Speech: Mapping Offensive Language in German User Comments on Immigration. Media and Communication 9, 1 (Feb. 2021), 171–180. doi:10.17645/mac.v9i1.3399 ↗
- Sinno Jialin Pan and Qiang Yang. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345–1359. doi:10.1109/TKDE.2009.191 ↗
- Pulkit Parikh, Harika Abburi, Pinkesh Badjatiya, Radhika Krishnan, Niyati Chhaya, Manish Gupta, and Vasudeva Varma. 2019. Multi-label Categorization of Accounts of Sexism using a Neural Framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 1642–1652. doi:10.18653/v1/D19-1174 ↗
- Pulkit Parikh, Harika Abburi, Niyati Chhaya, Manish Gupta, and Vasudeva Varma. 2021. Categorizing Sexism and Misogyny through Neural Approaches. ACM Transactions on the Web 15, 4 (Nov. 2021), 1–31. doi:10.1145/3457189 ↗
- Guy Paré, Marie-Claude Trudel, Mirou Jaana, and Spyros Kitsiou. 2015. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management 52, 2 (March 2015), 183–199. doi:10.1016/j.im.2014.08.008 ↗
- Juan Manuel Perez, Mariela Rajngewerc, Juan Carlos Giudici, Damián Ariel Furman, Franco Luque, Laura Alonso Alemany, and María Vanina Martínez. 2023. pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasks. doi:10.21203/rs.3.rs-3570648/v1 ↗
- Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, and Paolo Rosso. 2023. Definitions Matter: Guiding GPT for Multi-label Classification. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 4054–4063. doi:10.18653/v1/2023.findings-emnlp.267 ↗
- Jürgen Pfeffer, Angelina Mooseder, Jana Lasser, Luca Hammer, Oliver Stritzel, and David Garcia. 2023. This Sample Seems to Be Good Enough! Assessing Coverage and Temporal Reliability of Twitter’s Academic API. Proceedings of the International AAAI Conference on Web and Social Media 17 (June 2023), 720–729. doi:10.1609/icwsm.v17i1.22182 ↗
- Laura Plaza, Jorge Carrillo-de Albornoz, Roser Morante, Enrique Amigó, Julio Gonzalo, Damiano Spina, and Paolo Rosso. 2023. Overview of EXIST 2023 – Learning with Disagreement for Sexism Identification and Characterization. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Anastasia Giachanou, Dan Li, Mohammad Aliannejadi, Michalis Vlachos, Guglielmo Faggioli, and Nicola Ferro (Eds.). Vol. 14163. Springer Nature Switzerland, Cham, 316–342. doi:10.1007/978-3-031-42448-9_23 Series Title: Lecture Notes in Computer Science. ↗
- Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation 55, 2 (June 2021), 477–523. doi:10.1007/s10579-020-09502-8 ↗
- Ilan Price, Jordan Gifford-Moore, Jory Flemming, Saul Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. 2020. Six Attributes of Unhealthy Conversations. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 114–124. doi:10.18653/v1/2020.alw-1.15 ↗
- Gil Ramos, Fernando Batista, Ricardo Ribeiro, Pedro Fialho, Sérgio Moro, António Fonseca, Rita Guerra, Paula Carvalho, Catarina Marques, and Cláudia Silva. 2024. A comprehensive review on automatic hate speech detection in the age of the transformer. Social Network Analysis and Mining 14, 1 (Oct. 2024), 204. doi:10.1007/s13278-024-01361-3 ↗
- Pratik Ratadiya and Deepak Mishra. 2019. An Attention Ensemble Based Approach for Multilabel Profanity Detection. In 2019 International Conference on Data Mining Workshops (ICDMW). IEEE, Beijing, China, 544–550. doi:10.1109/ ICDMW.2019.00083 find
- Anchal Rawat, Santosh Kumar, and Surender Singh Samant. 2024. Hate speech detection in social media: Techniques, recent trends, and future challenges. WIREs Computational Statistics 16, 2 (March 2024), e1648. doi:10.1002/wics.1648 ↗
- Julian Risch and Ralf Krestel. 2020. Toxic Comment Detection in Online Discussions. In Deep Learning-Based Approaches for Sentiment Analysis, Basant Agarwal, Richi Nayak, Namita Mittal, and Srikanta Patnaik (Eds.). Springer Singapore, Singapore, 85–109. doi:10.1007/978-981-15-1216-2_4 Series Title: Algorithms for Intelligent Systems. ↗
- Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent Multi-Task Architecture Learning. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (July 2019), 4822–4829. doi:10.1609/aaai. v33i01.33014822 ↗
- Maciej Rybinski, William Miller, Javier Del Ser, Miren Nekane Bilbao, and José F. Aldana-Montes. 2018. On the Design and Tuning of Machine Learning Models for Language Toxicity Classification in Online Platforms. In Intelligent Distributed Computing XII, Javier Del Ser, Eneko Osaba, Miren Nekane Bilbao, Javier J. Sanchez-Medina, Massimo Vecchio, and Xin-She Yang (Eds.). Vol. 798. Springer International Publishing, Cham, 329–343. doi:10.1007/978-3-319-99626-4_29 Series Title: Studies in Computational Intelligence. ↗
- Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hate-Check: Functional Tests for Hate Speech Detection Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 41–58. doi:10.18653/v1/2021.acl-long.4 ↗
- Hafiz Hassaan Saeed, Khurram Shahzad, and Faisal Kamiran. 2018. Overlapping Toxic Sentiment Classification Using Deep Neural Architectures. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE, Singapore, Singapore, 1361–1366. doi:10.1109/ICDMW.2018.00193 ↗
- Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard Jansen. 2018. Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media. Proceedings of the International AAAI Conference on Web and Social Media 12, 1 (June 2018), 330–339. doi:10.1609/icwsm.v12i1.15028 ↗
- Brennan Schaffner, Arjun Nitin Bhagoji, Siyuan Cheng, Jacqueline Mei, Jay L Shen, Grace Wang, Marshini Chetty, Nick Feamster, Genevieve Lakier, and Chenhao Tan. 2024. "Community Guidelines Make this the Best Party on the Internet": An In-Depth Study of Online Platforms’ Content Moderation Policies. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–16. doi:10.1145/3613904.3642333 ↗
- Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R. Brubaker. 2021. A Framework of Severity for Harmful Content Online. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (Oct. 2021), 1–33. doi:10.1145/3479512 ↗
- Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, Valencia, Spain, 1–10. doi:10.18653/v1/W17-1101 ↗
- Johannes Schäfer. 2023. Bias Mitigation for Capturing Potentially Illegal Hate Speech. Datenbank-Spektrum 23, 1 (March 2023), 41–51. doi:10.1007/s13222-023-00439-0 ↗
- Andrew Sellars. 2016. Defining Hate Speech. Research Publication 2016-20. Berkman Klein Center for Internet and Society, Cambridge, MA, USA. https://www.ssrn.com/abstract=2882244 ↗
- Alexandra A. Siegel. 2020. Online Hate Speech. In Social Media and Democracy: The State of the Field, Prospects for Reform, Nathaniel Persily and Joshua A. Tucker (Eds.). Cambridge University Press, Cambridge, MA, USA, 56–88. ↗
- Mohit Singhal, Chen Ling, Pujan Paudel, Poojitha Thota, Nihal Kumarswamy, Gianluca Stringhini, and Shirin Nilizadeh. 2023. SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, Delft, Netherlands, 868–895. doi:10.1109/EuroSP57164.2023.00056 ↗
- Malliga Subramanian, Veerappampalayam Easwaramoorthy Sathiskumar, G. Deepalakshmi, Jaehyuk Cho, and G. Manikandan. 2023. A survey on hate speech detection and sentiment analysis using machine learning and deep learning models. Alexandria Engineering Journal 80 (Oct. 2023), 110–121. doi:10.1016/j.aej.2023.08.038 ↗
- Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-Label Classification: An Overview. International Journal of Data Warehousing and Mining 3, 3 (July 2007), 1–13. doi:10.4018/jdwm.2007070101 ↗
- Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for Toxic Comment Classification: An In-Depth Error Analysis. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Computational Linguistics, Brussels, Belgium, 33–42. doi:10.18653/v1/W18-5105 ↗
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Isabelle Guyon, Ulrike von Luxenburg, Samy Bengio, Hannah M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). Curran Associates, Inc., Long Beach, CA, USA, 5998–6008. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html ↗
- Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online. Association for Computational Linguistics, Florence, Italy, 80–93. doi:10.18653/v1/W19-3509 ↗
- Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 1667–1682. doi:10.18653/v1/2021.acl-long.132 ↗
- Jan Vom Brocke, Alexander Simons, Kai Riemer, Björn Niehaves, Ralf Plattfaut, and Anne Cleven. 2015. Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research. Communications of the Association for Information Systems 37 (2015), 205–224. doi:10.17705/1CAIS.03709 ↗
- Zeerak Waseem. 2016. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science. Association for Computational Linguistics, Austin, Texas, 138–142. doi:10.18653/v1/W16-5618 ↗
- Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In Proceedings of the NAACL Student Research Workshop. Association for Computational Linguistics, San Diego, California, 88–93. doi:10.18653/v1/N16-2013 ↗
- Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 Workshop. Austrian Academy of Sciences, Vienna, Austria, 1–10. Publisher: Verlag der Österreichischen Akademie der Wissenschaften. find
- Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Perth Australia, 1391–1399. doi:10.1145/3038912.3052591 ↗
- Wenjie Yin and Arkaitz Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science 7 (June 2021), e598. doi:10.7717/peerj-cs.598 ↗
- Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the 2019 Conference of the North. Association for Computational Linguistics, Minneapolis, Minnesota, 1415–1420. doi:10.18653/v1/N19-1144 ↗
- Min-Ling Zhang and Zhi-Hua Zhou. 2014. A Review on Multi-Label Learning Algorithms. IEEE Transactions on Knowledge and Data Engineering 26, 8 (Aug. 2014), 1819–1837. doi:10.1109/TKDE.2013.39 ↗
- Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional Long Short-Term Memory Networks for Relation Classification. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, Hai Zhao (Ed.). Association for Computational Linguistics, Shanghai, China, 73–78. https://aclanthology.org/Y15-1009/ ↗
- Yu Zhang and Qiang Yang. 2018. An overview of multi-task learning. National Science Review 5, 1 (Jan. 2018), 30–43. doi:10.1093/nsr/nwx105 ↗
- Zhixue Zhao, Ziqi Zhang, and Frank Hopfgartner. 2021. A Comparative Study of Using Pre-trained Language Models for Toxic Comment Classification. In Companion Proceedings of the Web Conference 2021. ACM, Ljubljana Slovenia, 500–507. doi:10.1145/3442442.3452313 ↗
- Frederike Zufall, Marius Hamacher, Katharina Kloppenborg, and Torsten Zesch. 2022. A Legal Approach to Hate Speech – Operationalizing the EU’s Legal Framework against the Expression of Hatred as an NLP Task. In Proceedings of the Natural Legal Language Processing Workshop 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 53–64. doi:10.18653/v1/2022.nllp-1.5 ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/baumler-2025-survey-machine-learning-models-datasets/