Hate Speech Dataset from a White Supremacy Forum
Summary
Automatic hate speech detection lacks shared, openly released training data. Earlier annotation efforts often used loosely specified criteria, which produced inconsistent labels. The Vicomtech authors address this with a public English corpus drawn from Stormfront, a large white nationalist forum. They sampled posts from 22 sub-forums, filtered out non-English text, and split the posts into sentences with ixa-pipes. Annotators then labelled each sentence with a custom tool that lets them open the surrounding post when meaning is unclear. The annotators wrote the guidelines together and refined them over the first batches. A sentence counts as hate only if it is a deliberate attack, targets a group, and is motivated by that group's identity.
The paper analyses the resulting data and reports baseline classifiers, including SVM and CNN models. Its error analysis shows two common failure types: sentences that need context or world knowledge, and offensive vocabulary used without hateful intent. The dataset matters because it offers sentence-level labels plus conversation metadata, giving a reusable benchmark. It also shows how subjective hate annotation remains.
Key points
- 10,568 sentences were annotated in 10 batches, each built from 500 posts (about 1,000 sentences). Forum content dates from 2002 to 2017.
- Cleaning removed sentences under 3 or over 50 words, discarding 652 sentences (6.17%). Both the raw and the "clean" versions are released on GitHub (aitor-garcia-p/hate-speech-dataset).
- The labels are HATE and NOHATE plus two auxiliary classes. One of these is RELATION, used when several sentences together form hate; it was rarely applied.
- Each sentence comes with anonymised post, user and sub-forum IDs, its position in the post, and the number of context posts the annotator consulted.
- Annotators needed less and less context in later batches as they gained experience. The authors suggest active learning to cope with class imbalance in future annotation.
Cited by
- Toward Gender-Inclusive Coreference Resolution
- Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle*
- CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech
- Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
- Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection
- ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Social Bias Frames: Reasoning about Social and Power Implications of Language
Cite
@inproceedings{gibert2018hate,
title = {{Hate Speech Dataset from a White Supremacy Forum}},
author = {Gibert, Ona de and Perez, Naiara and García-Pablos, Aitor and Cuadros, Montse},
booktitle = {Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)},
year = {2018},
doi = {10.18653/v1/w18-5102},
url = {https://doi.org/10.18653/v1/w18-5102}
}References
- Conclusions and Future Work R. Agerri, J. Bermudez, and G. Rigau. 2014. IXA pipeline: Efficient and Ready to Use Multilingual NLP tools. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3823–3828. find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/gibert-2018-hate-speech-dataset-white-supremacy/