Hate Speech Dataset from a White Supremacy Forum

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-5102.pdf

Summary

Automatic hate speech detection lacks shared, openly released training data. Earlier annotation efforts often used loosely specified criteria, which produced inconsistent labels. The Vicomtech authors address this with a public English corpus drawn from Stormfront, a large white nationalist forum. They sampled posts from 22 sub-forums, filtered out non-English text, and split the posts into sentences with ixa-pipes. Annotators then labelled each sentence with a custom tool that lets them open the surrounding post when meaning is unclear. The annotators wrote the guidelines together and refined them over the first batches. A sentence counts as hate only if it is a deliberate attack, targets a group, and is motivated by that group's identity.

The paper analyses the resulting data and reports baseline classifiers, including SVM and CNN models. Its error analysis shows two common failure types: sentences that need context or world knowledge, and offensive vocabulary used without hateful intent. The dataset matters because it offers sentence-level labels plus conversation metadata, giving a reusable benchmark. It also shows how subjective hate annotation remains.

Key points

• 10,568 sentences were annotated in 10 batches, each built from 500 posts (about 1,000 sentences). Forum content dates from 2002 to 2017.
• Cleaning removed sentences under 3 or over 50 words, discarding 652 sentences (6.17%). Both the raw and the "clean" versions are released on GitHub (aitor-garcia-p/hate-speech-dataset).
• The labels are HATE and NOHATE plus two auxiliary classes. One of these is RELATION, used when several sentences together form hate; it was rarely applied.
• Each sentence comes with anonymised post, user and sub-forum IDs, its position in the post, and the number of context posts the annotator consulted.
• Annotators needed less and less context in later batches as they gained experience. The authors suggest active learning to cope with class imbalance in future annotation.