📄 SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter

Summary

HatEval (SemEval-2019 Task 5) created a shared benchmark for detecting hate speech on Twitter aimed at two frequent targets, immigrants and women, in both English and Spanish. The organizers collected tweets in three ways: tracking likely victims, downloading the timelines of known haters, and filtering streams with neutral terms, slurs and polarized hashtags. The resulting 19,600 tweets were labeled with a hierarchical scheme. The first label marks whether hate speech is present. If it is, two further labels record whether the target is an individual or a group and whether the author is aggressive. Crowd workers on Figure Eight provided labels, which were combined with two expert annotations by majority vote. Participants could enter two subtasks: binary detection (Subtask A) and joint fine-grained labeling (Subtask B). The task attracted 74 teams, who submitted 108 runs for A and 70 for B. Subtask B proved clearly harder than Subtask A. An analysis of errors shared by the top systems showed that they mostly flag non-hateful tweets as hateful. The released data give the field a reusable multilingual resource for abusive language research.

Key points

• Corpus size: 13,000 English and 6,600 Spanish tweets. By target, 9,091 concern immigrants and 10,509 concern women.
• Test sets: 3,000 English tweets (1,260 hateful) and 1,600 Spanish tweets (660 hateful). Hateful tweets were deliberately over-represented compared with their natural frequency on Twitter.
• Annotation: each tweet received at least three crowd judgments on Figure Eight plus two expert labels, and the final label was decided by majority vote. Average crowd confidence for HS, TR and AG was 0.83, 0.70 and 0.73 in English and 0.89, 0.47 and 0.47 in Spanish.
• Evaluation: Subtask B was scored with EMR (exact match ratio) and macro-averaged F1, against SVC and MFC baselines. The best Spanish Subtask B EMR was 0.705 (CIC-2), compared with 0.570 for the best English system. Median scores for Subtask B were lower than for Subtask A in both languages.
• Error analysis: the three top systems made the same mistake on 569 of 2,971 English test tweets (19.1%) and on 234 of 1,600 Spanish tweets (14.6%). False positives made up 89.1% of these shared errors in English and 76.1% in Spanish.

Interesting ideas

• I1. Hierarchical labels: the target and aggressiveness labels exist only when a tweet is hateful. This keeps the scheme simple and lets one dataset support both a coarse task and a fine-grained task.
• I2. Crowd plus experts: two expert labels were added on top of the crowd votes before taking the majority. This is a cheap way to keep a benchmark consistent with the guidelines without discarding the crowd's input, and it suits any subjective labeling task.
• I3. Mixed collection strategy: combining victim monitoring, hater timelines and keyword filtering with neutral, derogatory and polarized terms yields data that is not dominated by obvious slurs. This helps models learn the line between hate, offensiveness and stance.
• I4. Controlled balancing: only the main class was rebalanced, while secondary attributes kept their natural distribution. This is a reasonable compromise for multi-level tasks with a rare positive class.
• I5. Shared-error analysis: examining the errors that all top systems make reveals systematic weaknesses rather than model-specific noise. Here it exposed playful swearing and convoluted negation as recurring problems, and the same check can be run cheaply on any shared task or model ensemble.