Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.tacl-1.6.pdf

Summary

In subjective tasks such as hate speech or emotion detection, labels from several annotators are usually merged into one gold label through majority vote or averaging. The authors argue that this hides systematic, value-driven differences between annotators and sidelines minority viewpoints. It can even give similar texts inconsistent labels. Instead of aggregating before training, they propose multi-annotator models. These are multi-task networks in which each annotator's judgements get their own prediction head, built on a shared representation tuned on the whole dataset. They test this on seven binary classification tasks: hate speech detection on GHC and six Ekman emotions from GoEmotions. The approach equals or beats models trained on majority-vote labels, and it captures individual annotators better. The spread of predictions across heads also gives an uncertainty estimate that tracks real annotator disagreement more closely than conventional estimates. This matters in practice because a deployed system can abstain or send uncertain items to human review. A single model could also be adapted to different value systems or cultures without retraining.

Key points

• Architecture: one classifier head per annotator on a shared multi-task backbone. Each head learns from the subset of instances that annotator labeled, while the shared layers use all the data.
• Evaluation: seven binary tasks. One is hate speech on GHC, which has 18 annotators. The other six are anger, disgust, fear, joy, sadness and surprise from GoEmotions, which has 82 annotators.
• Uncertainty: the uncertainty derived from the multiple heads correlates with annotation disagreement more strongly than traditional methods on all seven tasks. It also signals certain types of prediction error.
• Scaling limits: datasets with very large annotator pools, such as over 1.4K in Díaz (2020) or over 8K in Jigsaw (2019), make one head per annotator expensive. The authors suggest clustering annotators and using one head per cluster. Sparse annotators, such as the many GoEmotions annotators with fewer than 1000 labels, are another obstacle.
• Recommendation: data creators should release anonymized annotator-level labels with consent. They should also collect enough annotations per annotator to make this kind of modeling possible.