📄 On Releasing Annotator-Level Labels and Information in Datasets

Summary

NLP datasets with multiple annotators per item usually collapse those judgements into one label, by majority vote, averaging, or expert adjudication. For subjective tasks, this can hide systematic disagreement rooted in annotators' backgrounds and experiences. The authors test whether majority voting represents annotators fairly. They examine eight binary tasks from three datasets: hate speech, sentiment, and six Ekman emotions. For each annotator, they compute Cohen's Kappa between that person's labels and the majority label. They then compare these scores across socio-demographic groups in the sentiment data, which includes self-reported annotator demographics. In most tasks, a large share of annotators agree only weakly with the majority. In the sentiment data, White annotators match the aggregated labels significantly better than Black annotators. Aggregation can therefore make a dataset under-represent some groups' views. The authors recommend releasing individual annotator labels and, where it can be done responsibly, annotator demographics. They also point to modelling work that already benefits from annotator-level labels.

Key points

• Data analysed:
  • Hate-speech (Kennedy et al., 2020): 27,665 instances, 18 annotators, 86,529 annotations.
  • Sentiment (Díaz et al., 2018): 14,071 instances, 1,481 annotators, 59,240 annotations. Its 5-point scale was binarised.
  • Emotion (Demszky et al., 2020): 58,011 instances, 82 annotators, 211,225 annotations. Only the raw labels for anger, disgust, fear, joy, sadness and surprise were used.
• The majority label was the one chosen by at least half of an item's annotators. Each annotator was scored with Cohen's Kappa against it.
• Low agreement was common:
  • For disgust, 27 of 82 annotators had κ < 0.3 with the majority.
  • For sentiment, roughly a third of annotators (450+) agreed poorly.
  • Joy was the main task where agreement was fairly uniform across annotators.
• Group differences:
  • One-way ANOVA found a significant race effect (F(2, 2387)=3.77, p=0.02).
  • White annotators averaged κ 0.42, versus 0.37 for Black annotators (Tukey p=0.03).
  • Gender and political-affiliation differences were not significant.
• The paper describes four levels of annotation release: aggregated label only, label distribution, anonymised per-annotator labels, and per-annotator labels with demographics. It argues for the richer levels.