Elamur Research⌕
☰
home / papers / 2020 / Identifying and Measuring Annotator Bias Based on Annotators' Demographic Characteristics

Identifying and Measuring Annotator Bias Based on Annotators' Demographic Characteristics

Paper · 2020 · Hala Al Kuwatly, Maximilian Wich and Georg Groh

Summary

The paper asks whether the demographic profile of crowd annotators (gender, first language, age, education) leaves a measurable imprint on abusive-language labels and on the models trained from them. The authors use the Wikipedia Detox personal attack corpus and its "attack" label. For each demographic feature, they build parallel datasets over the same comments. In each dataset, labels are aggregated only from annotators of one group. A mixed control set draws half its annotators from each group.

For every group, they fine-tune 20 DistilBERT classifiers, each time resampling which annotators supply the labels. They then compare the resulting F1 distributions with a Kolmogorov-Smirnov test. Gender shows no significant effect. First language, age and education all produce statistically significant performance gaps.

The work matters because it offers a simple, model-based way to detect annotator bias. It also suggests that a single pooled ground truth may not treat all annotator groups fairly, even when the pool is balanced across groups.

Key points

  • Data: the Detox corpus has 115,864 Wikipedia comments labeled by 4,053 crowdworkers. Demographic data exist for 2,190 of these workers.
  • Binarized features: age was split into under 30 vs. over 30. Education was split into high school or below vs. above high school. Both regroupings were made because the original categories were unevenly populated.
  • Comment selection and labeling: a comment was used only if at least 6 annotators from each group labeled it. Group labels come from 6 random in-group annotators, and mixed labels from 3 + 3.
  • Dataset sizes: gender has 5,501 comments, first language 2,547, age 8,478, and education 3,968.
  • Model setup: distilbert-base-uncased, max sequence length 100, learning rate 5×10⁻⁶, 1cycle policy, 2 epochs.
  • KS p-values (F1 on the mixed test set):
    • gender: 0.83
    • first language: 1.0×10⁻³
    • age: 1.1×10⁻⁸
    • education: 1.4×10⁻⁷
  • Direction of effects:
    • Models trained on labels from native speakers, over-30 annotators, and annotators with high school or less scored higher F1.
    • Krippendorff's alpha was lower among female annotators (0.45) than male annotators (0.51), yet this did not translate into a model performance gap.

Interesting ideas

  • I1. Use classifiers as probes for annotator bias. Instead of only comparing raw label agreement, train separate models on each group's labels and compare their behavior. This captures whether a labeling difference is large enough to change what a model learns. The probe could be reused for any subjective annotation task, such as sentiment or toxicity in Ukrainian data.
  • I2. Hold the comments fixed and vary only the annotators. Every group dataset uses the identical set of comments, so any performance difference comes from the labels rather than the text. This kind of controlled design is a clean template for isolating one source of variation.
  • I3. Resample annotators to get a distribution, not a single score. Each of the 20 runs draws different annotators, which yields a spread of F1 values. A nonparametric test such as Kolmogorov-Smirnov can then be applied to that spread. This is a cheap way to add significance testing to comparisons that would otherwise rest on one number.
  • I4. Include a mixed-annotator control set. Building a balanced mixed-label dataset alongside the per-group ones provides a baseline. It shows which group's labels the pooled ground truth most resembles, and that check is useful whenever labels are aggregated from heterogeneous annotators.
  • I5. Look at sensitivity and specificity, not just F1. Plotting both metrics for each run revealed that some groups mainly differ in how readily they mark content as an attack. A single F1 score would have hidden this asymmetry.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany18 works · 48,618 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Safety Research · 221 works

Cite

Cite as Hala Al Kuwatly, Maximilian Wich and Georg Groh. 2020. Identifying and Measuring Annotator Bias Based on Annotators' Demographic Characteristics. Proceedings of the Fourth Workshop on Online Abuse and Harms.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.alw-1.21

@inproceedings{kuwatly2020identifying,
  title = {{Identifying and Measuring Annotator Bias Based on Annotators' Demographic Characteristics}},
  author = {Hala {Al Kuwatly}, Maximilian Wich and Georg Groh},
  booktitle = {Proceedings of the Fourth Workshop on Online Abuse and Harms},
  year = {2020},
  doi = {10.18653/v1/2020.alw-1.21},
  url = {https://doi.org/10.18653/v1/2020.alw-1.21}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agathe Balayn, Panagiotis Mavridis, Alessandro Bozzon, Benjamin Timmermans, and Zoltán Szlávik. 2018. Characterising and mitigating aggregation-bias in crowdsourced toxicity annotations. In Proceedings of the 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Pape… find
  2. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  3. Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. 2017. Like trainer, like bot? inheritance of bias in algorithmic content moderation. In International conference on social informatics, pages 405–415. Springer. find
  4. Dimitrios Bountouridis, Mykola Makhortykh, Emily Sullivan, Jaron Harambam, Nava Tintarev, and Claudia Hauff. 2019. Annotating credibility: Identifying and mitigating bias in credibility datasets. find
  5. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.12516. our card
  6. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009. our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  8. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  9. Maeve Duggan. 2017. Online Harassment 2017. Pew Research Center. find
  10. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. arXiv preprint arXiv:1802.00393. — [arXiv:1802.00393](https://arxiv.org/abs/1802.00393) find
  11. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898. find
  12. Michael Wojatzki Tobias Horsmann Darina Gold and Torsten Zesch. 2018. Do women perceive hate differently: Examining the relationship between hate speech, gender, and agreement judgments. find
  13. Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77–89. find
  14. Jae Yeon Kim, Carlos Ortiz, Sarah Nam, Sarah Santiago, and Vivek Datta. 2020. Intersectional bias in hate speech and abusive language datasets. arXiv preprint arXiv:2005.05921. — [arXiv:2005.05921](https://arxiv.org/abs/2005.05921) find
  15. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv preprint arXiv:1701.08118. find
  16. Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-Gvo Jung, and Bernard J Jansen. 2018. Online hate interpretation varies by country, but more by individual: A statistical analysis using crowdsourced ratings. In 2018 Fifth International Conference on Social Networks Analysis, Management and Securi… find
  17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  18. Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data: Garbage in, garbage out. arXiv preprint arXiv:2004.01670. find
  19. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019a. Challenges and frontiers in abusive content detection. Association for Computational Linguistics. our card
  20. Bertie Vidgen, Helen Margetts, and Alex Harris. 2019b. How much online abuse is there? a systematic review of evidence for the uk. The Alan Turing Institute. find
  21. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proc. 1st Workshop on NLP and Computational Social Science, pages 138–142. our card
  22. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899. our card
  23. Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020a. Investigating annotator bias with a graph-based approach. In Proc. 4th Workshop on Online Abuse and Harms. find
  24. Maximilian Wich, Jan Bauer, and Georg Groh. 2020b. Impact of politically biased data on hate speech classification. In Proc. 4th Workshop on Online Abuse and Harms. find
  25. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: the problem of biased datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short… our card
  26. Wikimedia. n.d. Research:detox/data release. https://meta.wikimedia.org/wiki/Research: Detox/Data_Release . ↗
  27. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678. our card
  28. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1–10. our card
  29. Leslie N Smith. 2018. A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820. — [arXiv:1803.09820](https://arxiv.org/abs/1803.09820) find
  30. Bertie Vidgen, Austin Botelho, David Broniatowski, Ella Guest, Matthew Hall, Helen Margetts, Rebekah Tromble, Zeerak Waseem, and Scott Hale. 2020. Detecting east asian prejudice on social media. find
  31. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, pages 1391–1399. find
  32. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. arXiv preprint arXiv:2005.12246. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuwatly-2020-identifying-measuring-annotator-bias-based/