Elamur Research⌕
☰
home / papers / 2021 / Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement

Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement

Summary

Offensive language detection depends on supervised training, but labelling it is subjective, and annotators often disagree. Datasets usually collapse that disagreement into one label or discard it as noise. This paper treats disagreement as information. The authors gathered English tweets from 2020 on three topics: Covid-19, the US Presidential elections, and Black Lives Matter. They used an ensemble of five BERT classifiers to estimate in advance how contested each tweet would be. Using that estimate, they sampled more than 10k tweets so that the three agreement levels were evenly represented. Each tweet then received five crowd-sourced judgments. Experiments in-domain and across domains show that classifier performance and robustness change strongly depending on the agreement level of the training and test data. Low-agreement tweets still carry useful training signal and are not random labels. Applied to the Offenseval 2020 English test set, the analysis suggests that high system scores rest largely on easy, unanimous items. The work argues that future benchmarks, especially test sets, should contain more ambiguous cases.

Key points

  • The resource contains over 10k tweets with about 50k Amazon Mechanical Turk judgments, five per tweet. Tweets were collected January–November 2020 from a pool of 400,000 (about 130,000 per domain). Each tweet is labelled with an agreement level: A++ (5/5), A+ (4/5), or A0 (3/2).
  • For pre-selection, five BERT-Base uncased models were fine-tuned on a binarised Founta et al. (2018) dataset, with Abusive and Hateful mapped to offensive, Normal to non-offensive, and Spam dropped. Each model used a different training/validation split or a different oversampling setup.
  • The ensemble's own verdicts on the pool were highly skewed: about 92% A++, 5% A+, and 3% A0. Random sampling would therefore have under-represented ambiguous tweets.
  • On Offenseval 2020, the authors averaged 81 of 85 runs with F1 > 0.75. Mean F1 was 0.915 on A++ tweets (887), 0.817 on A+ (173), and 0.656 on A0 (113).
  • Preliminary attempts to predict ambiguity directly with BERT, either as a binary or a three-way task, worked poorly. The data is released as tweet IDs on request for non-commercial research, without worker IDs.

Interesting ideas

  • I1. Cheap pre-screening with a model ensemble. Disagreement among a few slightly different models can stand in for human disagreement and steer the annotation budget toward rare ambiguous items. This is useful whenever the target phenomenon is skewed and labelling is expensive.
  • I2. Low-cost ensemble diversity. The five models share one architecture and one source dataset. They differ only in data proportions, duplication of instances, or the train/validation ratio. This yields varied votes without needing different model families.
  • I3. Matching ensemble size to annotator count. Five models and five crowd-workers allow a direct, like-for-like comparison of agreement patterns. This design could help validate any proxy for annotation difficulty.
  • I4. Reporting scores by agreement tier. Breaking evaluation down by how strongly humans agreed shows whether a benchmark's headline score depends on easy items. This applies to any subjective task, such as sentiment, stance, or toxicity in Ukrainian data.
  • I5. Keeping disaggregated labels and fair crowd practices. The authors retained all five judgments rather than forcing a single gold label. They also paid workers even for rejected submissions and released only IDs to respect deletion rights. These are reusable norms for sensitive social-media corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.822

@inproceedings{leonardelli2021agreeing,
  title = {{Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement}},
  author = {Leonardelli, Elisa and Menini, Stefano and Aprosio, Alessio Palmero and Guerini, Marco and Tonelli, Sara},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.822},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.822}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opinions to improve hate speech detection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1):151–154. find
  2. Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 1100–1105, New York, NY, USA. Association for… find
  3. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find our card
  4. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555– 596. find our card
  5. Valerio Basile. 2020. It’s the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In Proceedings of the AIxIA 2020 Discussion Papers Workshop co-located with the the 19th International Conference of the Italian Association for Artif… find
  6. Beata Beigman Klebanov and Eyal Beigman. 2009. From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. find
  7. Beata Beigman Klebanov and Eyal Beigman. 2014. Difficult cases: From data to learning, and back. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 390–396, Baltimore, Maryland. Association for Computational Linguistics. find
  8. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17. find our card
  9. Barbara Di Eugenio and Michael Glass. 2004. Squibs and discussions: The kappa statistic: A second look. Computational Linguistics, 30(1):95–101. find
  10. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 21… find
  11. Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In 11th International Conference on W… find
  12. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2010. Part-of-speech tagging for twitter: Annotation, features, and experiments. Technical report, Carnegie-Mellon University, School of… find our card
  13. Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York,… find
  14. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Associati… find our card
  15. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. find our card
  16. Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009. Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27–35, Boulder, Colorado. Association for Computational Linguist… find
  17. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297, Lisbon, Portugal. Association for Computatio… find
  18. Manfred Klenner, Anne Göhring, and Michael Amsler. 2020. Harmonization sometimes harms. In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing, SwissText/KONVENS 2020, Zurich, Switzerland, June 23-25, 2020 [online only], volume 2624 of CEUR W… find
  19. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). find
  20. J. Peterson, R. Battleday, T. Griffiths, and O. Russakovsky. 2019. Human uncertainty makes classification more robust. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616–9625. find
  21. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  22. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden. Association for Computational Li… find our card
  23. Ines Rehbein and Josef Ruppenhofer. 2011. Evaluating the impact of coder errors on active learning. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 43–51, Portland, Oregon, USA. Association for Computational Linguistics. find
  24. Dennis Reidsma and Rieks op den Akker. 2008. Exploiting ‘subjective’ annotations. In Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 8–16, Manchester, UK. find
  25. Coling 2008 Organizing Committee. find
  26. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artifici… find
  27. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. find our card
  28. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, p… find our card
  29. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, page 614–622, New York, NY, US… find
  30. Edwin Simpson, Jonas Pfeiffer, and Iryna Gurevych. 2020. Low resource sequence tagging with weak labels. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8862–8869. find
  31. Fei-Yue Wang. 2007. Toward a paradigm shift in social computing: The acp approach. IEEE Intelligent Systems, 22(5):65–67. find
  32. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of NAACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leonardelli-2021-agreeing-disagree-annotating-offensive-language/