Elamur Research⌕
☰
home / papers / 2024 / Analyzing Dataset Annotation Quality Management in the Wild

Analyzing Dataset Annotation Quality Management in the Wild

Paper · 2024 · Jan-Christoph Klie, Richard Eckart de Castilho and Iryna Gurevych

Summary

Even popular NLP benchmarks contain many mislabeled items, yet no one had systematically checked how dataset authors actually handle annotation quality. The authors address this gap in two steps. First, they bring together advice scattered across books, articles and crowdsourcing studies into a structured overview of quality management methods, with practical suggestions for each. These methods include annotator selection and training, pilot rounds, iterative guideline refinement, inter-annotator agreement, adjudication, validation and error-rate estimation. Second, they build a corpus of 591 papers that introduce text datasets, sampled through Papers With Code, and annotate each paper for these quality aspects plus an overall grade. To check how representative the sample is, they compare it with papers from the ACL Anthology, with corpora distributed by the LDC, and with datasets from shared tasks. Most papers involving human annotation handle quality well or very well, but about 30% fall short. Typical problems are agreement scores left uninterpreted, samples too small for reliable agreement or error-rate figures, and weak reporting of how annotation was done. The released resource and recommendations help dataset creators avoid these pitfalls and improve reproducibility.

Key points

  • Motivation: estimated label error rates are about 6.1% for the CoNLL-2003 test split, 23.9% for TACRED and up to 30% for GoEmotions.
  • Corpus: 591 dataset-introducing publications, of which 314 involve human annotation or validation. Each was rated as excellent, sufficient or subpar, and about 30% were judged subpar.
  • Majority voting is the dominant adjudication method. Only three datasets reported probabilistic aggregation (e.g., MACE-style models).
  • Annotator training, pilot studies and iterative annotation are rarely mentioned, and agreement values often lack interpretation (e.g., via Landis & Koch or Krippendorff scales).
  • Code is available at github.com/UKPLab/qanno (GPL v3). The annotations are on TUdatalib (CC BY-NC 4.0) and can serve as a reference for finding examples of specific quality methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🔬Hessian Center for Artificial IntelligenceDarmstadt · Germany12 works · 529 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Jan-Christoph Klie, Richard Eckart de Castilho and Iryna Gurevych. 2024. Analyzing Dataset Annotation Quality Management in the Wild. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00516

@article{klie2024analyzing,
  title = {{Analyzing Dataset Annotation Quality Management in the Wild}},
  author = {Jan-Christoph Klie, Richard Eckart {de Castilho} and Iryna Gurevych},
  journal = {Computational Linguistics},
  year = {2024},
  doi = {10.1162/coli\_a\_00516},
  url = {https://doi.org/10.1162/coli_a_00516}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex, Bea, Claire Grover, Rongzhou Shen, and Mijail Kabadjov. 2010. Agile corpus annotation in practice: An overview of manual and automatic annotation of CVs. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 29–37. find
  2. Allan, Donner. 1999. Sample size requirements for interval estimation of the intraclass kappa statistic. Communications in Statistics - Simulation and Computation, 28(2):415–429. https://doi.org/10 .1080/03610919908813557 Alt, Christoph, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED Revisi… find
  3. Aroyo, Lora and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. https://doi.org/10.1609/aimag.v36i1.2564 Artstein, Ron and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):55… ↗
  4. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254.
  5. Cer, Daniel, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. https://doi.org/10…
  6. Demszky, Dorottya, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054. https://doi.org/10.18653/v1/2020 .acl… ↗
  7. Donner, Allan and Michael Eliasziw. 1992. A goodness-of-fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance-testing and sample size estimation. Statistics in Medicine, 11(11):1511–1519. https://doi.org/10 .1002/sim.4780111109, PubMed: 1410963… find
  8. Sonography, 8(4):185–190. https://doi.org/10.1002/sono.12276 Efron, B. and R. Tibshirani. 1986. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1(1):54–75. https://doi.org/10.1214/ss/1177013815 Ferracane, Elisa, Greg Dur… ↗
  9. Oliver and Boyd, Edinburgh. Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. https://doi.org/10.1037/h0031619 — [doi:10.1037/h0031619](https://doi.org/10.1037/h0031619) ↗
  10. Fleiss, Joseph L., Bruce Levin, and Myunghee Cho Paik. 2003. Statistical Methods for Rates and Proportions, 1st edition. Wiley Series in Probability and Statistics. Wiley. https://doi.org/10 .1002/0471445428 Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach,… — [doi:10.1145/3458723](https://doi.org/10.1145/3458723) find
  11. Geva, Mor, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con… find
  12. Princeton University Press, Princeton. Harris, Christopher. 2011. You’re hired! An examination of crowdsourcing incentive models in human resource tasks. In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the Fourth ACM International Conference on Web Search and Dat… find
  13. Haselbach, Boris, Kerstin Eckart, Wolfgang Seeker, Kurt Eberle, and Ulrich Heid. 2012. Approximating theoretical linguistics classification in real data: The case of German “nach” particle verbs. In Proceedings of COLING 2012, pages 1113–1128. find
  14. Hayes, Andrew F. and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77–89. https://doi.org/10.1080/19312450709336664 Ho, Chien Ju, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015. Ince… ↗
  15. Horbach, Andrea, Yuning Ding, and Torsten Zesch. 2017. The influence of spelling errors on content scoring performance. In Proceedings of the 4th Workshop on Natural Language Processing Techniques for Educational Applications (NLPTEA 2017), pages 45–53. find
  16. Hovy, Dirk, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130. https://doi.org/10.3115/v…
  17. Hovy, Eduard and Julia Lavid. 2010. Towards a ‘science’ of corpus annotation: A new methodological challenge for corpus linguistics. International Journal of Translation Studies, 22:13–36. find
  18. Hsueh, Pei Yun, Prem Melville, and Vikas Sindhwani. 2009. Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27–35. https://doi.org/10.3115/1564131.1564137 Hutchinson, Ben… ↗
  19. Kirk, Hannah, Bertie Vidgen, Paul Rottger, Tristan Thrush, and Scott Hale. 2022. HATEMOJI: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lingui… find
  20. Klie, Jan Christoph, Bonnie Webber, and Iryna Gurevych. 2023. Annotation error detection: Analyzing the past and present for a more coherent future. Computational Linguistics, 49(1):157–198. https://doi.org/10.1162/coli a 00464 Kottner, Jan, Laurent Audigé, Stig Brorson, Allan Donner, Byron J. Gaj… ↗
  21. SAGE, Los Angeles. Krippendorff, Klaus. 1995. On the reliability of unitizing continuous data. Sociological Methodology, 25:47–76. https://doi.org/10.2307/271061 Krippendorff, Klaus. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research,… ↗
  22. Lindahl, Anna, Lars Borin, and Jacobo Rouces. 2019. Towards assessing argumentation annotation - a first step. In Proceedings of the 6th Workshop on Argument Mining, pages 177–186. https://doi.org/10.18653/v1/W19-4520 Lombard, Matthew, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2002. Cont… ↗
  23. Mihaylov, Todor, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. https://doi.org/10.18653/v1/D18… ↗
  24. Manning Publications. Mostafazadeh, Nasrin, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll. 2020. GLUCOSE: Generalized and contextualized story explanations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (…
  25. Northcutt, Curtis G., Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, pages 1–13. find
  26. Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  27. Parmar, Mihir, Swaroop Mishra, Mor Geva, and Chitta Baral. 2023. Don’t blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1779–1789. https://doi.org/10 .18653/v… find
  28. Paun, Silviu, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6(0):571–585. https://doi.org/10.1162/tacl_a_00040 ↗
  29. Peters, Matthew E., Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7–14. https://doi.org/10.18653/v1/W19-4302 Piskorski, Jakub, Nicol… ↗
  30. Prasad, Rashmi, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pages 2961–2968.
  31. Pushkarna, Mahima, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data cards: Purposeful and transparent dataset documentation for responsible AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1776–1826. https://doi.org/10.1145/3531146 .3533231 Pustejovsky, J. and Amber… — [doi:10.1145/3531146](https://doi.org/10.1145/3531146) ↗
  32. O’Reilly Media, Sebastopol, California, USA. Qian, Kun, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021. Annotation inconsistency and entity bias in MultiWOZ. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dia… find
  33. Sambasivan, Nithya, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Kumar Paritosh, and Lora Mois Aroyo. 2021. “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. In SIGCHI, pages 1–21. https://doi.org/10.1145/3411764.3445518 Schreibman, Susan, Ray Sie… — [doi:10.1086/266577](https://doi.org/10.1086/266577) ↗
  34. Sheshadri, Aashish and Matthew Lease. 2013. SQUARE: A benchmark for research on computing crowd consensus. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 1156–164. https://doi.org/10.1609/ hcomp.v1i1.13088 Shmueli, Boaz, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyo… find
  35. Stab, Christian and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46–56. https://doi.org/10.3115/v1/D14 -1006 Stoica, George, Emmanouil Antonios Plata… ↗
  36. Wang, Zihan, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. CrossWeigh: Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La… find
  37. Wei, Jason, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, pages 1–46. Wynne, Martin, editor. 2005. Developing Linguistic Corp…
  38. David Brown Book Company, Oakville, Connecticut. Yao, Yuan, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/klie-2024-analyzing-dataset-annotation-quality-management/