Elamur Research
☰
home / papers / 2021 / We Need to Consider Disagreement in Evaluation

We Need to Consider Disagreement in Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.bppf-1.3.pdf

Summary

This position paper questions the usual NLP practice of scoring predictions against a single aggregated "gold standard." The authors argue that this assumption breaks down because many items allow several valid labels and because annotators often disagree. This happens even on tasks seen as objective, such as POS tagging, coreference, word sense disambiguation and semantic role labelling. They trace disagreement to three origins: differences between annotators, properties of the item itself (ambiguity, humour, multi-label content) and the situation in which annotation happens, such as attention lapses. They reject proposals to clean datasets by dropping contested items, because this discards information and hides what the task really involves. Rather than offering new data, the paper reviews existing remedies. These include soft evaluation, which compares a model's predicted label distribution with the human one (for example, via cross-entropy), per-annotator evaluation for subjective tasks (Basile, 2020) and the SemEval-2021 Task 12 (LeWiDi) benchmark. The main message is that disagreement-aware evaluation is already possible and should become routine, as it long has been for generation metrics like BLEU and ROUGE.

Key points

  • Three sources of disagreement: individual differences between annotators, stimulus characteristics (lexical or syntactic ambiguity, humour, poetry, political discourse, multi-label items) and context (the same coder may answer differently at different times).
  • Soft evaluation: model output distributions are compared with full human label distributions rather than one-hot labels. Peterson et al. (2019) applied this to image classification, and Uma et al. (2020) applied it to NLP while also testing different ways of deriving soft labels.
  • SemEval-2021 Task 12 (LeWiDi): the task used five multiply-annotated datasets, including Gimpel et al. (2011) Twitter POS tags, the humour data of Simpson et al. (2019), LabelMe and CIFAR-10 (Peterson et al., 2019). Systems were scored with both F1 and cross-entropy. Models that account for noise and disagreement almost always achieved the lowest cross-entropy.
  • Polarization index (Akhtar et al., 2019): this measure compares within-group and between-group agreement for each instance. Akhtar et al. (2020) used it to build alternative gold standards for hate speech and to train models representing different perspectives.
  • Per-annotator evaluation (Basile, 2020): models predict each annotator's label individually. The framework was tested on synthetic data controlled by two parameters, difficulty and subjectivity, and is argued to make evaluation of highly subjective tasks fairer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation · Part-of-speech tagging

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 109 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.bppf-1.3

@inproceedings{basile2021we,
  title = {{We Need to Consider Disagreement in Evaluation}},
  author = {Basile, Valerio and Fell, Michael and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo and Uma, Alexandra},
  booktitle = {Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future},
  year = {2021},
  doi = {10.18653/v1/2021.bppf-1.3},
  url = {https://doi.org/10.18653/v1/2021.bppf-1.3}
}

References

49 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The SEMEVAL 2021 Campaign The objective of SEMEVAL -2021 Task 12 on Learning with Disagreements (LeWiDi) (Uma et al., 2021) was to provide a unified testing framework for learning from disagreements in NLP and CV using datasets containing information about disagreements for interpreting language and… find
  2. Evaluation in Light of Disagreement While the research mentioned in the previous section questions the assumption that a single ‘hard’ label (a gold label) exists for every item in a dataset, the models proposed for learning from multiple interpretations are still largely evaluated under this assump… find
  3. Peterson et al. (2019) tested this approach on image classification tasks, generating the soft label by transforming the item annotation distribution using standard normalization. find
  4. Uma et al. (2020) employed this form of soft metric evaluation for NLP , also comparing different ways to obtain a soft label from the raw data. They use soft metrics to compare the classifiers’ distribution to the human-derived label distributions, complementing traditional hard evaluation measures… find
  5. Basile (2020) suggested a more extreme evaluation framework, where a model is required to produce different outputs encoding the individual annotators’ labels. The predictions are then individually evaluated against the single annotations, rather than against an aggregated gold standard. This propos… find
  6. Evaluation of Highly Subjective Tasks Basile (2020) explored the impact of disagreement caused by polarization on evaluation, focusing on NLP tasks with high levels of subjectivity. They argue that aggregated test sets lead to unfair evaluation concerning the multiple perspectives stemming from the… find
  7. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2019. A new measure of polarization in the annotation of hate speech. In AI*IA 2019 – Advances in Artificial Intelligence, pages 588–603, Cham. Springer International Publishing. find
  8. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opinions to improve hate speech detection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1):151–154. find
  9. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3318–3329, Santa Fe, New Mexico, USA. Association for Computational Linguistics. \#\# Conclusion In this pos… find
  10. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find our card
  11. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555– 596. find
  12. Valerio Basile. 2020. It’s the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In Proc. of the AIXIA Workshop. Universitá di Torino. find
  13. Beata Beigman Klebanov and Eyal Beigman. 2009. Squibs: From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. find
  14. Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2008. Analyzing disagreements. In Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 2– 7, Manchester, UK. find
  15. Coling 2008 Organizing Committee. find
  16. Samuel R Bowman and George E Dahl. 2021. What will it take to fix benchmarking in natural language understanding? arXiv preprint arXiv:2104.02145. \#\# Acknowledgements Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2009. SemEval-2010 task 9: The in… find
  17. This research was supported in part by the DALI project, ERC Grant 695662, and the European Union’s Horizon 2020 research and innovation program (grant agreement No. 949944, INTEGRA-TOR) and by the Independent Research Fund Denmark (DFF) grants No. 9131-00019B and 9063-00077B. TF and DH are members… find
  18. Kenneth Ward Church and Joel Hestness. 2019. A survey of 25 years of evaluation. Natural Language Engineering, 25(6):753–767. find
  19. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Assoc… find
  20. Tommaso Fornaciari, Silviu Uma, Alexandra Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin… find
  21. Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: It’s complicated! In Proceedings of the 2018 Con… find
  22. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Assoc… find
  23. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  24. You-Min Lin and Michelle Y. Chen. 2020. Understanding writing quality change: A longitudinal study of repeaters of a high-stakes standardized english proficiency test. Language Testing, 37(4):523–549. find
  25. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  27. Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013. SemEval-2013 task 4: Free paraphrases of noun compounds. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop… find
  28. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):219–252. find
  29. Rebecca J. Passonneau and Bob Carpenter. 2013. The benefits of a model of annotation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 187–195, Sofia, Bulgaria. Association for Computational Linguistics. find
  30. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Associati… find our card
  31. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585. find
  32. Nancy Ide and James Pustejovsky, editors. 2017. The Handbook of Linguistic Annotation. Springer. find
  33. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. find our card
  34. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297, Lisbon, Portugal. Association for Computatio… find
  35. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616–9625. generated text. In Proceedings of the 12th International Conference on Na… find
  36. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. find
  37. Susanne Winkler, editor. 2015. Ambiguity: Language and Communication. De Gruyter. find
  38. Massimo Poesio. 2020. Ambiguity. In Daniel Gutzmann, Lisa Matthewson, and Cécile Meier and Hotze Rullmann and Thomas Ede Zimmermann, editors, The Companion to Semantics. Wiley. find
  39. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83. find
  40. Victor Raskin. 1985. Semantic Mechanisms of Humor. D. Reidel, Dordrecht and Boston. find
  41. Marta Recasens, Ed Hovy, and M. Antonia Martí. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
  42. Philip Resnik and Jimmy Lin. 2010. 11 evaluation of nlp systems. The handbook of computational linguistics and natural language processing, 57. find
  43. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In AAAI Conference on Artificial Intelligence. find
  44. Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2008. LabelMe: A database and Web-based tool for image annotation. International Journal of Computer Vision, 77:157– 173. find
  45. Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5716–5728, Florence, Italy. Association… find
  46. Soon P. Su. 1994. Lexical Ambiguity in Poetry. Longman, London. find
  47. Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, and Massimo Poesio. 2021. Semeval-2021 task 12: Learning with disagreements. In Proceedings of the Fifteenth Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  48. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft-loss functions. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing, pages 173–177. find
  49. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically find