Elamur Research⌕
☰
home / papers / 2023 / SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)

SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)

Paper · 2023 · Elisa Leonardelli, Gavin Abercrombie, Dina Almanea et al.

Summary

Annotators of subjective NLP tasks such as hate speech or offensiveness detection often disagree, and merging their labels into one "gold" answer misrepresents the data. This paper reports the second LeWiDi shared task, which offered a common benchmark for training and testing models on data that keeps every annotator's judgment. Compared with the first edition, it covered only text, only subjective tasks, and included a non-English language (Arabic). Four datasets were used: HS-Brexit, ArMIS, ConvAbuse and MD-Agreement. They span tweets and human–chatbot dialogues, and their annotators range from experts to targeted demographic groups to crowdworkers. Systems were ranked mainly by a soft metric, cross-entropy against the annotator label distribution, with F1 reported as a secondary hard metric. Participation rose sharply over the first edition. The organizers' analysis shows that hard predictions break down on low-agreement items, and that teams which exploited disagreement information did comparatively better under cross-entropy. The task supports treating disagreement as signal rather than noise, while also exposing weaknesses of current soft metrics.

Key points

  • Participation: over 130 registered groups, 29 submitted runs, and 13 system description papers, far more than the first edition.
  • The four datasets:
    • HS-Brexit: 1,120 English tweets, 6 annotators divided between a target group made up of UK-based Muslim immigrants and a separate control group.
    • ArMIS: Arabic sexism tweets labelled by 3 annotators with different self-declared political and gender profiles.
    • ConvAbuse: dialogues with conversational agents, annotated by gender-studies experts.
    • MD-Agreement: 10,753 tweets, 5 AMT annotators each.
  • Share of items with full agreement differs widely: 86% (ConvAbuse), 69% (HS-Brexit), 65% (ArMIS), 42% (MD-Agreement).
  • In HS-Brexit, every case of total disagreement between the groups went one way: the target group saw hate and the control group did not.
  • On items with roughly 40–60% agreement, systems' hard labels were wrong 46% of the time. Cross-entropy and F1 correlated only weakly on low-agreement data.

Interesting ideas

  • I1. Collect annotations from contrasting groups on purpose. Recruiting annotators by group (target vs. control, conservative vs. liberal) makes systematic, one-directional disagreement visible. This could reveal perspective bias in any subjective labelling project.
  • I2. Select items that are likely to be contested. MD-Agreement chose tweets expected to provoke disagreement, which yields a benchmark rich in hard cases. This is a cheap way to stress-test models.
  • I3. Report performance by agreement level. Splitting the test set into high-, mid- and low-agreement subsets shows where hard-label models fail. It is a simple diagnostic that suits any multi-annotator dataset.
  • I4. Check the dataset's own entropy before comparing scores. Cross-entropy cannot fall below the entropy of the gold distribution, so raw scores are not comparable across datasets. Any benchmark mixing corpora with different ambiguity levels should account for this.
  • I5. Run one task across several heterogeneous datasets. This pushes participants toward general solutions instead of dataset-specific tricks, and the same design could benefit other benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany44 works · 55,193 citations🎓University of TurinTurin · Italy27 works · 6,505 citations🔬Fondazione Bruno KesslerTrento · Italy25 works · 43,498 citations🎓Queen Mary University of LondonLondon · United Kingdom39 works · 3,307 citations🎓Heriot-Watt UniversityEdinburgh · United Kingdom10 works · 5,081 citations🎓Bocconi UniversityMilan · Italy16 works · 2,850 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma and Massimo Poesio. 2023. SemEval-2023 Task 11: Learning with Disagreements (LeWiDi). Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.semeval-1.314

@inproceedings{leonardelli2023semeval,
  title = {{SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)}},
  author = {Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma and Massimo Poesio},
  booktitle = {Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)},
  year = {2023},
  doi = {10.18653/v1/2023.semeval-1.314},
  url = {https://doi.org/10.18653/v1/2023.semeval-1.314}
}

References

51 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, and Alexandra Uma, editors. 2022. Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022. European Language Resources Association, Marseille, France. find
  2. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? Perspective-aware models to identify opinions of hate speech victims in abusive language detection. CoRR, abs/2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) find
  3. Dina Almanea and Massimo Poesio. 2022. ArMIS -the Arabic misogyny and sexism corpus with annotator subjective disagreements. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2282–2291, Marseille, France. European Language Resources Association.
  4. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on twitter. In Natural Language Processing and Information Systems, pages 57–64, Cham. Springer International Publishing. find
  5. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24.
  6. Valerio Basile. 2020. It’s the end of the gold standard as we know it. On the impact of pre-aggregation on the evaluation of highly subjective tasks. In Proc. of the AIXIA Workshop. Universitá di Torino. find
  7. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computa…
  8. Beata Beigman Klebanov and Eyal Beigman. 2009. From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. find
  9. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence.
  10. Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7388–7403, Online and Punta Cana, Dominican Re… find
  11. Xia Cui. 2023. xiacui at semeval-2023 task 11: Learning a model in mixed-annotator datasets using annotator ranking scores as training weights. In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 1076–1084, Toronto, Canada. Association for Computational Linguistics. find
  12. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the ACL, 10:92–110.
  13. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, volume 1, pages 2164– 2170. Association for Computational Linguistics. find
  14. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin…
  15. Tommaso Fornaciari, Alexandra Uma, Massimo Poesio, and Dirk Hovy. 2022. Hard and soft evaluation of NLP models with BOOtSTrap SAmpling - BooStSa. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 127–134, Dublin, Ireland. Associa… find
  16. Ewelina Gajewska. 2023. eevvgg at semeval-2023 task 11: Offensive language classification with rater-based information. In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 171–176, Toronto, Canada. Association for Computational Linguistics. find
  17. José Antonio García-Díaz, Ronghao Pan, Gema Alcaráz-Mármol, María José Marín-Pérez, and Rafael Valencia-García. 2023. Umuteam at semeval-2023 task 11: Ensemble learning applied to binary supervised classifiers with disagreements. In Proceedings of the 17th International Workshop on Semantic Evaluati… find
  18. Rob van der Goot. 2023. MaChAmp at SemEval-2023 Task 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12: On the effectiveness of intermediate training on an uncurated collection of datasets. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). Association for Computational Li… find
  19. Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021a. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–14. ACM. find
  20. Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021b. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York,… find
  21. Dennis Grötzinger, Simon Heuschkel, and Matthias Drews. 2023. Cicl_dms at semeval-2023 task 11: Learning with disagreements (le-wi-di). In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 1030–1036, Toronto, Canada. Association for Computational Linguistics. find
  22. Peyman Hosseini, Mehran Hosseini, Sana Sabah Al-Azzawi, Marcus Liwicki, Ignacio Castro, and Matthew Purver. 2023. Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction. In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto… find
  23. Kian Kenyon-Dean, Eisha Ahmed, Scott Fujimoto, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: It’s complicated! In Proceedings of the 2018 Con… find
  24. Guneet Singh Kohli and Vinayak Tiwari. 2023. Arguably at semeval-2023 task 11: Learning the disagreements using unsupervised behavioral clustering and language models. In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 2137–2142, Toronto, Canada. Association for Computat… find
  25. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539,…
  26. Ankita Maity, Pavan Kandru, Bhavyajeet Singh, Kancharla Aditya Hari, and Vasudeva Varma. 2023. Irel at semeval-2023 task 11: User conditioned modelling for toxicity detection in subjective tasks. In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 2133–2136, Toronto, Cana… find
  27. Héctor Martínez Alonso, Anders Johannsen, and Barbara Plank. 2016. Supersense tagging with inter-annotator disagreement. In Proceedings of the 10th Linguistic Annotation Workshop, pages 43–48. Association for Computational Linguistics. find
  28. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):219–252. find
  29. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694.
  30. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, pages 9616–9625. find
  31. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  32. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014a. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751. Association for Computational Linguistics.
  33. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014b. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics.
  34. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83. Association for Computational Linguistics.
  35. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis…
  36. Marta Recasens, Ed Hovy, and M. Antònia Martí. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138– 1152. find
  37. Giulia Rizzi, Alessandro Astorino, Daniel Scalena, Paolo Rosso, and Elisabetta Fersini. 2023. MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement. In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association for Computational… find
  38. Filipe Rodrigues and Francisco C. Pereira. 2018. Deep learning from crowds. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 1611–1618. find
  39. Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. 2023. Why don’t you do it right? analysing annotators’ disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2420–2433, Dubrovnik, Cr… find
  40. Sadat Shahriar and Thamar Solorio. 2023. SafeWebUH at SemEval-2023 Task 11: Learning Annotator Disagreement in Derogatory Text: Comparison of Direct Training vs Aggregation. In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association for Computational Lingu… find
  41. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 614–622. find
  42. Alexandra Uma, Dina Almanea, and Massimo Poesio. 2022. Scaling and disagreements: Bias, noise and ambiguity. Frontiers in Artificial Intelligence: Human-Centered AI. find
  43. Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021a. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338–347, Onl… find
  44. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft-loss functions. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing, pages 173–177. find
  45. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021b. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  46. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6(3):333–353. find
  47. Nikolas Vitsakis, Amit Parekh, Tanvi Dinkar, Gavin Abercrombie, Ioannis Konstas, and Verena Rieser. 2023. iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives? In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association fo… find
  48. Ruyuan Wan and Karla Badillo-Urquiola. 2023. Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty. In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association for Computational Li… find
  49. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. find
  50. Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5716–5728. Association for Computationa… find
  51. Michael J. Sullivan, Mohammed Nasheed Yasin, and Cassandra L. Jacobs. 2023. University at buffalo at semeval-2023 task 11: Masda–modelling annotator sensibilities through disaggregation. In Proceedings of the 17th International Workshop on Semantic Evaluation, pages 978–985, Toronto, Canada. Associa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leonardelli-2023-semeval-2023-task-11-learning/