Elamur Research⌕
☰
home / papers / 2026 / Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

Paper · 2026 · Emily Sofi Ohman and Anna Koufakou

Summary

Emotion datasets in NLP usually reduce several annotators' judgments to one "gold" label, often by majority vote, and treat disagreement as noise. Ohman and Koufakou ask whether that disagreement carries useful information in multilabel emotion annotation. Three expert annotators labeled Hemingway's The Old Man and the Sea roughly sentence by sentence. They used Plutchik's eight core emotions, up to three emotions per item, intensity ratings, and an optional free-text field for other emotions. The resulting resource is called TOMATS. The authors show that classical agreement coefficients fit this setting poorly and that annotator behavior varies a lot. Instead of hard labels, they build soft vote-share targets, including a variant weighted by intensity. They then evaluate classifiers with thresholded scores (macro-/micro-F1) and with a probabilistic measure, Bernoulli cross-entropy (SoftBCE), plus diagnostics of disagreement. The main finding is that disagreement follows patterns and shows up in model behavior. Hard labels tend to give the best F1, while soft supervision produces predictions closer to real annotator uncertainty. This gives practical guidance on how to aggregate and evaluate subjective multilabel datasets.

Key points

  • TOMATS resource: 1,677 English segments, 3 annotators, and 8 Plutchik labels (joy and trust, fear and surprise, sadness and disgust, anger and anticipation). Intensities on a 0–10 scale were rescaled to [0,1] so they can serve as aggregation weights.
  • Low per-emotion agreement: Krippendorff's α for binary presence ranged from 0.092 (trust) to 0.279 (sadness). The authors argue these scores are poorly suited to multilabel emotion data.
  • Two similar annotators: Annotators 1 and 3 behaved alike. Their primary-emotion agreement was 42%, rising to 62% when any overlapping label counted.
  • One sparse annotator: Annotator 2 labeled only about 10% of items (184 primary, 13 secondary, 3 tertiary labels). The authors attribute this to weak engagement with the protocol rather than a different reading of the text.
  • Structured confusions and open release: Cross-annotator overlap matrices show fear often paired with anticipation and anger with disgust. Code and derived annotation tables (without the copyrighted novel) are available at github.com/esohman/SoftBCE_Emotions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Waseda UniversityTokyo · Japan20 works · 584 citations

Also on the byline: Florida Gulf Coast University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksDecision Sciences · 209 worksStatistics, Probability and Uncertainty · 61 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 972 works

Cite

Cite as Emily Sofi Ohman and Anna Koufakou. 2026. Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework. Proceedings of Computational Affective Science (CAS) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3mad4saen3x8

@inproceedings{ohman2026quality,
  title = {{Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework}},
  author = {Emily Sofi Ohman and Anna Koufakou},
  booktitle = {Proceedings of Computational Affective Science (CAS) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3mad4saen3x8},
  url = {https://doi.org/10.63317/3mad4saen3x8}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alina Andreevskaia and Sabine Bergler. 2007. CLaC and CLaC-NB: Knowledge-based and corpus-based approaches to sentiment tagging. In Proceedings of the 4th international workshop on semantic evaluations, pages 117–120. Association for Computational Linguistics. find
  2. Ernest Hemingway. 1995. The old man and the sea. 1952. New York: Scribner. find
  3. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. As…
  4. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. find
  5. Christina Barz, Melanie Siegel, Daniel Hanss, and Michael Wiegand. 2025. Understanding disagreement: An annotation study of sentiment and emotional language in environmental communication. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 1–20. find
  6. Anna Koufakou and Elijah Nieves. 2025. Review of recent emotion-annotated text corpora and resources. Language Resources and Evaluation, pages 1–35. find
  7. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15–21. Association for Computationa…
  8. Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. 2023. SemEval-2023 task 11: Learning with disagreements (LeWiDi). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval…
  9. Adam Bermingham and Alan F Smeaton. 2009. A study of inter-annotator agreement for opinion retrieval. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 784–785. find
  10. Elisa Leonardelli, Silvia Casola, Siyao Peng, Giulia Rizzi, Valerio Basile, Elisabetta Fersini, Diego Frassinelli, Hyewon Jang, Maja Pavlovic, Barbara Plank, and Massimo Poesio. 2025. LeWiDi-2025 at NLPerspectives: Third edition of the learning with disagreements shared task. In Proceedings of the T… find
  11. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860–6868.
  12. Alan S Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences, 114(38):E7900–E7909. find
  13. Saif Mohammad. 2016. A practical guide to sentiment annotation: Challenges and solutions. In WASSA@ NAACL-HLT, pages 174–179. find
  14. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054. find
  15. Saif Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. In Proceedings of the 6th joint conference on lexical and computational semantics (* SEM 2017), pages 65–77.
  16. Saif M Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Computational Linguistics, 48(2):239–278.
  17. Paul Ekman. 1971. Universals and cultural differences in facial expressions of emotion. In Nebraska symposium on motivation. University of Nebraska Press. find
  18. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  19. Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2025. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation, 59(2):1719–1746. find
  20. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. find
  21. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufiño, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ib… find
  22. Robert Plutchik. 1984. Emotions: A general psychoevolutionary theory. Approaches to emotion, 1984(197-219):2–4. find
  23. Giulia Rizzi, Elisa Leonardelli, Massimo Poesio, Alexandra Uma, Maja Pavlovic, Silviu Paun, Paolo Rosso, and Elisabetta Fersini. 2024. Soft metrics for evaluation with disagreements: an assessment. In 3rd Workshop on Perspectivist Approaches to NLP, NLPerspectives 2024, pages 84–94. European Languag… find
  24. Riikka Rossi and Emily Öhman. 2025. Combining qualitative and computational approaches for literary analysis of finnish novels. Scandinavian Studies, 97(3):27–51. find
  25. James A Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. Journal of research in Personality, 11(3):273–294. find
  26. M. Munezero, C. Montero, E. Sutinen, and J. Pajunen. 2014. Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text. IEEE Transactions on Affective Computing, 5(02):101–111. find
  27. Carlo Strapparava and Rada Mihalcea. 2010. Annotating and identifying emotions in text. In Intelligent information access, pages 21–38. Springer. find
  28. Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, and Barbara Plank. 2024. Varierr nli: Separating annotation error from human label variation.
  29. Emily Öhman. 2020. Emotion annotation: Rethinking emotion categorization. In DHN Post-Proceedings, pages 134–144. find
  30. Janyce Wiebe and Ellen Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 486–497. Springer. find
  31. Emily Öhman. 2021. The language of emotions: Building and applying computational methods for emotion detection for English and beyond. Helsingin yliopisto. find
  32. Emily Öhman, Yuri Bizzoni, Pascale Feldkamp Moreira, and Kristoffer Nielbo. 2024. Emotionarcs: Emotion arcs for 9,000 literary texts. In Proceedings of the 8th joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature (LaTeCH-CLfL 2024), pag… find
  33. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  34. Flor Miriam Plaza-del Arco, Alba A. Cercas Curry, Amanda Cercas Curry, and Dirk Hovy. 2024. Emotion analysis in NLP: Trends, gaps and roadmap for future directions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources *Figure 3: Annotator 1 emoti… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ohman-2026-quality-agreement-multilabel-emotion-annotation/