Elamur Research⌕
☰
home / papers / 2026 / Less Is More? The Role of Demographic Author Information in Emotion Classification of Ambiguous Text

Less Is More? The Role of Demographic Author Information in Emotion Classification of Ambiguous Text

Summary

Emotion labels for short texts often vary between annotators because a text alone may not reveal how its writer felt. The authors test whether knowing who wrote a text (age, gender, education, ethnicity) helps readers converge on one interpretation and recover the writer's own label. They took the 250 most ambiguous texts from the crowd-enVENT corpus, chosen by low reader agreement on emotion and appraisal labels. Via Prolific, 500 annotators labelled these texts, split into two groups: one saw author demographics and one saw the text alone. Author details did not raise inter-annotator agreement or accuracy against the gold labels. Two small exceptions appeared. Performance rose slightly for texts whose polarity was unclear, and annotators did better when the gold emotion matched gender stereotypes. Zero-shot experiments with four open-weight LLMs showed partly similar patterns, but the models trailed humans, and author information usually lowered their performance. The study suggests that adding demographic context is no simple fix for annotation ambiguity and may introduce bias.

Key points

  • Data: 250 low-agreement texts from crowd-enVENT (Troiano et al., 2023), in which each event description carries the author's own emotion label plus labels from 5 readers.
    • The texts are split evenly: in 125, the original reader majority guessed the gold label; in the other 125, it did not.
  • Annotation setup:
    • 500 Prolific annotators, each labelling 5 texts, for 10 annotations per text and 2,500 in total, at a cost of £1286.
    • The label set has 12 emotions (no "no-emotion" category) and 21 appraisal dimensions.
  • Human results:
    • Demographics did not improve agreement among annotators (RQ1) or with the gold label (RQ2).
    • F1 rose slightly only for texts with ambiguous polarity.
    • Accuracy was higher when the gold label was stereotype-congruent, e.g. fear for women and anger for men.
  • LLM experiments:
    • Zero-shot prompting via Ollama of Gemma, Mistral, Mixtral and a fourth model of about 3.2B parameters.
    • All models scored below humans, and author information mostly reduced performance except in the stereotype-aligned and unclear-polarity cases (RQ3: partially similar to humans).
  • Resource: All study data is publicly released by the University of Bamberg NLP group.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2cw8tpo82h55

@inproceedings{weber2026less,
  title = {{Less Is More? The Role of Demographic Author Information in Emotion Classification of Ambiguous Text}},
  author = {Weber, Sabine and Greschner, Lynn and Klinger, Roman},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2cw8tpo82h55},
  url = {https://doi.org/10.63317/2cw8tpo82h55}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Henning Bartsch, Ole Jorgensen, Domenic Rosati, Jason Hoelscher-Obermaier, and Jacob Pfau. 2023. Self-consistency of large language models under ambiguity. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 89–105, Singapore. Association for Com… find
  2. Aidan Combs, Graham Tierney, Fatima Alqabandi, Devin Cornell, Gabriel Varela, Andrés Castro Araújo, Lisa P Argyle, Christopher A Bail, and Alexander Volfovsky. 2023. Perceived gender and political persuasion: a social media field experiment during the 2020 us democratic presidential primary election… find
  3. Aline Étienne, Delphine Battistelli, and Gwénolé Lecorvé. 2024. Emotion identification for French in written texts: Considering modes of emotion expression as a step towards text complexity analysis. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social… find
  4. Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2025. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation, 59:1719–1746. find
  5. Samuel D. Gosling, Peter J. Rentfrow, and William B. Swann. 2003. A very brief measure of the big-five personality domains. Journal of Research in Personality, 37(6):504–528. find
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, and (many others). 2024. The Llama 3 herd of models. arXiv preprint, arXiv:2407.21783. find
  7. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  8. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  9. Przemysław Kazienko, Julita Bielaniewicz, Marcin Gruza, Kamil Kanclerz, Konrad Karanowski, Piotr Miłkowski, and Jan Kocoń. 2023. Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor. Information Fusion, 94:43–65. find
  10. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  11. Lea Krause and Piek T.J.M. Vossen. 2024. The Gricean maxims in NLP - a survey. In Proceedings of the 17th International Natural Language Generation Conference, pages 470–485, Tokyo, Japan. Association for Computational Linguistics. find our card
  12. Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, and Markus Strohmaier. 2025. The prompt makes the person(a): A systematic evaluation of sociodemographic persona prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23212–23237, Suz… find
  13. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie… find
  14. Wiktoria Mieleszczenko-Kowszewicz, Kamil Kanclerz, Julita Bielaniewicz, Marcin Oleksy, Marcin Gruza, Stanisław Woźniak, Ewa Dzięcioł, Przemysław Kazienko, and Jan Kocoń. 2023. Capturing human perspectives in NLP: Questionnaires, annotations, and biases. In European Conference on Artificial Intellige… find
  15. Piotr Miłkowski, Marcin Gruza, Kamil Kanclerz, Przemyslaw Kazienko, Damian Grimling, and Jan Kocoń. 2021. Personal bias in prediction of emotions elicited by textual opinions. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint… find
  16. Piotr Miłkowski, Stanisław Saganowski, Marcin Gruza, Przemysław Kazienko, Maciej Piasecki, and Jan Kocoń. 2022. Multitask personalized recognition of emotions evoked by textual content. In 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Eve… find
  17. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Lima Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino Dario Mario Ali,… find
  18. E. Ashby Plant, Janet Shibley Hyde, Dacher Keltner, and Patricia G Devine. 2000. The gender stereotyping of emotions. Psychology of women quarterly, 24(1):81–92. find
  19. Flor Miriam Plaza-del-Arco, Alba Cercas Curry, Amanda Cercas Curry, and Dirk Hovy. 2024a. Emotion analysis in NLP: Trends, gaps and roadmap for future directions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 20… find
  20. Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Alba Curry, Gavin Abercrombie, and Dirk Hovy. 2024b. Angry men, sad women: Large language models reflect gendered stereotypes in emotion attribution. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  21. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. find our card
  22. Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel A. Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. 2025. Value profiles for encoding human variation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2… find
  23. Huaman Sun, Jiaxin Pei, Minje Choi, and David Jurgens. 2025. Sociodemographic prompting is not yet an effective approach for simulating subjective judgments with LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human… find
  24. Enrica Troiano, Laura Oberländer, and Roman Klinger. 2023. Dimensional modeling of emotions in text with appraisal theories: Corpus creation, annotation reliability, and prediction. Computational Linguistics, 49(1):1–72. find
  25. Eileen Wemmer, Sofie Labat, and Roman Klinger. 2024. EmoProgress: Cumulated emotion progression analysis in dreams and customer service dialogues. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5660–… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weber-2026-less-is-more-role-demographic/