Elamur Research⌕
☰
home / papers / 2026 / Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment

Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment

Paper · 2026 · Farida Said and Jeanne Villaneau

Summary

The paper asks whether agreement among several LLM annotators reliably signals label quality for emotion annotation. The test case is children's speech, where feelings are usually implied rather than named. The authors use 2,106 clause-level segments from interviews with 184 French-speaking children (ages 6–11), who described the concepts FATHER and MOTHER. Claude Sonnet 4 and GPT-5.2 each labelled valence and intensity under three zero-shot prompt postures, giving six pseudo-annotators. Their outputs were combined with Majority Vote, Dawid–Skene and MACE. FEEL-based lexicon baselines with Stanza lemmatization were also evaluated. Three experts annotated a stratified 500-segment gold standard, which serves as the external reference.

The internal reliability estimates from Dawid–Skene turned out to be badly miscalibrated. Both models tend to over-assign neutral labels, and this shared error looks like consensus to the aggregator. The gold standard reveals the problem and even reverses which model looks better. The authors conclude that high inter-LLM agreement cannot replace human validation for affect labelling. They release the gold subset and the probabilistic corpus.

Key points

  • Dawid–Skene credits GPT-5.2 with 90.7% valence accuracy, but the gold standard gives 71.0%, an overestimate of 19.7 points. Claude Sonnet 4 is underestimated by 6.8 points, which flips the model ranking.
  • The three aggregation methods produce nearly the same labels: 98.6% three-way agreement (493/500). Their gold accuracies are also nearly identical (MV 69.2%, DS 69.0%, MACE 69.2%; κ ≈ 0.475). This points to annotator bias, not the aggregation rule, as the source of error.
  • A conservative probe used a 61-entry simplified FEEL lexicon with Stanza lemmatization. It finds no overt affect words in about 76% of segments.
  • The gold labels are 68.4% positive and 26.0% neutral. The LLM-predicted distribution was 53.7% positive and 40.8% neutral, consistent with systematic over-neutralization.
  • Expert inter-annotator agreement was κ = 0.682–0.731. The final gold matches a Dawid–Skene aggregation of the human labels alone (κ = 0.983–0.991). The source data is the COS-MOS corpus (CC-BY-4.0).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institut de Recherche en Informatique et Systèmes AléatoiresRennes · France7 works · 1,944 citations🎓Université de Bretagne SudLorient · France2 works · 35 citations🔬Laboratoire de Mathématiques de Bretagne AtlantiqueBrest · France2 works

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksGeneral Social Sciences · 227 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 works

Cite

Cite as Farida Said and Jeanne Villaneau. 2026. Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment. Proceedings of Computational Affective Science (CAS) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/39tk69v6ca4p

@inproceedings{said2026multi,
  title = {{Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment}},
  author = {Farida Said and Jeanne Villaneau},
  booktitle = {Proceedings of Computational Affective Science (CAS) @ LREC 2026},
  year = {2026},
  doi = {10.63317/39tk69v6ca4p},
  url = {https://doi.org/10.63317/39tk69v6ca4p}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. Computational Linguistics, 34:555–596, 2008. find
  2. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. find
  3. A. Philip Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C, 28:20–28, 1979. find
  4. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382, 1971. find
  5. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv:2303.15056, 2023.
  6. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017), pages 1321–1330. find
  7. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120– 1130, Atlanta, Georgia, 2013. Associat…
  8. Saurav Kadavath et al. Language models (mostly) know what they don’t know. arXiv preprint arXiv:2207.05221, 2022. find
  9. Klaus Krippendorff. Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC), Annenberg School for Communication, University of Pennsylvania, 2011. find
  10. Chao Liu and Charis Chen. Text mining and sentiment analysis: A new lens to explore the emotion dynamics of mother–child interactions. Social Development, 33, 2024. https://doi.org/10.1 111/sode.12733. find
  11. Minxue Niu, Mimansa Jaiswal, and Emily Mower Provost. From text to emotion: Unveiling the emotion annotation capabilities of LLMs. In Proceedings of Interspeech 2024, pages 2650–2654, 2024. https://doi.org/10.2 1437/Interspeech.2024-2282. find
  12. Minxue Niu, Yara El-Tawil, Amrit Romana, and Emily Mower Provost. Rethinking emotion annotations in the era of large language models. IEEE Transactions on Affective Computing, 16 (04):2668–2679, 2025. find
  13. Marissa Ogren and Catherine M. Sandhofer. Emotion words in early childhood: A language transcript analysis. Cognitive Development, 60: 101122, 2021. https://doi.org/10.1016/ j.cogdev.2021.101122. find
  14. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. Automated annotation with generative AI requires validation. arXiv preprint arXiv:2306.00176, 2023. https://doi.org/ 10.48550/arXiv.2306.00176.
  15. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585, 2018.
  16. Barbara Plank. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, 2022. https://doi.org/10.18653/v1/2022.emnlp-m ain.731. ↗
  17. Peng Qi et al. Stanza: A python NLP toolkit for many human languages. In Proceedings of ACL 2020: System Demonstrations, pages 101–108, 2020. find
  18. Kenji Sagae, Eric Davis, Alon Lavie, Brian MacWhinney, and Shuly Wintner. Morphosyntactic annotation of CHILDES transcripts. Journal of Child Language, 37(3):705–729, 2010. find
  19. Woosuk Seo, Chanmo Yang, and Young-Ho Kim. Chacha: Leveraging large language models to prompt children to share their emotions about personal events. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–20. ACM, 2024. CHI ’24. find
  20. Carlo Strapparava and Rada Mihalcea. Semeval-2007 task 14: Affective text. In Proceedings of SemEval-2007, pages 70–74, 2007. spans. If uncertain, use confidence in [0,1] accordingly and keep evidence_span minimal but valid. Do not add any extra keys.
  21. Petter Törnberg. ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages. arXiv:2304.06588, 2023.
  22. Posture variants (appended to the base prompt). P1 (conservative): “Annotate carefully and conservatively. Prefer fewer labels if ambiguous.” P2 (implicit): “Annotate based on the child’s expressed feeling/attitude in the text, even if implicit.” P3 (lexical): “Annotate with emphasis on explicit lex… — [doi:10.1613/jair.1](https://doi.org/10.1613/jair.1) find
  23. Language Resource References User message (per segment). { "segment_id": \<int\>, "child_id": \<int\|null\>, "concept": "father\|mother", "question": "\<question_type\>", "segment_text": "\<exact child , → utterance\>" Amine Abdaoui, Jérôme Azé, Sandra Bringay, and Pascal Poncelet. FEEL: a french expanded emo… find
  24. Abdelhadi Bellachhab, Olga Galatanu, Frédéric Pugniere-Saavedra, Valérie Rochaix, Farida Saïd, and Jeanne Villaneau. Projet COS-MOS : Transcriptions anonymisées d’interviews d’enfants sur la construction sémantique et les stéréotypes. HAL: hal-04893220, 2025. find
  25. Humberto Pérez-Espinosa, Juan Martínez-Miranda, Ismael Espinosa-Curiel, Josefina Rodríguez-Jacobo, Luis Villaseñor-Pineda, and Himer Avila-George. IESC-Child: An interactive emotional children’s speech corpus. Computer Speech & Language, 59:55–74, 2020. https://doi.org/10.1016/j.csl.2019.06.006. — [doi:10.1016/j.csl.2019.06.006](https://doi.org/10.1016/j.csl.2019.06.006) ↗
  26. Arielle Syssau, Adil Yakhloufi, Edouard Giudicelli, Catherine Monnier, and Royce Anders. FAN-Cat: French affective norms for ten emotional categories. Behavior Research Methods, 53:447– 465, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/said-2026-multi-source-emotion-annotation-children/