Elamur Research⌕
☰
home / papers / 2023 / Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

Paper · 2023 · Enrica Troiano, Laura Oberländer and Roman Klinger

Summary

The paper asks whether appraisal theories from psychology, which link emotions to how people cognitively evaluate events, are useful for computational emotion analysis in text. Appraisals include whether an event was sudden, pleasant, goal-relevant, or under one's control. The authors crowdsourced crowd-enVENT, a corpus of 6,600 event descriptions written by English native speakers. Each author recalled an event tied to a given emotion and rated their own appraisals on 21 dimensions. They also reported demographics, personality traits, and their current emotional state. External crowdworkers then read part of the texts and tried to infer the writers' emotions and appraisals, which provides a human baseline. Text classifiers reconstructed both emotions and appraisals about as well as human readers, and sometimes better. Adding appraisal information to text-based emotion classifiers gave a modest gain. The study shows that appraisals can be annotated and predicted reliably. This makes them a theoretically grounded alternative or complement to discrete and valence–arousal emotion models, especially for texts where emotions are implied only through described events.

Key points

  • Resource: crowd-enVENT has 6,600 English texts labeled with 13 categories (12 emotions plus "no emotion") and 21 appraisal variables rated on 5-point scales. It also includes author metadata such as demographics, Big Five–style traits, and current emotion.
  • Design: first-person (writer) annotations are compared with reader reconstructions from a separate validation phase.
  • Appraisal prediction: the average F1 was .62 for humans and .75 for the model. Regression RMSE was 1.46 for humans and 1.40 for the model.
  • Emotion prediction: macro F1 was .56 for humans and .59 for the text model. It rose to .61 with gold appraisals added and .60 with predicted appraisals.
  • Appraisals alone: using only appraisals as input, without the text, yielded weaker emotion classification, with macro F1 of about .31–.35.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany94 works · 46,609 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksSocial Psychology · 330 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Enrica Troiano, Laura Oberländer and Roman Klinger. 2023. Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00461 · arXiv 2206.05238

@article{troiano2023dimensional,
  title = {{Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction}},
  author = {Enrica Troiano, Laura Oberländer and Roman Klinger},
  journal = {Computational Linguistics},
  year = {2023},
  doi = {10.1162/coli\_a\_00461},
  eprint = {2206.05238},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/coli_a_00461}
}

References

64 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, Muhammad and Lyle Ungar. 2017. EmoNet: Fine-grained emotion detection with gated recurrent neural networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 718–728. https://doi.org/10.18653/v1/P17-1067 ↗
  2. Adolphs, Ralph. 2017. How should neuroscience study emotions? By distinguishing emotion states, concepts, and experiences. Social Cognitive and Affective Neuroscience, 12(1):24–31. https://doi.org/10.1093/scan/nsw153, PubMed: 27798256 Akhtar, Shad, Deepanway Ghosal, Asif Ekbal, Pushpak Bhattacharyya… — [doi:10.1093/scan/nsw153](https://doi.org/10.1093/scan/nsw153) ↗
  3. Alm, Cecilia Ovesdotter, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 579–586. https://doi.org/10.3115/12205… ↗
  4. Aman, Saima and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue, pages 196–205, Springer, Berlin Heidelberg. find
  5. Balahur, Alexandra, Jesus M. Hermida, and Andres Montoyo. 2011. Building and exploiting EmotiNet, a knowledge base for emotion detection based on the appraisal theory model. IEEE Transactions on Affective Computing, 3(1):88–101. https://doi.org/10.1109/T-AFFC.2011.33 Balahur, Alexandra, Jesus M. He… — [doi:10.1109/t-affc.2011.33](https://doi.org/10.1109/t-affc.2011.33) ↗
  6. Bostan, Laura Ana Maria and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2104–2119. find
  7. Bradley, Margaret M. and Peter J. Lang. 1994. Measuring emotion: The Self-Assessment Manikin and the Semantic Differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1):49–59. https://doi.org /10.1016/0005-7916(94)90063-9 Bradley, Margaret M. and Peter J. Lang. 1999. Affective norm… — [doi:10.1016/0005-7916(94)90063-9](https://doi.org/10.1016/0005-7916%2894%2990063-9) ↗
  8. Breazeal, Cynthia, Kerstin Dautenhahn, and Takayuki Kanda. 2016. Social robotics. Springer Handbook of Robotics. Springer International Publishing, Cham, pages 1935–1972. https://doi.org/10 .1007/978-3-319-32552-1 72 Buechel, Sven and Udo Hahn. 2016. Emotion analysis as a regression problem–dimensio… find
  9. Buechel, Sven and Udo Hahn. 2017a. EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 578–585.… find
  10. Buechel, Sven, Luise Modersohn, and Udo Hahn. 2021. Towards label-agnostic emotion embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9231–9249. https://doi.org/10 .18653/v1/2021.emnlp-main.728 Buechel, Sven, Susanna Rücker, and Udo Hahn. 20… find
  11. Canty, Angelo and B. D. Ripley. 2021. boot: Bootstrap R (S-Plus) Functions. R package version 1.3-28. find
  12. Casel, Felix, Amelie Heindl, and Roman Klinger. 2021. Emotion recognition under consideration of the emotion component process model. In Proceedings of the 17th Conference on Natural Language Processing (KONVENS 2021), pages 49–61. find
  13. Cheng, Yu-Ya, Wen-Chao Yeh, Yan-Ming Chen, and Yung-Chun Chang. 2021. Using valence and arousal-infused Bi-LSTM for sentiment analysis in social media product reviews. In Proceedings of the 33rd Conference on Computational Linguistics and Speech Processing (ROCLING 2021), pages 210–217. find
  14. Clark, Elizabeth A., J’Nai Kessinger, Susan E. Duncan, Martha Ann Bell, Jacob Lahne, Daniel L. Gallagher, and Sean F. O’Keefe. 2020. The facial action coding system for characterization of human affective response to consumer product-based stimuli: A systematic review. Frontiers in Psychology, Artic… — [doi:10.3389/fpsyg.2020.00920](https://doi.org/10.3389/fpsyg.2020.00920) find
  15. Clore, Gerald. L. and Andrew Ortony. 2013. Psychological construction in the OCC model of emotion. Emotion Review, 5(4):335–343. https://doi.org/10.1177/1754073913489751, PubMed: 25431620 Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 2… ↗
  16. Davison, Anthony C. and David V. Hinkley. 1997. Bootstrap Methods and Their Applications. Cambridge University Press, Cambridge. De Bruyne, Luna, Orphée De Clercq, and Véronique Hoste. 2021. Prospects for Dutch emotion detection: Insights from the new emotionl dataset. Computational Linguistics in… find
  17. Demszky, Dorottya, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054. https://doi.org/10.18653/v1/2020 .acl… ↗
  18. Ekman, Paul. 1972. Universals and cultural differences in facial expressions of emotion. In Nebraska Symposium on Motivation 1971, volume 19, Lincoln University of Nebraska Press. find
  19. Ekman, Paul. 1992. An argument for basic emotions. Cognition & Emotion, 6(3–4):169–200. https://doi.org/10 .1080/02699939208411068 Ekman, Paul and Wallace V. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA.…
  20. Ellsworth, Phoebe C. and Craig A. Smith. 1988. From appraisal to emotion: Differences among unpleasant feelings. Motivation and Emotion, 12(3):271–302. https://doi.org/10.1007/BF00993115 ↗
  21. Felbo, Bjarke, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 161…
  22. Feldman Barrett, Lisa. 2017. The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(11):1833. https://doi.org/10.1093/scan/nsx060, PubMed: 28472391 Feldman Barrett, Lisa. 2018. How Emotions Are Made: The Se… ↗
  23. HMH Books. Frijda, Nico H., Peter Kuipers, and Elisabeth Ter Schure. 1989. Relations among emotion, appraisal, and emotional action readiness. Journal of Personality and Social Psychology, 57(2):212. https://doi.org/10.1037/0022-3514.57.2.212 Gardner, Matt, Joel Grus, Mark Neumann, Oyvind Tafjord,… — [doi:10.1037/0022-3514.57.2.212](https://doi.org/10.1037/0022-3514.57.2.212) ↗
  24. Haider, Thomas, Steffen Eger, Evgeny Kim, Roman Klinger, and Winfried Menninghaus. 2020. PO-EMO: Conceptualization, annotation, and modeling of aesthetic emotions in German and English poetry. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1652–1663. find
  25. Hall, Judith A., Marianne Schmid Mast, and Tessa V. West. 2016. Accurate interpersonal perception: Many traditions, one topic. In The Social Psychology of Perceiving Others Accurately. Cambridge University Press, pages 3–22. https://doi.org/10.1017/CBO9781316181959.001 Hofmann, Jan, Enrica Troiano,… ↗
  26. Hofmann, Jan, Enrica Troiano, Kai Sassenberg, and Roman Klinger. 2020. Appraisal theories for emotion classification in text. In Proceedings of the 28th International Conference on Computational Linguistics, pages 125–138. https://doi.org/10.18653/v1/2020 .coling-main.11 Israel, Laura Sophia Finja a… ↗
  27. Kim, Hyoung Rock and Dong-Soo Kwon. 2010. Computational model of emotion generation for human–robot interaction based on the cognitive appraisal theory. Journal of Intelligent & Robotic Systems, 60(2):263–283. https://doi.org/10 .1007/s10846-010-9418-7 Klinger, Roman, Orphée De Clercq, Saif Mohamma… find
  28. Li, Yanran, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995.
  29. Liu, Bing. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167. https://doi.org/10.1007/978-3-031 -02145-9 1, https://doi.org/10.1007/978-3-031-02145-9 Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Le… ↗
  30. Louviere, Jordan J., Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. https://doi.org/10.1017/CBO9781107337855 Mancini, Giacomo, Roberta Biolcati, Sergio Agnoli, Federica Andrei, and Elena Trombini. 2018. Recognition of fac… — [doi:10.1017/cbo9781107337855](https://doi.org/10.1017/cbo9781107337855) ↗
  31. Manstead, A. S. R. and Philip E. Tetlock. 1989. Cognitive appraisals and emotional experience: Further evidence. Cognition and Emotion, 3(3):225–240. https://doi.org/10.1080/02699938908415243 Mihalcea, Rada and Carlo Strapparava. 2012. Lyrics, music, and emotions. In Proceedings of the 2012 Joint C… — [doi:10.1080/02699938908415243](https://doi.org/10.1080/02699938908415243) ↗
  32. Mohammad, Saif. 2012. \#emotional tweets. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 246–…
  33. Mohammad, Saif. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184. https://doi.org/10.18653/v1/P18-1017 ↗
  34. Mohammad, Saif, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 Task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17. https://doi.org/10.18653/v1/S18-1001 Mohammad, Saif, Xiaodan Zhu, and Joel Martin. 201… ↗
  35. Nair, Vinod and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807–814. find
  36. Oatley, Keith and Philip N. Johnson-Laird. 2014. Cognitive approaches to emotions. Trends in Cognitive Sciences, 18(3):134–140. https://doi.org/10.1016/j.tics .2013.12.004, PubMed: 24389368 Omdahl, Becky Lynn. 1995. — [doi:10.1016/j.tics](https://doi.org/10.1016/j.tics) ↗
  37. Cognitive Appraisal, Emotion, and Empathy. Mahwah, NJ: Lawrence Erlbaum. Park, Sungjoon, Jiseon Kim, Seonghyeon Ye, Jaeyeol Jeon, Hee Young Park, and Alice Oh. 2021. Dimensional emotion detection from categorical emotion. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language… find
  38. Pennebaker, James W., Martha E. Francis, and Roger J. Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahwah, NJ: Lawrence Erlbaum Associates, 71:2001. find
  39. Plutchik, Robert. 1970. Emotions, evolution and adaptive processes. In Arnold, M. B., editor, Feelings and Emotions. New York: Academic Press, pages 3–24. https://doi.org/10.1016/B978-0-12-063550 -4.50007-3 Plutchik, Robert. 2001. The nature of emotions. American Scientist, 89(4):344–350. https://d… — [doi:10.1016/b978-0-12-063550](https://doi.org/10.1016/b978-0-12-063550) ↗
  40. Roseman, I. J. and C. A. Smith. 2001. Appraisal theory: Overview, assumptions, varieties, controversies. Appraisal Processes in Emotion: Theory, Methods, Research, pages 3–19. find
  41. Roseman, Ira J. 1984. Cognitive determinants of emotion: A structural theory. Review of Personality & Social Psychology, 5:11–36. find
  42. Roseman, Ira J. 1996. Appraisal determinants of emotions: Constructing a more accurate and comprehensive theory. Cognition and Emotion, 10(3):241–278. https://doi.org/10.1080/026999396380240 Roseman, Ira J. 2001. A model of appraisal in the emotion system. Appraisal Processes in Emotion: Theory, Me… — [doi:10.1080/026999396380240](https://doi.org/10.1080/026999396380240) ↗
  43. Roseman, Ira J., Martin S. Spindel, and Paul E. Jose. 1990. Appraisals of emotion-eliciting events: Testing a theory of discrete emotions. Journal of Personality and Social Psychology, 59(5):899–915. https://doi.org/10.1037/0022-3514 .59.5.899 — [doi:10.1037/0022-3514](https://doi.org/10.1037/0022-3514) ↗
  44. Rubner, Yossi, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99–121. find
  45. Sailunaz, Kashfia, Manmeet Dhaliwal, Jon Rokne, and Reda Alhajj. 2018. Emotion detection from text and speech: A survey. Social Network Analysis and Mining, 8(1):1–26. https://doi.org/10.1007/s13278-018-0505-2 Sander, David, Didier Grandjean, and Klaus R. Scherer. 2005. A systems approach to apprai… — [doi:10.1016/j.neunet.2005.03.001](https://doi.org/10.1016/j.neunet.2005.03.001) ↗
  46. Scherer, Klaus R. 2005. What are emotions? And how can they be measured? Social Science Information, 44(4):695–729. https://doi.org/10.1177/0539018405058216 Scherer, Klaus R., Tanja Bänziger, and Etienne Roesch. 2010. A Blueprint for Affective Computing: A Sourcebook and Manual. ↗
  47. Oxford University Press. Scherer, Klaus R. and Johnny J. R. Fontaine. 2013. Driving the emotion process: The appraisal component. In J. J. R. Fontaine, Klaus R. Scherer, and C. Soriano, editors, Series in Affective Science. Components of Emotional Meaning: A Sourcebook. Oxford University Press, Oxfo… find
  48. Scherer, Klaus R., A. Schorr, and T. Johnstone. 2001a. Appraisal Considered as a Process of Multi-level Sequential Checking, volume 92. find
  49. Oxford University Press. Scherer, Klaus R., Angela Schorr, and Tom Johnstone. 2001b. Appraisal Processes in Emotion: Theory, Methods, Research. find
  50. Oxford University Press. Scherer, Klaus R. and Harald G. Wallbott. 1997. The ISEAR questionnaire and codebook. Geneva Emotion Research Group. https://www.unige.ch/cisa/research/materials-and-online -research/research-material/. Schuff, Hendrik, Jeremy Barnes, Julian Mohme, Sebastian Padó, and Roma… — [doi:10.18653/v1/w17](https://doi.org/10.18653/v1/w17) ↗
  51. A. and Phoebe. C. Ellsworth. 1985. Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48(4):813–838. https://doi.org/10.1037/0022-3514.48.4.813, PubMed: 3886875 Srivastava, Nitish, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.… ↗
  52. Staller, Alexander and Paolo Petta. 2001. Introducing emotions into the computational study of social norms: A first evaluation. Journal of Artificial Societies and Social Simulation, 4(1):U27–U60. find
  53. Steunebrink, Bas R., Mehdi Dastani, and John-Jules Ch. Meyer. 2009. The OCC model revisited. Online: https://people.idsia.ch/steunebrink/Publications/KI09 OCC revisited.pdf. ↗
  54. Stranisci, Marco Antonio, Simona Frenda, Eleonora Ceccaldi, Valerio Basile, Rossana Damiano, and Viviana Patti. 2022. APPReddit: A corpus of reddit posts annotated for appraisal. In Proceedings of the 13th Language Resources and Evaluation Conference. find
  55. Strapparava, Carlo and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74. https://doi.org/10.3115/1621474 .1621487 Strapparava, Carlo and Alessandro Valitutti. 2004. WordNet affect: An a… ↗
  56. Toprak, Cigdem, Niklas Jakob, and Iryna Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575–584.
  57. Tracy, Jessica L. and Richard W. Robins. 2006. Appraisal antecedents of shame and guilt: Support for a theoretical model. Personality and Social Psychology Bulletin, 32(10):1339–1351. https://doi.org/10 .1177/0146167206290212, PubMed: 16963605 Troiano, Enrica, Laura Oberländer, Maximilian Wegge, an… find
  58. Wang, Yingqian, Skyler T. Hawk, Yulong Tang, Katja Schlegel, and Hong Zou. 2019. Characteristics of emotion recognition ability among primary school children: Relationships with peer status and friendship quality. Child Indicators Research, 12(4):1369–1388. https://doi.org/10 .1007/s12187-018-9590-z… find
  59. Wierzbicka, Anna. 1994. Emotion, language, and cultural scripts. Emotion and Culture: Empirical Studies of Mutual Influence, pages 133–196. https://doi.org/10 .1037/10152-004 Wilson, Theresa. 2008. Annotating subjective content in meetings. In Proceedings of the Sixth International Conference on Lan… find
  60. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  61. Xia, Rui and Zixiang Ding. 2019. Emotion-cause pair extraction: A new task to emotion analysis in texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1003–1012. https://doi.org/10 .18653/v1/P19-1096 Yanchus, Nancy Jane. 2006. Development and Valid… find
  62. Yih, Jennifer, Leslie D. Kirby, and Craig A. Smith. 2020. Profiles of appraisal, motivation, and coping for positive emotions. Cognition and Emotion, 34(3):481–497. https://doi.org/10 .1080/02699931.2019.1646212, PubMed: 32314674 Yih, Jennifer, Andero Uusberg, Weiqiang Qian, and James J. Gross. 2020… find
  63. Zhang, Lei and Bing Liu. 2011b. Identifying noun product features that imply opinions. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 575–580. find
  64. Zhuang, Liu, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/troiano-2023-dimensional-modeling-emotions-text-appraisal/