Elamur Research⌕
☰
home / papers / 2024 / EmoBench: Evaluating the Emotional Intelligence of Large Language Models

EmoBench: Evaluating the Emotional Intelligence of Large Language Models

Paper · 2024 · Sahand Sabour, Siyang Liu, Zheyuan Zhang et al.

Summary

Most existing tests of emotional intelligence (EI) in large language models reuse older emotion datasets. These cover only emotion and cause recognition, and they let models succeed through frequent patterns, causes stated outright in the text, and noisy labels. To fix this, the authors created EmoBench, grounded in psychological theories of EI such as Salovey and Mayer and Goleman. It splits machine EI into two abilities:

  • Emotional Understanding (EU): working out what the people in a scenario feel and why.
  • Emotional Application (EA): using that insight to pick the most effective way to handle an emotional dilemma.

The resource contains 400 hand-written multiple-choice questions in English and Chinese. The scenarios involve several people, and answers must be inferred rather than copied from the context. For example, how upsetting a lost item is depends on its unstated sentimental value. Across the tested LLMs, even the best one, GPT-4, scores below the average human, and understanding is the weaker dimension. EmoBench therefore offers a harder, theory-based target for building emotionally aware systems, for example in mental health support.

Key points

  • 400 manually crafted MCQs, available in English and Chinese. Code and data are released at github.com/Sahandfer/EmoBench.
  • EU taxonomy categories:
    • Complex Emotions (transitions, mixtures, unexpected outcomes)
    • Personal Beliefs and Experiences (cultural value, sentimental value, persona)
    • Emotional Cues (vocal, visual)
    • Perspective-taking (faux pas, strange stories, false belief)
  • EA scenarios are organized by relationship type (personal vs. social) and problem type (concerning self vs. others).
  • Label quality checks:
    • EU gold labels and causes were verified by four workers.
    • EA showed high human agreement (Fleiss' Kappa = 0.852), with each question designed to have one clearly best answer.
  • Stated limitations:
    • The dataset is text-only and relatively small.
    • Only two languages are covered.
    • GPT-4 outputs were used as inspiration for the questions, which may introduce bias.
    • Only chain-of-thought prompting was explored, so the scores may not reflect peak model performance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSentiment & emotion · 1,035 works

Cite

Cite as Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea and Minlie Huang. 2024. EmoBench: Evaluating the Emotional Intelligence of Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.326 · arXiv 2402.12071

@inproceedings{sabour2024emobench,
  title = {{EmoBench: Evaluating the Emotional Intelligence of Large Language Models}},
  author = {Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea and Minlie Huang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.326},
  eprint = {2402.12071},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.326}
}

References

69 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa M. Amin, Rui Mao, Erik Cambria, and Björn W. Schuller. 2023. A wide evaluation of chatgpt on affective computing tasks. find
  2. Neal M Ashkanasy and Catherine S Daus. 2005. Rumors of the death of emotional intelligence in organizational behavior are vastly exaggerated. Journal of Organizational Behavior, 26(4):441–452. find
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  4. Reuven Bar-On. 1997. BarOn emotional quotient inventory, volume 40. Multi-health systems. find
  5. Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985. Does the autistic child have a “theory of mind”? Cognition, 21(1):37–46. find
  6. Simon Baron-Cohen, Michelle O’riordan, Valerie Stone, Rosie Jones, and Kate Plaisted. 1999. Recognition of faux pas by normally developing children and children with asperger syndrome or high-functioning autism. Journal of autism and developmental disorders, 29:407–418. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  8. Jeffrey M Conte. 2005. A review and critique of emotional intelligence measures. Journal of organizational behavior, 26(4):433–440. find
  9. Marzia Del Prete. 2021. Emotional artificial intelligence: detecting and managing customer emotions in automated customer service. find
  10. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547. find
  11. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
  12. Murray J Dyck, Kara Ferguson, and Ian M Shochet. 2001. Do autism spectrum disorders differ from each other and from non-spectrum disorders on emotion recognition tests? European child & adolescent psychiatry, 10:105–116. find
  13. Paul Ekman. 1984. Expression and the nature of emotion. Approaches to emotion, 3(19):344. find
  14. Lisa Fan, Matthias Scheutz, Monika Lohani, Marissa McCoy, and Charlene Stokes. 2017. Do we need emotionally intelligent artificial agents? first results of human perceptions of emotional intelligence in humans compared to robots. In Intelligent Virtual Agents: 17th International Conference, IVA 2017… find
  15. Fiona J Ferguson and Elizabeth J Austin. 2010. Associations of trait and ability emotional intelligence with performance on theory of mind tasks in an adult sample. Personality and individual differences, 49(5):414–418. find
  16. Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619. find
  17. Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2022. CICERO: A dataset for contextualized commonsense inference in dialogues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5010–5028, Dublin… find
  18. Daniel Goleman. 1996. Emotional intelligence. why it can matter more than iq. Learning, 24(6):49–50. find
  19. Francesca GE Happé. 1994. An advanced test of theory of mind: Understanding of story characters’ thoughts and feelings by able autistic, mentally handicapped, and normal children and adults. Journal of autism and Developmental disorders, 24(2):129–154. find
  20. Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. arXiv preprint arXiv:2310.16755. find
  21. Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. 2023. Who is chatgpt? benchmarking llms’ psychological portrayal using psychobench. arXiv preprint arXiv:2310.01386. — [arXiv:2310.01386](https://arxiv.org/abs/2310.01386) find
  22. Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut. 2023. CRoW: Benchmarking commonsense reasoning in real-world tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9785–9821, Singapore. Association for Computational… find
  23. Mirjana Ivanović, Miloš Radovanović, Zoran Budimac, Dejan Mitrović, Vladimir Kurbalija, Weihui Dai, and Weidong Zhao. 2014. Emotional intelligence and agents: Survey and possible applications. In Proceedings of the 4th International Conference on Web Intelligence, Mining and Semantics (WIMS14), p… find
  24. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. 2021. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574.
  25. Mridula C Jobson. 2020. Emotional maturity among adolescents and its importance. Indian Journal of Mental Health, 7(1):35–41. find
  26. Elke Kalbe, Marius Schlegel, Alexander T. Sack, Dennis A. Nowak, Manuel Dafotakis, Christopher Bangard, Matthias Brand, Simone Shamay-Tsoory, Oezguer A. Onur, and Josef Kessler. 2010. Dissociating cognitive from affective theory of mind: A tms study. Cortex, 46(6):769–780. find
  27. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397–14413, Singapore. A… find
  28. Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2305.15068.
  29. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
  30. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. arXiv preprint arXiv:2106.01144.
  31. Paulo N Lopes, Marc A Brackett, John B Nezlek, Astrid Schütz, Ina Sellin, and Peter Salovey. 2004. Emotional intelligence and social interaction. Personality and social psychology bulletin, 30(8):1018–1034. find
  32. Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023. Tom-challenges: A principle-guided dataset and diverse evaluation tasks for exploring theory of mind. arXiv preprint arXiv:2305.15068.
  33. Carolyn MacCann and Richard D Roberts. 2008. New paradigms for assessing emotional intelligence: theory and data. Emotion, 8(4):540. find
  34. John D Mayer, David R Caruso, and Peter Salovey. 1999. Emotional intelligence meets traditional standards for an intelligence. Intelligence, 27(4):267–298. find
  35. John D Mayer, Peter Salovey, and David R Caruso. 2007. Mayer-salovey-caruso emotional intelligence test. find
  36. Daniela Mier, Stefanie Lis, Kerstin Neuthe, Carina Sauer, Christine Esslinger, Bernd Gallhofer, and Peter Kirsch. 2010. The involvement of emotion recognition in affective theory of mind. Psychophysiology. find
  37. Peter J O’Connor, Andrew Hill, Maria Kaya, and Brett Martin. 2019. The measurement of emotional intelligence: A critical review of the literature and recommendations for researchers and practitioners. Frontiers in psychology, 10:1116. find
  38. OpenAI. 2023. Gpt-4 technical report.
  39. Samuel J Paech. 2023. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281.
  40. Rosalind W Picard. 2008. Toward machines with emotional intelligence. find
  41. Rosalind W. Picard, Elias Vyzas, and Jennifer Healey. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE transactions on pattern analysis and machine intelligence, 23(10):1175–1191. find
  42. Robert Plutchik. 1982. A psychoevolutionary theory of emotions. find
  43. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527– 536, Flore…
  44. Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Deepanway Ghosal, Rishabh Bhardwaj, Samson Yu Bai Jian, Pengfei Hong, Romila Ghosh, Abhinaba Roy, Niyati Chhaya, et al. 2021. Recognizing emotion cause in conversations. Cognitive Computation, 13:1317–1332. find
  45. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207. find
  46. Byron Reeves and Clifford Nass. 1996. The media equation: How people treat computers, television, and new media like real people. Cambridge, UK, 10(10). find
  47. Susan E Rivers, Isaac J Handley-Miner, John D Mayer, and David R Caruso. 2020. Emotional intelligence. find
  48. Sahand Sabour, Wen Zhang, Xiyao Xiao, Yuwei Zhang, Yinhe Zheng, Jiaxin Wen, Jialu Zhao, and Minlie Huang. 2022. Chatbots for mental health support: Exploring the impact of emohaa on reducing mental distress in china. arXiv preprint arXiv:2209.10183. — [arXiv:2209.10183](https://arxiv.org/abs/2209.10183) find
  49. Peter Salovey and John D Mayer. 1990. Emotional intelligence. Imagination, cognition and personality, 9(3):185–211. find
  50. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. find
  51. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–3035. find
  52. Dagmar Schuller and Björn W Schuller. 2018. The age of artificial emotional intelligence. Computer, 51(9):38–46. find
  53. Nicola S Schutte, John M Malouff, Chad Bobik, Tra-cie D Coston, Cyndy Greeson, Christina Jedlicka, Emily Rhodes, and Greta Wendorf. 2001. Emotional intelligence and interpersonal relations. Journal of social psychology, 141(4):523–536. find
  54. Nicola S Schutte, John M Malouff, Maureen Simunek, Jamie McKenley, and Sharon Hollander. 2002. Characteristic emotional intelligence and emotional well-being. Cognition & Emotion, 16(6):769–785. find
  55. Candace L Sidner. 2016. Engagement, emotions, and relationships: on building intelligent agents. In Emotions, Technology, Design, and Learning, pages 273– 294. Elsevier. find
  56. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  57. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399. find
  58. Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Jia Liu. 2023. Emotional intelligence of large language models. Journal of Pacific Rim Psychology, 17:18344909231213958. find
  59. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  60. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045.
  61. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. find
  62. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.
  63. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023a. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921.
  64. Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023b. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667.
  65. Lynn Waterhouse. 2006. Multiple intelligences, the mozart effect, and emotional intelligence: A critical review. Educational psychologist, 41(4):207. find
  66. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  67. Yige Xu, Zhiwei Zeng, and Zhiqi Shen. 2023. Efficient cross-task prompt tuning for few-shot conversational emotion recognition. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11654–11666. find
  68. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023a. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  69. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2023b. On the evaluations of chatgpt and emotion-enhanced prompting for mental health analysis. arXiv preprint arXiv:2304.03347. — [arXiv:2304.03347](https://arxiv.org/abs/2304.03347) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabour-2024-emobench-evaluating-emotional-intelligence-large/