Elamur Research⌕
☰
home / papers / 2023 / EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models

EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models

Paper · 2023 · Samuel J. Paech

Summary

Large language models talk with people constantly, yet standard benchmarks rarely test whether they grasp emotions in social situations. Existing human EI tests are a poor fit because they rely on self-assessment, images, paid assessors or expert scoring. EQ-Bench addresses this by measuring emotional understanding (EU) through an objectively scorable task. Models read a short GPT-4-generated dialogue built around conflict or tension and rate how strongly a character feels each of four emotions on a 0–10 scale. The design builds on SECEU but changes it in four ways:

  • the author sets the reference answers instead of averaging crowd responses;
  • dialogues replace plain scenario descriptions;
  • ratings no longer have to sum to 10;
  • the four emotions are deliberately diverse rather than all plausible.

With only 60 English questions, the benchmark yields repeatable scores, separates models across a wide capability range and tracks broad-intelligence benchmarks very closely. This suggests EU could serve as a cheap proxy for general LLM ability.

Key points

  • EQ-Bench correlates with MMLU at r=0.97 and with HellaSwag at r=0.91, but only weakly with SECEU.
  • The test set has 60 English questions, each asking for intensity ratings of four emotions after a GPT-4-written dialogue.
  • Scores spread widely across models without bunching at the top: coefficient of variation 0.741, IQR 53.81, skewness 0.13.
  • A full run takes under 10 minutes for OpenAI models and roughly 20–60 minutes for open models on one Nvidia RTX A6000.
  • An open-source pipeline is available on GitHub (EQ-bench/EQ-Bench), with a public leaderboard at eqbench.com.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSentiment & emotion · 1,035 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Samuel J. Paech. 2023. EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models. arXiv:2312.06281.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.06281 · arXiv 2312.06281

@misc{paech2023eq,
  title = {{EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models}},
  author = {Samuel J. Paech},
  year = {2023},
  doi = {10.48550/arxiv.2312.06281},
  eprint = {2312.06281},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.06281}
}

References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 . Dettmers, T., Lewis, M., Shleifer, S., & Zettlemoyer, L. (2021). 8-bit optimizers via bl…
  2. Goleman, D. (1996). Emotional intelligence. why it can matter more than iq. Learning, 24 (6), 49–50. find
  3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 . Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., . . . Steinhardt, J. (2021). Measuring mathema…
  4. Karcioglu, O., Topacoglu, H., Dikme, O., & Dikme, O. (2018). A systematic review of the pain scales in adults: which to use? The American journal of emergency medicine, 36 (4), 707–714. find
  5. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35 , 22199–22213. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., . . . Hashimoto, T. B. (2023). Alpacaeval: An…
  6. HuggingFace. Lin, S., Hilton, J., & Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 . LMSYS. (2023). Chatbot arena leaderboard. Retrieved from https://huggingface.co/spaces/ lmsys/chatbot-arena-leaderboard (Accessed: 2023-12-06) Mayer, J., &… ↗
  7. McRae, K., Rhee, S. H., Gatt, J. M., Godinez, D., Williams, L. M., & Gross, J. J. (2017). Genetic and environmental influences on emotion regulation: A twin study of cognitive reappraisal and expressive suppression. Emotion, 17 (5), 772. find
  8. Ogurlu, U. (2021). A meta-analytic review of emotional intelligence in gifted individuals: A multilevel analysis. Personality and Individual Differences, 171 , 110503. find
  9. OpenAI. (2023). Gpt-4 technical report.
  10. Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, cognition and personality, 9 (3), 185–211. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., . . . others (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 . T…
  11. Wang, X., Li, X., Yin, Z., Wu, Y., & Liu, J. (2023). Emotional intelligence of large language models. Journal of Pacific Rim Psychology, 17 , 18344909231213958. find
  12. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L. (2023). The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 , 9 , 1.
  13. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 . Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). Hellaswag: Can a machine really… find
  14. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., . . . others (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685 .

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paech-2023-eq-bench-emotional-intelligence-benchmark/