Elamur Research⌕
☰
home / papers / 2026 / Who Generates More Empathetic Responses–-Humans or LLMs? A Comparative Evaluation with Human and LLM Judges

Who Generates More Empathetic Responses–-Humans or LLMs? A Comparative Evaluation with Human and LLM Judges

Summary

The study asks whether people or large language models write more empathetic replies in everyday emotional conversations. Earlier work often relied on automatic metrics, focused on negative or healthcare-related emotions, and used within-subjects designs that are prone to order and carry-over effects. The authors took 2,000 single-exchange dialogues from EmpatheticDialogues, covering 32 positive and negative emotions. The original human replies served as a baseline. Four LLMs (GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, Mixtral-8×7B-Instruct) produced their own replies from a prompt that defined empathy. In a between-subjects design, 1,000 Prolific crowdworkers rated the replies, with each response source assigned to its own group of 200 raters. GPT-4o-mini then scored the same replies as an LLM judge. Both human raters and the LLM judge consistently rated the model replies as more empathetic than the human-written ones. Each type of judge also tended to favour responses from its own kind. The work matters because it provides a scalable evaluation framework that new models can be added to. It also cautions that empathy ratings from either type of judge carry systematic biases.

Key points

  • Scale of the study: 1,000 participants (200 per response source) and 2,000 prompts drawn roughly evenly from the 32 emotions of EmpatheticDialogues. Only the first two turns of each dialogue were used, together with the emotion label and situation description.
  • Generation prompt: The prompt described cognitive, affective and compassionate empathy. It asked for replies averaging about 28 words, with a 97-word maximum.
  • Dual-judge evaluation: The study collected 10,000 human ratings and 10,000 ratings from GPT-4o-mini. Both sets of judges preferred LLM replies across emotions, and each showed a self-favouring tendency.
  • Acknowledged limitations: The setup is single-turn and decontextualised, and it relies on only one LLM judge. Empathy is collapsed into a single overall score. LLM replies were longer, and the authors did not control for response length.
  • Planned release: The LLM responses and all ratings are to be published under CC BY-NC 4.0 so that newer models can be compared without rerunning the whole study.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.21

@inproceedings{welivita2026who,
  title = {{Who Generates More Empathetic Responses–-Humans or LLMs? A Comparative Evaluation with Human and LLM Judges}},
  author = {Welivita, Anuradha and Zeitoun, Fawzia and Pu, Pearl},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.21},
  url = {https://doi.org/10.18653/v1/2026.conll-main.21}
}

References

51 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Syeda Nahida Akter, Zichun Yu, Aashiq Muhamed, Tianyue Ou, Alex Bäuerle, Ángel Alexander Cabrera, Krish Dholakia, Chenyan Xiong, and Graham Neubig. 2023. An in-depth look at gemini’s language abilities. arXiv preprint arXiv:2312.11444. — [arXiv:2312.11444](https://arxiv.org/abs/2312.11444) find
  2. John W Ayers, Adam Poliak, Mark Dredze, Eric C Leas, Zechariah Zhu, Jessica B Kelley, Dennis J Faix, Aaron M Goodman, Christopher A Longhurst, Michael Hogarth, et al. 2023. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. J… find
  3. Ahmed Belkhir and Fatiha Sadat. 2023. Beyond information: Is chatgpt empathetic enough? In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 159–169. find
  4. Liliana L Bove. 2019. Empathy for service: benefits, unintended consequences, and future research agenda. Journal of Services Marketing, 33(1):31–43. find
  5. Gary Charness, Uri Gneezy, and Michael A Kuhn. 2012. Experimental methods: Between-subject and within-subject design. Journal of economic behavior & organization, 81(1):1–8. find
  6. Angelina Chen, Oliver Hannon, Sarah Koegel, and Raffaele Ciriello. 2024. Feels like empathy: How “emotional” ai challenges human essence. In Australasian Conference on Information Systems. find
  7. Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023. Llm-empowered chatbots for psychiatrist and patient simulation: Application and evaluation. arXiv preprint arXiv:2305.13614. — [arXiv:2305.13614](https://arxiv.org/abs/2305.13614) find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Jacob Cohen. 1992. Quantitative methods in psychology: A power primer. Psychol. Bull., 112:1155– 1159. find
  10. Mark H Davis. 1980. Interpersonal reactivity index. find
  11. Jean Decety, Claus Lamm, et al. 2006. Human empathy through the lens of social neuroscience. The scientific World journal, 6:1146–1163. find
  12. David R Heise. 1970. The semantic differential and attitude research. Attitude measurement, 4:235–253. find
  13. Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher. 2025. Training language models to be warm and empathetic makes them less reliable and more sycophantic. arXiv preprint arXiv:2507.21919. — [arXiv:2507.21919](https://arxiv.org/abs/2507.21919) find
  14. Flora Ioannidou and Vaya Konstantikaki. 2008. Empathy and emotional intelligence: What is it really about? International Journal of caring sciences, 1(3):118. find
  15. Richard D Lane, Donald M Quinlan, Gary E Schwartz, Pamela A Walker, and Sharon B Zeitlin. 1990. The levels of emotional awareness scale: A cognitive-developmental measure of emotion. Journal of personality assessment, 55(1-2):124–134. find
  16. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Computational Linguistics: ACL 2023, pages 431–469, Toronto, Canad… find our card
  17. Emma J Lawrence, Philip Shaw, Dawn Baker, Simon Baron-Cohen, and Anthony S David. 2004. Measuring empathy: reliability and validity of the empathy quotient. Psychological medicine, 34(5):911–920. find
  18. Yoon Kyung Lee, Jina Suh, Hongli Zhan, Junyi Jessy Li, and Desmond C Ong. 2024. Large language models produce responses perceived to be empathic. arXiv preprint arXiv:2403.18148. — [arXiv:2403.18148](https://arxiv.org/abs/2403.18148) find
  19. Igor Douven. 2018. A bayesian perspective on likert scales and central tendency. Psychonomic bulletin & review, 25:1203–1211. find
  20. Young-Jun Lee, Chae-Gyun Lim, and Ho-Jin Choi. 2022. Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 669–… find
  21. Paul Ekman. 2004. Emotions revealed. Bmj, 328(Suppl S5). find
  22. Zohar Elyoseph, Dorit Hadar-Shoval, Kfir Asraf, and Maya Lvovsky. 2023. Chatgpt outperforms humans in emotional awareness evaluations. Frontiers in Psychology, 14:1199058. find
  23. Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Statistical power analyses using g* power 3.1: Tests for correlation and regression analyses. Behavior research methods, 41(4):1149– 1160. find
  24. Yahui Fu, Koji Inoue, Chenhui Chu, and Tatsuya Kawahara. 2023. Reasoning before responding: Integrating commonsense-based causality explanation for empathetic response generation. arXiv preprint arXiv:2308.00085. find
  25. Jennifer Goetz, Sara Kiesler, and Aaron Powers. 2003. Matching robot appearance and behavior to tasks to improve human-robot cooperation. In The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. find
  26. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Nat… find
  27. Siru Liu, Allison B McCoy, Aileen P Wright, Babatunde Carew, Julian Z Genkins, Sean S Huang, Josh F Peterson, Bryan Steitz, and Adam Wright. 2023. Leveraging large language models for generating responses to patient messages. medRxiv, pages 2023–07. find
  28. Yuping Liu-Thompkins, Shintaro Okazaki, and Hairong Li. 2022. Artificial empathy in marketing interactions: Bridging the human-ai gap in affective and social customer experience. Journal of the Academy of Marketing Science, 50(6):1198–1218. find
  29. Siyuan Brandon Loh and Aravind Sesagiri Raamkumar. 2023. Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support. arXiv preprint arXiv:2310.08017. — [arXiv:2310.08017](https://arxiv.org/abs/2310.08017) find
  30. Lars Malmqvist. 2025. Sycophancy in large language models: Causes and mitigations. In Intelligent Computing-Proceedings of the Computing Conference, pages 61–74. Springer. find
  31. Mary L McHugh. 2013. The chi-square test of independence. Biochemia medica, 23(2):143–149. find
  32. MistralAI. 2024. Mixtral of experts. find
  33. Amanda K Montoya. 2023. Selecting a within-or between-subject design for mediation: Validity, causality, and statistical power. Multivariate Behavioral Research, 58(3):616–636. find
  34. Sundar Pichai. 2023. Introducing gemini: Our largest and most capable ai model. find
  35. Philip A Powell and Jennifer Roberts. 2017. Situational determinants of cognitive, affective, and compassionate empathy in naturalistic digital interactions. Computers in Human Behavior, 68:137–148. find
  36. Yushan Qian, Wei-Nan Zhang, and Ting Liu. 2023. Harnessing the power of large language models for empathetic response generation: Empirical investigations and improvements. arXiv preprint arXiv:2310.05140. find
  37. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Com… find
  38. Kristina Schaaff, Caroline Reinig, and Tim Schlippe. 2023. Exploring chatgpt’s empathic abilities. arXiv preprint arXiv:2308.03527. — [arXiv:2308.03527](https://arxiv.org/abs/2308.03527) find
  39. Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276, Online. Association for… find our card
  40. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. find our card
  41. John J Shaughnessy, Eugene B Zechmeister, and Jeanne S Zechmeister. 2000. Research methods in psychology. McGraw-Hill. find
  42. R Nathan Spreng, Margaret C McKinnon, Raymond A Mar, and Brian Levine. 2009. The toronto empathy questionnaire: Scale development and initial validation of a factor-analytic solution to multiple empathy measures. Journal of personality assessment, 91(1):62–71. find
  43. Steven J Stroessner and Jonathan Benitez. 2019. The social perception of humanoid and non-humanoid robots: Effects of gendered and machinelike features. International Journal of Social Robotics, 11:305– 315. find
  44. Ekaterina Svikhnushina and Pearl Pu. 2022. Peace: A model of key social and emotional qualities of conversational chatbots. ACM Transactions on Interactive Intelligent Systems, 12(4):1–29. find
  45. Ekaterina Svikhnushina and Pearl Pu. 2023. Approximating online human evaluation of social chatbots with prompting. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 268–281, Prague, Czechia. Association for Computational Linguistics. find
  46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  47. Anuradha Welivita and Pearl Pu. 2020. A taxonomy of empathetic response intents in human social conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4886– 4899, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  48. Henry J Xie, Jinghan Zhang, Xinhao Zhang, and Kunpeng Liu. 2024. Scoring with large language models: A study on measuring empathy of responses in dialogues. arXiv preprint arXiv:2412.20264. — [arXiv:2412.20264](https://arxiv.org/abs/2412.20264) find
  49. Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023. Is chatgpt equipped with emotional dialogue capabilities? arXiv preprint arXiv:2304.09582. — [arXiv:2304.09582](https://arxiv.org/abs/2304.09582) find
  50. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. find
  51. Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core competency: Survey on evaluation of large language models. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Fr… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/welivita-2026-who-generates-more-empathetic-responses/