Elamur Research⌕
☰
home / papers / 2026 / AEQ-Bench: Measuring Empathy of Omni-Modal Large Models

AEQ-Bench: Measuring Empathy of Omni-Modal Large Models

Paper · 2026 · Xuan Luo, Lewei Yao, Lanqing Hong et al.

Summary

Omni-modal large models (OLMs) increasingly talk to users through speech, yet evaluations of empathy usually turn audio into transcripts and ignore how something is said. AEQ-Bench (Audio Empathy Quotient Benchmark) addresses this gap. It tests two abilities: producing empathetic replies from a spoken utterance plus a short textual context, and rating the empathy of spoken replies directly, without transcription.

The benchmark draws on real human speech from MELD, GigaSpeech and EmoV-DB. It uses two parallel designs. In the first, one utterance is placed in different contexts. In the second, one context is paired with the same sentence spoken in different emotional tones.

State-of-the-art OLMs were scored on linguistic and paralinguistic dimensions against human annotators. Models trained to output audio generally beat text-only-output models, with GPT ranked most human-like and Qwen-Omni close behind. However, OLM judges only match humans on coarse quality ratings. They fail on fine-grained prosodic expressiveness, and generated speech tends to sound flat. This shows that audio-native evaluators are still needed.

Key points

  • AEQ-Bench has 1,885 English instances built from 1,020 audio clips and 1,558 generated contexts:
    • MELD: 281 clips × 3 contexts.
    • GigaSpeech: 303 clips × 2 contexts.
    • EmoV-DB: 436 clips in 4 tones (amused, disgusted, angry, neutral) over 109 contexts.
  • Instances fall into two types (985 chatting, 900 help-seeking) and span 6 topics.
  • In fine-grained 5-point judging of paralinguistic empathy, Qwen3-Omni roughly follows the human ranking of systems. Its agreement with human judgements is only 43%.
  • Replacing the audio with text captions of it does not work as a proxy:
    • Plain captions skew overly positive.
    • Captions prompted for objectivity skew negative.
  • The models show distinct styles:
    • GPT-4o gives structured answers with smooth delivery.
    • Qwen2.5-Omni sounds reflective and often opens with "well".
    • Qwen3-Omni is occasionally expressive but unstable.
    • Baichuan's accent fluctuates.
  • Limitations include English-only coverage and coarse Good/Fair/Poor ratings. Human annotators were paid 16 USD/hour.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🏢Huawei TechnologiesShenzhen · China75 works · 12,975 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

Cite

Cite as Xuan Luo, Lewei Yao, Lanqing Hong, Kai Chen, Dehua Tao, Daxin Tan, Yukun Deng, Ruifeng Xu and Jing Li. 2026. AEQ-Bench: Measuring Empathy of Omni-Modal Large Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1813

@inproceedings{luo2026aeq,
  title = {{AEQ-Bench: Measuring Empathy of Omni-Modal Large Models}},
  author = {Xuan Luo, Lewei Yao, Lanqing Hong, Kai Chen, Dehua Tao, Daxin Tan, Yukun Deng, Ruifeng Xu and Jing Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1813},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1813}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit. 2018. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514. — [arXiv:1806.09514](https://arxiv.org/abs/1806.09514) find
  2. Lisa Aziz-Zadeh, Tong Sheng, and Anahita Gheytanchi. 2010. Common premotor regions for the perception and production of prosody and correlations with empathy and prosodic ability. PloS one, 5(1):e8759. find
  3. Brant R Burleson and Adrianne Kunkel. 2009. Revisiting the different cultures thesis: An assessment of sex differences and similarities in supportive communication. In Sex differences and similarities in communication, pages 133–154. Routledge. find
  4. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  5. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, and 1 others. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In 22nd Annual Conference of the International Speech Communication… find
  6. Shun Chen, Faith Liao, David Murphy, and Stephen Joseph. 2023. Development and validation of a 12-item version of the barrett-lennard relationship inventory (bl ri: mini) using item response theory. Current Psychology, 42(13):10566–10580. find
  7. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. CoRR.
  8. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  9. Mark H Davis. 1983. Measuring individual differences in empathy: evidence for a multidimensional approach. Journal of personality and social psychology, 44(1):113. find
  10. Steven B. Davis and Paul Mermelstein. 1990. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences, page 65–74. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. find
  11. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, and 1 others. 2024. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. find
  12. Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. 2025. Llama-omni2: Llm-based real-time spoken chatbot with autoregressive streaming speech synthesis. arXiv preprint arXiv:2505.02625. — [arXiv:2505.02625](https://arxiv.org/abs/2505.02625) find
  13. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576, Mexico City, M… find
  14. Sreyan Ghosh and Ramani Duraiswami. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In TTIC Summer Workshop on Foundations of Speech and Audio Foundation Models 2025. find
  15. Yuan Gong, Jin Yu, and James Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE. find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  17. William Ickes, ANN Buysse, HAO Pham, Kerri Rivers, James R Erickson, Melanie Hancock, Joli Kelleher, and Paul R Gesn. 2000. On the difficulty of distinguishing “good” and “poor” perceivers: A social relations analysis of empathic accuracy data. Personal Relationships, 7(2):219–234. find
  18. Katharina Kühne, Martin H Fischer, and Yuefang Zhou. 2020. The human takes it all: Humanlike synthesized voices are perceived as less eerie and more likable. evidence from a subjective ratings study. Frontiers in neurorobotics, 14:593732. find
  19. Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, and 1 others. 2025. Baichuan-omni-1.5 technical report. CoRR.
  20. Kate Loveys, Mark Sagar, Xueyuan Zhang, Gregory Fricchione, Elizabeth Broadbent, and 1 others. 2021. Effects of emotional expressiveness of a female digital human on loneliness, stress, perceived support, and closeness across genders: randomized controlled trial. Journal of medical Internet research… find
  21. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527– 536, Flore…
  22. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12):nwae403.
  23. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 5370–5381. find
  24. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  25. Carl R Rogers. 1957. The necessary and sufficient conditions of therapeutic personality change. Journal of consulting psychology, 21(2):95. find
  26. Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276.
  27. Alberta B Szalita. 1976. Some thoughts on empathy: The eighteenth annual frieda fromm-reichmann memorial lecture. Psychiatry, 39(2):142–152. find
  28. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations.
  29. George L. Trager. 1958. Paralanguage: A first approximation. Studies in Linguistics, 13:1–12. find
  30. Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, and 1 others. 2025a. Sentient agent as a judge: Evaluating higher-order social cognition in large language models. arXiv preprint arXiv:2505.02847.
  31. Han Zhang, Zixiang Meng, Meng Luo, Hong Han, Lizi Liao, Erik Cambria, and Hao Fei. 2025b. Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark. In Proceedings of the ACM on Web Conference 2025, pages 2872–2881. find
  32. Yiman Zhang, Ziheng Luo, Qiangyu Yan, Wei He, Borui Jiang, Xinghao Chen, and Kai Han. 2025c. Omnieval: A benchmark for evaluating omni-modal models with visual, auditory, and textual inputs. arXiv preprint arXiv:2506.20960. — [arXiv:2506.20960](https://arxiv.org/abs/2506.20960) find
  33. Mahesh Viswanathan and Madhubalan Viswanathan. 2005. Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinion score (mos) scale. Computer speech & language, 19(1):55–83. find
  34. Haoyu Wang, Guangyan Zhang, Jiale Chen, Jingyu Li, Yuehai Wang, and Yiwen Guo. 2025. Emotion omni: Enabling empathetic speech response generation through large language models. arXiv preprint arXiv:2508.18655. — [arXiv:2508.18655](https://arxiv.org/abs/2508.18655) find
  35. Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE.
  36. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025a. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  37. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. Preprint, arXiv:2509.17765.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-aeq-bench-measuring-empathy-omni/