Elamur Research⌕
☰
home / papers / 2026 / Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation

Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation

Paper · 2026 · Eunseon Seong, Harim Lee, Dahye Kim et al.

Summary

The paper addresses multimodal Emotion Recognition in Conversation (ERC) and targets two weaknesses of earlier systems. First, they fuse modalities as if each carried the same kind and amount of information. Second, they treat emotion labels as unrelated categories, so confusing happy with excited costs as much as confusing happy with sad. The proposed framework, EMART, uses only text and audio. It splits RoBERTa-base in half. The lower six layers encode the dialogue text, with speaker and target-utterance markers. The upper six layers add cross-modal attention, so the text stream queries audio features from WavLM or Wav2vec. An alignment loss reduces the gap between the two modalities before fusion. In addition, emotions are placed at angles on Russell's circumplex model, and a supervised contrastive loss pushes classes apart in proportion to their angular distance. On IEMOCAP and MELD, this text-centric fusion beats concatenation, dot-product and plain attention fusion, and both losses add gains. The results suggest that modality-specific roles and psychologically grounded label geometry improve ERC.

Key points

  • With RoBERTa, EMART reaches about 70.8 Acc / 70.9 F1 on IEMOCAP and 66.7 Acc / 64.7 F1 on MELD. The best alternative fusion scored about 66.4 on IEMOCAP and 63.9 on MELD.
  • Unimodal baselines show why text is treated as primary. Audio alone gets 53.85 Acc on IEMOCAP and 48.97 on MELD, while RoBERTa text alone gets about 64.7 on both.
  • In the loss ablation, IEMOCAP Acc rises from 64.70 with no extra losses to 68.02 with the alignment loss only and 68.21 with the emotion-wheel loss only. Using both gives 70.79.
  • Starting fusion at layer 6 works best. Fusing earlier (layers 0 or 3) disrupts text representations, and fusing at layer 9 leaves too few layers for interaction.
  • The emotion-wheel loss outperforms SupCon and SCCL. Limitations noted by the authors are that visual input is not used and that results depend on the choice of audio encoder. Code: github.com/DILAB-HYU/EMART.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hanyang UniversitySeoul · South Korea6 works · 150 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 worksSpeech & spoken language · 480 works

Cite

Cite as Eunseon Seong, Harim Lee, Dahye Kim, Changhyun Kim and Dong-Kyu Chae. 2026. Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1875

@inproceedings{seong2026emotion,
  title = {{Emotion-Wheel-Guided Audio-Referred Text Representation for Multimodal Emotion Recognition in Conversation}},
  author = {Eunseon Seong, Harim Lee, Dahye Kim, Changhyun Kim and Dong-Kyu Chae},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1875},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1875}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  2. Feiyu Chen, Jie Shao, Shuyuan Zhu, and Heng Tao Shen. 2023. Multivariate, multi-frequency and multimodal: Rethinking graph neural networks for emotion recognition in conversation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10761–10770. find
  3. Michal Ephratt. 2011. Linguistic, paralinguistic and extralinguistic speech and silence. Journal of pragmatics, 43(9):2286–2307. find
  4. Lisa Feldman Barrett and James A Russell. 1998. Independence and bipolarity in the structure of current affect. Journal of personality and social psychology, 74(4):967. find
  5. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation…
  6. Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. 2022. Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7037–7041. find
  7. Jingwen Hu, Yuchen Liu, Jinming Zhao, and Qin Jin. 2021. MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… find
  8. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in neural information processing systems, pages 18661–18673. find
  9. Bryan Li, Dimitrios Dimitriadis, and Andreas Stolcke. 2019. Acoustic and lexical sentiment analysis for customer service calls. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5876–5880. find
  10. Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, and Zitao Liu. 2021a. Ctal: Pre-training cross-modal transformer for audio-and-language representations. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 3966–3977. find
  11. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021b. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in neural information processing systems, pages 9694–9705. find
  12. Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan…
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  14. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: an attentive rnn for emotion detection in conversations. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence. find
  15. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019a. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.
  16. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019b. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE access, 7:100943–100953. find
  17. Jonathan Posner, James A Russell, and Bradley S Peterson. 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(3):715–734. find
  18. James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161. find
  19. Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian MüLler, and Shrikanth Narayanan. 2013. Paralinguistics in speech and language—state-of-the-art and the challenge. Computer Speech & Language, 27(1):4–39. find
  20. Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji-Rong Wen, and Rui Yan. 2022. MISC: A mixed strategy-aware model integrating COMET for emotional support conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 308–319.
  21. Kailai Yang, Tianlin Zhang, Hassan Alhuzali, and Sophia Ananiadou. 2023. Cluster-level contrastive learning for emotion recognition in conversations. IEEE Transactions on Affective Computing, 14(4):3269–3280. find
  22. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pages 12310– 12320. find
  23. Tao Zhang and Zhenhua Tan. 2025. ECERC: Evidence-cause attention network for multi-modal emotion recognition in conversation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2064–2077. find
  24. Botao Zhao, Xiaoyang Qu, Zuheng Kang, Junqing Peng, Jing Xiao, and Jianzong Wang. 2025. Accon: Angle-compensated contrastive regularizer for deep regression. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22750–22758. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seong-2026-emotion-wheel-guided-audio-referred/