Elamur Research⌕
☰
home / papers / 2026 / Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion

Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion

Paper · 2026 · Li Xiao, Kotaro Funakoshi and Manabu Okumura

Summary

The paper addresses multimodal emotion recognition in conversations with several participants. It targets three problems: working out which visible person is actually speaking, the gap between strong text-based and weaker audio/visual emotion cues, and heavily skewed emotion label distributions. The authors build a pipeline with four parts:

  • LipSyncNet, a module that matches lip motion to the audio track to pick the true speaker's face and is trained jointly with the rest of the system instead of as a separate preprocessing step.
  • Graph-based teacher–student distillation, which uses the text branch to guide the audio and visual branches.
  • Hierarchical attention fusion, combining cross-modal attention with an adaptive gate.
  • A composite loss that adds polynomial and contrastive terms to the fusion objective.

On MELD and IEMOCAP, the model reports state-of-the-art weighted F1, with clear gains on rare emotion classes. The ablations show that speaker identification matters most. This is useful for any system that must analyse affect in multi-party video, such as TV dialogue or meetings.

Key points

  • Weighted F1 reaches 67.75% on MELD and 72.44% on IEMOCAP, averaged over 5 runs. On IEMOCAP this is about 1.2 points above MultiEMO.
  • The unimodal encoders are RoBERTa (text), Wav2Vec2.0 (audio) and TimeSformer (video), followed by modality-specific graph networks (GAT).
  • In the ablations, dropping speaker identification causes the largest loss on both datasets: without it, the model defaults to the first detected face. The fusion loss and distillation are the next most important components, while contrastive learning matters least.
  • The distillation and fusion stage has about 3M parameters and trains in 30–50 minutes. Code is released at github.com/llllxx1628/multimodalERC.
  • The paper notes several limitations:
    • Experiments cover only English data (TV series and lab recordings).
    • The benchmarks contain little overlapping speech, so heavy speaker overlap and occlusion are not handled.
    • The composite loss adds hyperparameters that need tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Institute of Science TokyoTokyo · Japan14 works · 20 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Li Xiao, Kotaro Funakoshi and Manabu Okumura. 2026. Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.212

@inproceedings{xiao2026emotion,
  title = {{Emotion Recognition in Multi-Speaker Conversations through Speaker Identification, Knowledge Distillation, and Hierarchical Fusion}},
  author = {Li Xiao, Kotaro Funakoshi and Manabu Okumura},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.212},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.212}
}

References

60 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The conversation: Deep audio-visual speech enhancement. arXiv preprint arXiv:1804.04121. find
  2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  3. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443. find
  4. Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML). find
  5. Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. Pyannote. audio: neural building blocks for speaker diarization. In ICASSP 2020-2020 IEEE International conference on acoustics,… find
  6. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359. find
  7. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, page 120. find
  8. J. S. Chung and A. Zisserman. 2016. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV. find
  9. Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer. find
  10. Najim Dehak, Patrick J Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet. 2010. Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 19(4):788–798. find
  11. Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. 2011. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern recognition, 44(3):572–587. find
  12. Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Kenji Nagamatsu, and Shinji Watanabe. 2019. End-to-end neural speaker diarization with permutation-free objectives. arXiv preprint arXiv:1909.05952. find
  13. Yusuke Fujita, Shinji Watanabe, Shota Horiguchi, Yawen Xue, and Kenji Nagamatsu. 2020. End-to-end neural diarization: Reformulating speaker diarization as simple multi-label classification. arXiv preprint arXiv:2003.02966. — [arXiv:2003.02966](https://arxiv.org/abs/2003.02966) find
  14. Daniel Garcia-Romero, David Snyder, Gregory Sell, Daniel Povey, and Alan McCree. 2017. Speaker diarization using deep neural network embeddings. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4930–4934. IEEE. find
  15. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv:1908.11540.
  16. Runfang Guo, Hongfei Guo, Liwen Wang, Mengmeng Chen, Dong Yang, and Bin Li. 2024. Development and application of emotion recognition technol-ogya systematic literature review. BMC psychology, 12(1):95. find
  17. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meet…
  18. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  19. Abhinav Joshi, Ashwani Bhat, Ayush Jain, Atin Singh, and Ashutosh Modi. 2022. COGMEN: COntextualized GNN based multimodal emotion recognitioN. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4148… find
  20. Jiawen Deng and Fuji Ren. 2021. A survey of textual emotion recognition and its challenges. IEEE Transactions on Affective Computing, 14(1):49–67. find
  21. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673. find
  22. Andrius Dzedzickis, Artūras Kaklauskas, and Vytautas Bucinskas. 2020. Human emotion recognition: Review of sensors and methods. Sensors, 20(3):592. find
  23. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
  24. Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, and Dragomir Anguelov. 2022. Polyloss: A polynomial expansion perspective of classification loss functions. Preprint, arXiv:2204.12511. — [arXiv:2204.12511](https://arxiv.org/abs/2204.12511) find
  25. Yuzhao Mao, Qi Sun, Guang Liu, Xiaojie Wang, Weiguo Gao, Xuan Li, and Jianping Shen. 2020. Dialoguetrm: Exploring the intra-and inter-modal emotional behaviors in the conversation. arXiv preprint arXiv:2010.07637. — [arXiv:2010.07637](https://arxiv.org/abs/2010.07637) find
  26. Dongyuan Li, Yusong Wang, Kotaro Funakoshi, and Manabu Okumura. 2023. Joyful: Joint modality fusion and graph contrastive learning for multimoda emotion recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16051–16069, Singapore. Association f… find
  27. Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems, 32. find
  28. Qiuchi Li, Dimitris Gkoumas, Alessandro Sordoni, Jian-Yun Nie, and Massimo Melucci. 2021a. Quantum-inspired neural network for conversational emotion recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13270–13278. find
  29. Zechao Li, Yanpeng Sun, Liyan Zhang, and Jinhui Tang. 2021b. Ctnet: Context-based tandem network for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9904–9917. find
  30. Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. arXiv preprint arXiv:1808.03920. find
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  32. Xin Lu, Yanyan Zhao, Yang Wu, Yijian Tian, Huipeng Chen, and Bing Qin. 2020. An iterative emotion interaction network for emotion recognition in conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4078–4088, Barcelona, Spain (Online). International… find
  33. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6818–6825. find
  34. Usman Malik, Julien Saunier, Kotaro Funakoshi, and Alexandre Pauchet. 2020. Who speaks next? turn change and next speaker prediction in multimodal multiparty interaction. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pages 349–354. IEEE. find
  35. Haposan Vincentius Manalu and Achmad Pratama Rifai. 2024. Detection of human emotions through facial expressions using hybrid convolutional neural network-recurrent neural network algorithm. Intelligent Systems with Applications, 21:200339. find
  36. Douglas OShaughnessy. 2025. Speaker diarization: A review of objectives and methods. Applied Sciences, 15(4):2002. find
  37. Patrícia Pereira, Helena Moniz, and Joao Paulo Carvalho. 2025. Deep emotion recognition in textual conversations: A survey. Artificial Intelligence Review, 58(1):1–37. find
  38. Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion, 37:98–125. find
  39. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508.
  40. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE access, 7:100943–100953. find
  41. Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha. 2022. Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions. Information Fusion, 81:203–239. find
  42. Raksha Ramakrishna, Anna Scaglione, Tong Wu, Nikhil Ravi, and Sean Peisert. 2023. Differential privacy for class-based data: A practical gaussian mechanism. IEEE Transactions on Information Forensics and Security, 18:5096–5108. find
  43. Anvita Saxena, Ashish Khanna, and Deepak Gupta. 2020. Emotion recognition and detection methods: A comprehensive survey. Journal of Artificial Intelligence and Systems, 2(1):53–79. find
  44. Gregory Sell, David Snyder, Alan McCree, Daniel Garcia-Romero, Jesús Villalba, Matthew Maciejewski, Vimal Manohar, Najim Dehak, Daniel Povey, Shinji Watanabe, and 1 others. 2018. Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge. In Interspe… find
  45. Tao Shi and Shao-Lun Huang. 2023. MultiEMO: An attention-based correlation-aware multimodal fusion framework for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14752–14766, Toronto, Canad… find
  46. Prachi Singh, Amrit Kaul, and Sriram Ganapathy. 2023. Supervised hierarchical clustering using graph neural networks for speaker diarization. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  47. David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5329–5333. IEEE. find
  48. Matteo Spezialetti, Giuseppe Placidi, and Silvia Rossi. 2020. Emotion recognition for human-robot interaction: Recent advances and future perspectives. Frontiers in Robotics and AI, 7:532279. find
  49. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558.
  50. Haidong Zhang and Yekun Chai. 2021. Coin: Conversational interactive networks for emotion recognition in conversation. In Proceedings of the Third Workshop on Multimodal Artificial Intelligence, pages 12–18. find
  51. Su-Fang Zhang, Jun-Hai Zhai, Bo-Jun Xie, Yan Zhan, and Xin Wang. 2019. Multimodal representation learning: Advances, trends and challenges. In 2019 International Conference on Machine Learning and Cybernetics (ICMLC), pages 1–6. IEEE. find
  52. Mingyi Zhao, Linrui Gong, and Abdul Sattar Din. 2025. A review of the emotion recognition model of robots. Applied Intelligence, 55(6):1–33. find
  53. Barbra Zupan and Michelle Eskritt. 2024. Facial and vocal emotion recognition in adolescence: a systematic review. Adolescent Research Review, 9(2):253– 277. find
  54. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, and 1 others. 2017. Graph attention networks. stat, 1050(20):10–48550. find
  55. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46–53. find
  56. Mike Wu and Noah Goodman. 2019. Multimodal generative models for compositional representation learning. arXiv preprint arXiv:1912.05075. — [arXiv:1912.05075](https://arxiv.org/abs/1912.05075) find
  57. Eman MG Younis, Someya Mohsen, Essam H Houssein, and Osman Ali Sadek Ibrahim. 2024. Machine learning for human emotion recognition: a comprehensive review. Neural Computing and Applications, 36(16):8901–8947. find
  58. Taeyang Yun, Hyunkuk Lim, Jeonghwan Lee, and Min Song. 2024. Telme: Teacher-leading multimodal fusion network for emotion recognition in conversation. arXiv preprint arXiv:2401.12987. find
  59. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250.
  60. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-emotion-recognition-multi-speaker-conversations/