Elamur Research⌕
☰
home / papers / 2026 / Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS

Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS

Paper · 2026 · Yue Zhao, Hongyan Li, Yong Chen et al.

Summary

Cascaded voice assistants often stream LLM output straight into TTS. Emotional speech synthesis needs its emotion label before the first audio chunk, but emotion recognition (ERC) only works once the full reply exists. Self-EmoQ fixes this ordering problem. It is a plug-and-play module, built on a pretrained LLM, that chooses the agent's own emotion before any text is produced. That choice then conditions both a fixed response generator and a streaming Emo-TTS system.

The authors frame dialogue as an utterance-level MDP: the conversation context is the state and the system's emotion is the action. They train the planner in a DQN style, deriving Q-values from averaged token log-probabilities. The reward mixes two parts: agreement with dataset labels, and a Plutchik-based score assigned by GPT-4o.

On DailyDialog, EmoryNLP, IEMOCAP and MELD, Self-EmoQ beats prompting, fine-tuning and tabular Q-learning baselines on emotion choice and response quality. A full real-time pipeline shows the approach can be deployed.

Key points

  • Reward design: a weighted sum of a binary imitation term (1−w) and a Plutchik score (w).
    • GPT-4o produces the Plutchik score by averaging three ratings: how well the chosen emotion matches the context, how believable the shift from the prior emotion is, and whether the emotion fits the utterance's communicative function.
    • Human annotation of a subset correlated strongly with the GPT-4o scores.
  • Training: the planner learns through Bellman-equation bootstrapping, so it optimizes long-term emotional return rather than assigning emotions turn by turn.
  • Baselines: 0-shot, CoCT, ECoT, MP, PS, SFT, FSM (tabular Q-learning) and EMDP.
  • Speech evaluation: BERT-based scores and PESQ compare the post-hoc ERC setting (B) against emotion-first planning (C).
  • Limitations: relying on an LLM judge adds compute cost and possible bias, and the emotion set is limited to Plutchik's discrete categories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GeelyHangzhou · China2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Yue Zhao, Hongyan Li, Yong Chen and Luo Ji. 2026. Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.740

@inproceedings{zhao2026self,
  title = {{Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS}},
  author = {Yue Zhao, Hongyan Li, Yong Chen and Luo Ji},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.740},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.740}
}

References

50 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Chandrakant Bothe, Sven Magg, Cornelius Weber, and Stefan Wermter. 2017. Dialogue-based neural learning to estimate the sentiment of a next upcoming utterance. In International conference on artificial neural networks, pages 477–485. Springer. find
  3. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  4. Yuyan Chen and Yanghua Xiao. 2024. Recent advancement of emotion cognition in large language models. CoRR. find
  5. Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, and Seong-Whan Lee. 2025. Emosphere++: Emotion-controllable zero-shot text-to-speech via emotion-adaptive spherical vector. IEEE Transactions on Affective Computing. find
  6. Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, and 1 others. 2025. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589. — [arXiv:2505.17589](https://arxiv.org/abs/2505.17589) find
  7. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, and 1 others. 2024. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117.
  8. Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. 2025a. Emo-dpo: Controllable emotional speech synthesis through direct preference optimization. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  9. Xiaoxue Gao, Huayun Zhang, and Nancy F Chen. 2025b. Prompt-unseen-emotion: Zero-shot expressive speech synthesis with prompt-llm contextual knowledge for mixed emotions. arXiv preprint arXiv:2506.02742. — [arXiv:2506.02742](https://arxiv.org/abs/2506.02742) find
  10. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation…
  11. Qingqing Gu, Dan Wang, Yue Zhao, Xiaoyu Wang, Zhonglin Jiang, Yong Chen, and Luo Ji. 2026. Chain-of-conceptual-thought elicits daily conversation in large language models. In PRICAI 2025: Trends in Artificial Intelligence, pages 353–369, Singapore. Springer Nature Singapore. find
  12. Jing Han, Zixing Zhang, Zhao Ren, and Björn Schuller. 2021. Exploring perception uncertainty for emotion recognition in dyadic conversation and music listening. Cognitive Computation, 13(2):231–240. find
  13. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. find
  14. Xincheng Ju, Dong Zhang, Suyang Zhu, Junhui Li, Shoushan Li, and Guodong Zhou. 2023. Real-time emotion pre-recognition in conversations with contrastive multi-modal dialogue pre-training. In Proceedings of the 32nd ACM international conference on information and knowledge management, pages 1045–1055… find
  15. Naoyuki Kanda, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Hemin Yang, Zirun Zhu, Min Tang, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, and 1 others. 2024. Making flow-matching-based zero-shot text-to-speech laugh as you like. arXiv preprint arXiv:2402.07383. — [arXiv:2402.07383](https://arxiv.org/abs/2402.07383) find
  16. Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation. In Proceedings of the 62nd Annual Meeting of the Association fo… find
  17. Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, Runqi Qiao, and Sirui Wang. 2023. Instructerc: Reforming emotion recognition in conversation with multi-task retrieval-augmented large language models. arXiv preprint arXiv:2309.11911. — [arXiv:2309.11911](https://arxiv.org/abs/2309.11911) find
  18. Yi Lei, Shan Yang, Xinsheng Wang, and Lei Xie. 2022. Msemotts: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:853–864. find
  19. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
  20. Runnan Li, Zhiyong Wu, Jia Jia, Jingbei Li, Wei Chen, and Helen Meng. 2018. Inferring user emotive state changes in realistic human-computer conversational dialogs. In Proceedings of the 26th ACM international conference on Multimedia, pages 136–144. find
  21. ITU-T Recommendation. 2001. Perceptual evaluation of speech quality (pesq): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs. Rec. ITU-T P. 862. find
  22. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
  23. Takaaki Saeki, Soumi Maiti, Shinnosuke Takamichi, Shinji Watanabe, and Hiroshi Saruwatari. 2024. Speechbertscore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics. arXiv preprint arXiv:2401.16812. — [arXiv:2401.16812](https://arxiv.org/abs/2401.16812) find
  24. Zaijing Li, Gongwei Chen, Rui Shao, Yuquan Xie, Dongmei Jiang, and Liqiang Nie. 2024. Enhancing emotional generation capability of large language models via emotional chain-of-thought. Preprint, arXiv:2401.06836. — [arXiv:2401.06836](https://arxiv.org/abs/2401.06836) find
  25. Shixiong Liang, Ruohua Zhou, and Qingsheng Yuan. 2025. Ece-tts: A zero-shot emotion text-to-speech model with simplified and precise control. Applied Sciences, 15(9):5108. find
  26. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  27. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6818–6825. find
  28. Zhiyu Shen, Yunhe Pang, Yanghui Rao, and Jianxing Yu. 2025. Coe: A clue of emotion framework for emotion recognition in conversations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23548–23563. find
  29. Haoxiang Shi, Ziqi Liang, and Jun Yu. 2024. Emotional cues extraction and fusion for multi-modal emotion prediction and recognition in conversation. arXiv preprint arXiv:2408.04547. — [arXiv:2408.04547](https://arxiv.org/abs/2408.04547) find
  30. Xiao Sun, Jiamin Wang, Fuji Ren, and Meng Wang. 2023. Dynamic emotional transition sampling and emotional guidance of individuals based on conversation. IEEE Transactions on Computational Social Systems, 11(1):1192–1204. find
  31. Xiao Sun, Chen Zhang, and Lian Li. 2019. Dynamic emotion modelling and anomaly detection in conversation based on emotional transition tensor. Information Fusion, 46:11–22. find
  32. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, and 1 others. 2015. Human-level control through deep reinforcement learning. nature, 518(7540):529–533. find
  33. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  35. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece. European Language Resources Association (ELRA).
  36. Robert Plutchik. 1982. A psychoevolutionary theory of emotions. find
  37. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 8…
  38. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.
  39. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  40. Tianrui Wang, Haoyu Wang, Meng Ge, Cheng Gong, Chunyu Qiang, Ziyang Ma, Zikang Huang, Guanrou Yang, Xiaobao Wang, Eng Siong Chng, and 1 others. 2025a. Word-level emotional expression control in zero-shot text-to-speech synthesis. arXiv preprint arXiv:2509.24629. — [arXiv:2509.24629](https://arxiv.org/abs/2509.24629) find
  41. Xiaoyu Wang, Yue Zhao, Qingqing Gu, Zhonglin Jiang, Yong Chen, and Luo Ji. 2025b. Convert language model into a value-based strategic planner. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 1444–1456, Vienna, Austria. Asso… find
  42. Yuqing Wang and Yun Zhao. 2024. Metacognitive prompting improves understanding in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1914–1926, Mexico… find
  43. Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker, Daniel Tompkins, Chung-Hsien Tsai, Canrun Li, Zhen Xiao, Sheng Zhao, Jinyu Li, and 1 others. 2024. Laugh now cry later: Controlling time-varying emotional states of flow-matching-based zero-shot text-to-speech. In 2024 IEEE Spoken Languag… find
  44. Yunhe Xie, Yifan Liu, Chengjie Sun, and Shaobo Li. 2025. Pseudo-utterance-guided contrastive network for emotion forecasting in conversations. Expert Systems with Applications, 279:127382. find
  45. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. arXiv preprint arXiv:2304.03347. find
  46. Qing Yang, Zhenghao Liu, Junxin Wang, Yangfan Du, Pengcheng Huang, and Tong Xiao. 2025. Rlaif-spa: Optimizing llm-based emotional speech synthesis via rlaif. arXiv preprint arXiv:2510.14628. — [arXiv:2510.14628](https://arxiv.org/abs/2510.14628) find
  47. Sung-Lin Yeh, Yun-Shao Lin, and Chi-Chun Lee. 2020. A dialogical emotion decoder for speech emotion recognition in spoken dialog. In ICASSP 2020-2020 IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 6479–6483. IEEE. find
  48. Sayyed M Zahiri and Jinho D Choi. 2018. Emotion detection on tv show transcripts with sequence-based convolutional neural networks. In AAAI Workshops, volume 18, pages 44–52. find
  49. Rui Zhang, Zhenyu Wang, Zhenhua Huang, Li Li, and Mengdan Zheng. 2021. Predicting emotion reactions for human–computer conversation: A variational approach. IEEE Transactions on Human-Machine Systems, 51(4):279–287. find
  50. Yue Zhao, Qingqing Gu, Xiaoyu Wang, Teng Chen, Zhonglin Jiang, Yong Chen, and Luo Ji. 2025. Fis-miness: A finite state machine based paradigm for emotional support conversations. Preprint, arXiv:2504.11837. — [arXiv:2504.11837](https://arxiv.org/abs/2504.11837) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-self-emoq-plutchik-guided-value/