Elamur Research⌕
☰
home / papers / 2026 / Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models

Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models

Paper · 2026 · Yifu Chen, Shengpeng Ji, Zhengqing Liu et al.

Summary

Full-duplex spoken dialogue models (SDMs) must decide both what to say and when to say it. Reinforcement learning could improve this, but current automatic metrics make poor rewards. Timing-oracle proxies, event counts and metric suites each capture only part of interaction quality, and human ratings are expensive and hard to scale. Even strong LLM judges such as GPT-4o and Gemini 2.5 Pro miss turn-taking violations. The authors define a formal taxonomy of dyadic interaction, covering turns, pauses, gaps, overlaps, backchannels and interruptions, along with typical failure modes. They then build annotated synthetic and real-world full-duplex datasets. On this data, an audio-capable LLM is trained in three stages: grounding in dual-track audio, distillation of chain-of-thought reasoning, and GRPO. The resulting generative reward model gives one overall score plus separate judgments of semantic coherence and turn-taking timing. It reaches state-of-the-art accuracy on synthetic, human–machine and human–human dialogues. Its reasoning also agrees closely with expert rationales, which makes it a candidate reward signal for RL training of SDMs.

Key points

  • The taxonomy splits errors into two axes:

    • Semantic errors: contextual incoherence and "interruption amnesia."
    • Timing errors: over-reactive cases (inappropriate barge-in, overly deferential ceding) and under-reactive cases (delayed turn transition, ignored interruption).

  • The taxonomy is grounded in public corpora plus 10 hours of in-house annotated human–machine dialogues from three SOTA systems.

  • In the training-stage ablation, the full pipeline reached these accuracies:

    SettingFull pipeline (SFT-1 + SFT-2 + GRPO with some real data)SFT-2 alone
    In-distribution98.54%64.96%
    Real-world human–human (RW-HH)86.79%45.28%
    Real-world human–machine (RW-HM)77.27%63.64%

    When GRPO used only synthetic data, the model overfit and RW-HM fell to 56.82%.

  • In a blind expert review of 40 snippets, the model's chain-of-thought had the highest consistency with ground-truth rationales (4.125 on a 5-point scale). Experts chose it as the best reasoning in 55% of cases.

  • Limitations:

    • The model has not yet been tested inside an online RL loop.
    • Its reward is binary, and the authors suggest finer-grained or separate semantic and timing reward channels as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Beijing University of TechnologyBeijing · China13 works · 1,910 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Yifu Chen, Shengpeng Ji, Zhengqing Liu, Qian Chen, Wen Wang, Ziqing Wang, Yangzhuo Li, Tianle Liang and Zhou Zhao. 2026. Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.6

@inproceedings{chen2026dual,
  title = {{Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models}},
  author = {Yifu Chen, Shengpeng Ji, Zhengqing Liu, Qian Chen, Wen Wang, Ziqing Wang, Yangzhuo Li, Tianle Liang and Zhou Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.6},
  url = {https://doi.org/10.18653/v1/2026.acl-long.6}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Praveen Chowdary, Joe Chuang, Antony D’Avirro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, and 65 others. 2… — [arXiv:2506.22554](https://arxiv.org/abs/2506.22554) find
  2. Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe. 2025. Talking turns: Benchmarking audio foundation models on turn-taking dynamics. arXiv preprint arXiv:2503.01174. — [arXiv:2503.01174](https://arxiv.org/abs/2503.01174) find
  3. Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. 2024. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems. find
  4. Chen Chen, Ke Hu, Chao-Han Huck Yang, Ankita Pasad, Edresson Casanova, Weiqing Wang, Szu-Wei Fu, Jason Li, Zhehuai Chen, Jagadeesh Balam, and Boris Ginsburg. 2025a. Reinforcement learning enhanced full-duplex spoken dialogue language models for conversational interactions. In Second Conference on La… find
  5. Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang, and Eng Siong Chng. 2025b. Audio large language models can be descriptive speech quality evaluators. Preprint, arXiv:2501.17202. — [arXiv:2501.17202](https://arxiv.org/abs/2501.17202) find
  6. Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, and 1 others. 2025c. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282. — [arXiv:2501.06282](https://arxiv.org/abs/2501.06282) find
  7. Yifu Chen, Shengpeng Ji, Haoxiao Wang, Ziqing Wang, Siyu Chen, Jinzheng He, Jin Xu, and Zhou Zhao. 2025d. WavRAG: Audio-integrated retrieval augmented generation for spoken dialogue models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
  8. Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung yi Lee, and Lijuan Wang. 2025. Audio-aware large language models as judges for speaking styles. Preprint, arXiv:2506.05984. find
  9. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  10. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037.
  11. Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, and 1 others. 2025. Kimi-audio technical report. arXiv preprint arXiv:2504.18425.
  12. Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. 2025. Llama-omni2: Llm-based real-time spoken chatbot with autoregressive streaming speech synthesis. arXiv preprint arXiv:2505.02625. — [arXiv:2505.02625](https://arxiv.org/abs/2505.02625) find
  13. Sarah E. Finch, James D. Finch, and Jinho D. Choi. 2023. Don’t forget your ABC’s: Evaluating the state-of-the-art in chat-oriented dialogue systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15044–15071, Toronto, Canada.… find
  14. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and 1 others. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  15. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  16. Zhaori Guo, Timothy J Norman, and Enrico H Gerding. 2024. Multi-trainer binary feedback interactive reinforcement learning. Annals of Mathematics and Artificial Intelligence, pages 1–26. find
  17. Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, and 1 others. 2025. Reinforcement learning from user feedback. arXiv preprint arXiv:2505.14946. — [arXiv:2505.14946](https://arxiv.org/abs/2505.14946) find
  18. Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, Xiaoda Yang, Zehan Wang, Qian Yang, Jian Li, Yidi Jiang, Jingzhen He, Yunfei Chu, Jin Xu, and Zhou Zhao. 2024a. Wavchat: A survey of spoken dialogue models. Preprint, arXiv:24… find
  19. Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, and 1 others. 2024b. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532. — [arXiv:2408.16532](https://arxiv.org/abs/2408.16532) find
  20. Shengpeng Ji, Tianle Liang, Yangzhuo Li, Jialong Zuo, Minghui Fang, Jinzheng He, Yifu Chen, Zhengqing Liu, Ziyue Jiang, Xize Cheng, and 1 others. 2025. Wavreward: Spoken dialogue models with generalist reward evaluators. arXiv preprint arXiv:2505.09558. — [arXiv:2505.09558](https://arxiv.org/abs/2505.09558) find
  21. LI Kai, FU Qiang, and YAN Yonghong. 2012. Speech enhancement using robust generalized sidelobe can-celler with multi-channel post-filtering in adverse environments. Chinese Journal of Electronics, 21(1):85–90. find
  22. Yangzhuo Li, Shengpeng Ji, Yifu Chen, Tianle Liang, Haorong Ying, Yule Wang, Junbo Li, Jun Fang, and Zhou Zhao. 2026. Wavbench: Benchmarking reasoning, colloquialism, and paralinguistics for end-to-end spoken dialogue models. arXiv preprint arXiv:2602.12135. — [arXiv:2602.12135](https://arxiv.org/abs/2602.12135) find
  23. Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. 2025. Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. arXiv preprint arXiv:2503.04721. — [arXiv:2503.04721](https://arxiv.org/abs/2503.04721) find
  24. Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. 2024. Align-slm: Textless spoken language models with reinforcement learning from ai feedback. arXiv preprint arXiv:2411.01834. find
  25. Jingyu Lu, Yuhan Wang, Fan Zhuo, Xize Cheng, Changhao Pan, Xueyi Pu, Yifu Chen, Chenyuhao Wen, Tianle Liang, and Zhou Zhao. 2026. Modeling and benchmarking spoken dialogue rewards with modality and colloquialness. Preprint, arXiv:2603.14889. — [arXiv:2603.14889](https://arxiv.org/abs/2603.14889) find
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  27. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  28. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. 2025. Salmonn-omni: A standalone speech llm without codec injection for full-duplex conversation. arXiv preprint arXiv:2505.17060. — [arXiv:2505.17060](https://arxiv.org/abs/2505.17060) find
  29. Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chao-hong Tan, Zhihao Du, and 1 others. 2024a. Omni-flatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410.17799. — [arXiv:2410.17799](https://arxiv.org/abs/2410.17799) find
  30. Yu Zhang, Changhao Pan, Wenxiang Guo, Ruiqi Li, Zhiyuan Zhu, Jialei Wang, Wenhao Xu, Jingyu Lu, Zhiqing Hong, Chuxin Wang, and 1 others. 2024b. Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37:1… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2026-dual-axis-generative-reward-model/