Elamur Research⌕
☰
home / papers / 2026 / RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis

RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis

Paper · 2026 · Enzhi Wang, Jiaming Zhou, Yuhang Jia et al.

Summary

Speech LLMs such as GPT-4o can now handle spoken interaction end to end. However, it is unclear how well they help users finish concrete tasks when speech is spontaneous, spans many turns, varies across speakers, or alternates with typed input. Existing task-oriented dialogue (TOD) benchmarks are mostly text-only, and the only large speech TOD set, SpokenWOZ, is in English and has no disfluency or speaker annotations. The authors fill this gap with RealTalk-CN, a Chinese speech–text TOD corpus. It consists of real human-to-human recordings, partly scripted and partly improvised, captured on both professional microphones and smartphones. The recordings are annotated with intents, slot values, disfluency types, and speaker attributes such as age, gender, and region. The authors also define a cross-modal chat task in which users switch between speech and text across turns. An accompanying evaluation protocol covers disfluency robustness, speaker sensitivity, and cross-domain generalization. Experiments with current speech LLMs show that the benchmark is hard. They also reveal fairness gaps for older and accented speakers. The resource matters for building reliable Chinese voice assistants.

Key points

  • RealTalk-CN contains about 5.4k dialogues (roughly 60K turns, ~150 hours). Its 113 speakers were chosen from a pool of 300 volunteers with a near-even gender split, covering all major regions of China. Data collection and annotation cost approximately $35,000.
  • About 10% of the data deliberately keeps spontaneous disfluencies, including repetitions, hesitations, self-corrections, and modal particle drag.
  • Multi-domain dialogues mainly reduce intent classification accuracy, while slot filling stays relatively stable. Slot filling, however, drops significantly for older speakers and dialect-heavy regions (p < 0.05). GPT-4-Audio-mini shows clear declines on these groups.
  • Repetition and Modal Particle Drag cause the largest performance losses, and GPT-4o-Audio is the most robust model across disfluency types. Substituting text into the dialogue history helps models in the cross-modal chat task.
  • The dataset is available at huggingface.co/datasets/BAAI/RealTalk-CN under a CC BY-NC license with gated access.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nankai UniversityTianjin · China24 works · 762 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksDialogue systems · 624 works

Cite

Cite as Enzhi Wang, Jiaming Zhou, Yuhang Jia, Aobo Kong, Qicheng Li and Yong Qin. 2026. RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.131

@inproceedings{wang2026realtalk,
  title = {{RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis}},
  author = {Enzhi Wang, Jiaming Zhou, Yuhang Jia, Aobo Kong, Qicheng Li and Yong Qin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.131},
  url = {https://doi.org/10.18653/v1/2026.acl-long.131}
}

References

52 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Junyi Ao and 1 others. 2024. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. In Advances in Neural Information Processing Systems, volume 37, pages 56898–56918. find
  3. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205.
  4. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language…
  5. Yucheng Cai, Wentao Ma, Yuchuan Wu, Shuzheng Si, Yuan Shao, Zhijian Ou, and Yongbin Li. 2024. UniPCM: Universal pre-trained conversation model with task-aware automatic prompt. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (… find
  6. Ronald Carter. 1995. Spoken grammar: what is it and how can we teach it? michael mccarthy and ronald carter. ELT journal, 49:3. find
  7. Yang Chen, Hui Wang, Shiyao Wang, Junyang Chen, Jiabei He, Jiaming Zhou, Xi Yang, Yequan Wang, Yonghua Lin, and Yong Qin. 2025. Seniortalk: A chinese conversation dataset with rich annotations for super-aged seniors. arXiv preprint arXiv:2503.16578. — [arXiv:2503.16578](https://arxiv.org/abs/2503.16578) find
  8. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T Tan, and Haizhou Li. 2024. Voicebench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196.
  9. Xize Cheng and 1 others. 2025. Voxdialogue: Can spoken dialogue systems understand information beyond words? In The Thirteenth International Conference on Learning Representations. find
  10. Anna Seo Gyeong Choi and Hoon Choi. 2025. Fairness of automatic speech recognition: Looking through a philosophical lens. arXiv preprint arXiv:2508.07143. — [arXiv:2508.07143](https://arxiv.org/abs/2508.07143) find
  11. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  12. Changwoo Chun, Daniel Rim, and Juhee Park. 2025. Llm contextbridge: A hybrid approach for intent and dialogue understanding in ivsr. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 794–806. find
  13. Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. routledge. find
  14. Wenjie Dong, Sirong Chen, and Yan Yang. 2025. Pro-TOD: Proactive task-oriented dialogue system based on large language model. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9147–9164, Abu Dhabi, UAE. Association for Computational Linguistics. find
  15. Mihail Eric and Christopher D Manning. 2017. Key-value retrieval networks for task-oriented dialogue. arXiv preprint arXiv:1705.05414.
  16. Jean E. Fox Tree. 1995. The effects of false starts and repetitions on the processing of subsequent words in spontaneous speech. Journal of Memory and Language, 34(6):709–738. find
  17. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), pages 263–272. find
  18. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991–63010…
  19. Paria Jamshid Lou, Peter Anderson, and Mark Johnson. 2020. End-to-end speech recognition and disfluency removal. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2051–2061. find
  20. Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2022. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference on Learning Representations. find
  21. Bahjat Kawar, Gregory Vaksman, and Michael Elad. 2021. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757–21769. find
  22. Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Karthik Gopalakrishnan, Behnam Hedayatnia, and Dilek Hakkani-Tür. 2021. “how robust ru?”: Evaluating task-oriented dialogue systems on spoken conversations. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1147–… find
  23. Jean C Krause and Louis D Braida. 2004. Acoustic properties of naturally produced clear speech at normal speaking rates. The Journal of the Acoustical Society of America, 115(1):362–378. find
  24. William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47(260):583– 621. find
  25. Ajinkya Kulkarni, Anna Tokareva, Rameez Qureshi, and Miguel Couceiro. 2024. The balancing act: Unmasking and alleviating ASR biases in Portuguese. In Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, pages 31–40, St. Julian’s, Malta. Association for Comput… find
  26. Kathleen C Fraser, Jed A Meltzer, and Frank Rudzicz. 2015. Linguistic features identify alzheimer’s disease in narrative speech. Journal of Alzheimer’s disease, 49(2):407–422. find
  27. Linqin Li, Dong Zhang, Suyang Zhu, Shoushan Li, and Guodong Zhou. 2024a. Response generation in multi-modal dialogues with split pre-generation and cross-modal contrasting. Information Processing & Management, 61(1):103581. find
  28. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. find
  29. Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, and 1 others. 2025. Baichuan-audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17239.
  30. Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, and 1 others. 2024b. Baichuan-omni technical report. arXiv preprint arXiv:2410.08565, 3(7).
  31. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  32. Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024c. Panda (pedantic answer-correctness determination and adjudication): Improving automatic evaluation for question answering and text generation. CoRR, abs/2402.11161. — [arXiv:2402.11161](https://arxiv.org/abs/2402.11161) find
  33. Ivan Sekulic, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. 2024. Reliable LLM-based user simulator for task-oriented dialogue systems. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat… find
  34. Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. 2024. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786. find
  35. Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung yi Lee. 2025. Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. CoRR, abs/2503.04721. — [arXiv:2503.04721](https://arxiv.org/abs/2503.04721) find
  36. Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, and Minlie Huang. 2021. Robustness testing of language understanding in task-oriented dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 2467–2480… find
  37. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  38. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60. find
  39. Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2023. Disfluency generation for more robust dialogue systems. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11479– 11488. find
  40. Yao Qian, Ximo Bianv, Yu Shi, Naoyuki Kanda, Leo Shen, Zhen Xiao, and Michael Zeng. 2021. Speech-language pre-training for end-to-end spoken language understanding. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7458–7462. IEEE. find
  41. Jun Quan, Shian Zhang, Qian Cao, Zizhong Li, and Deyi Xiong. 2020. RiSAWOZ: A large-scale multi-domain Wizard-of-Oz dataset with rich semantic annotations for task-oriented dialogue modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 930… find
  42. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. find
  43. E Shriberg. 1994. Preliminaries to a theory of speech disfluency. PhD Diss, Univ. of California. find
  44. Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. 2023. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. Advances in Neural Information Processing Systems, 36:39088–39118. find
  45. Student. 1908. The probable error of a mean. Biometrika, pages 1–25. find
  46. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215.
  47. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and 1 others. 2024. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729.
  48. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  49. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages…
  50. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612.
  51. Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. CrossWOZ: A large-scale Chinese cross-domain task-oriented dialogue dataset. Transactions of the Association for Computational Linguistics, 8:281–295.
  52. Su Zhu, Zijian Zhao, Tiejun Zhao, Chengqing Zong, and Kai Yu. 2019. Catslu: The 1st chinese audio-textual spoken language understanding challenge. In 2019 International Conference on Multimodal Interaction, pages 521–525. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-realtalk-cn-realistic-chinese-speech/