Elamur Research⌕
☰
home / papers / 2026 / Evaluation of Paralinguistic-Aware Spoken Dialogue Systems using Next-Utterance Classification

Evaluation of Paralinguistic-Aware Spoken Dialogue Systems using Next-Utterance Classification

Summary

Typical spoken dialogue systems (SDSs) chain ASR, an LLM and speech synthesis, so they see only transcribed words. They miss cues such as intonation and prosody, which can change what an utterance means and lead to dialogue breakdowns. Adding a paralinguistic recognizer could help, but measuring one module's contribution through user studies is expensive and hard to isolate. The authors propose an offline, corpus-based alternative built on next-utterance classification. Given a transcribed dialogue history from human–human speech data, the system's own LLM ranks candidate dialogue acts (DAs) for the next turn, and performance is scored with mean reciprocal rank (MRR). Two conditions are compared: text only, and text plus attitude labels from an acoustic attitude recognition model. Using the Japanese MapTask corpus, adding attitude information raised MRR. A pairwise human preference study showed that the corpus-based scores track subjective judgments. The method therefore offers a scalable proxy for human evaluation when developing affective and multimodal dialogue systems.

Key points

  • Data: the experiments draw on Chiba University's Japanese Map Task Dialogue Corpus (MapTask), which contains 128 sessions, 64 participants, about 23 hours of speech and 53,109 manually transcribed inter-pausal units.
  • DA taxonomy: the scheme is adapted from Araki et al. (1999). After dropping the opening/closing and hold classes and merging ambiguous pairs, 11 classes remain. Prediction quality is measured by MRR over the LLM's top-5 ranked DAs.
  • Attitude recognizer: the paralinguistic model comes from Miyazawa et al. (2025). It uses a HuBERT backbone with a light fully connected head and assigns one of four attitude classes from acoustics alone. The labels are applied only to turn-final utterances.
  • Human study design: 2,106 utterances were sampled, 702 for each of three outcome types: only the agnostic system correct, both incorrect, and only the aware system correct. Thirty-four non-expert raters judged 120 response pairs each on a 1–4 scale, for 4,080 trials.
  • Human study results: responses from the system that predicted the correct DA were preferred. The paralinguistic-aware system was also rated somewhat better when both systems were wrong. Extending the method to chat-oriented dialogue and richer paralinguistic categories is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.50

@inproceedings{miyazawa2026evaluation,
  title = {{Evaluation of Paralinguistic-Aware Spoken Dialogue Systems using Next-Utterance Classification}},
  author = {Miyazawa, Kouki and Sato, Yoshinao},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.50},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.50}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2026. System card: Claude Opus 4.6. find
  2. Masahiro Araki, Toshihiko Itoh, Tomoko Kumagai, and Masato Ishizaki. 1999. Proposal of a standard utterance-unit tagging scheme (in japanese). Journal of the Japanese Society for Artificial Intelligence, 14(2):251–260. find
  3. Harry Bunt, Jan Alexandersson, Jae-Woong Choe, Alex Chengyu Fang, Koiti Hasida, Volha Petukhova, Andrei Popescu-Belis, and David Traum. 2012. ISO 24617-2: A semantically-based standard for dialogue annotation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation… find our card
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other… find
  5. Maurice Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. find
  6. Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, pages 2012–2021, Osaka, Japan. find our card
  7. Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-Yi Lee, and Ivan Bulyko. 2024. Paralinguistics-enhanced large language modeling of spoken dialogue. In Proceedings of the IEEE International Conference on Acoustics, Speech a… find
  8. Kouki Miyazawa, Zhi Zhu, and Yoshinao Sato. 2025. Paralinguistic attitude recognition for spoken dialogue systems. In Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, pages 137–142, Bilbao, Spain. Association for Computational Linguistics. find
  9. Dietrich Rebholz-Schuhmann, Antonio José Jimeno Yepes, Erik M. van Mulligen, Ning Kang, Jan Kors, David Milward, Peter Corbett, Ekaterina Buyko, Katrin Tomanek, Elena Beisswanger, and Udo Hahn. 2010. The CALBC silver standard corpus for biomedical named entities — a study in harmonizing the contribu… find
  10. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025… find our card
  11. The Japanese MapTask Corpus Project. 2007. Chiba university japanese map task dialogue corpus (Map-Task). find
  12. Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Mengzhe Chen, Qian Chen, and Lei Xie. 2024. E-Chat: Emotion-sensitive spoken dialogue system with large language models. In Proceedings of the International Symposium on Chinese Spoken Language Processing, pages 586–590. find
  13. Yueyi Yang, Haotian Liu, Fang Kang, Mengqi Zhang, Zheng Lian, Hao Tang, and Haoyu Chen. 2026. SayNext-Bench: Why do LLMs struggle with next-utterance prediction? Preprint, arXiv:2602.00327. — [arXiv:2602.00327](https://arxiv.org/abs/2602.00327) find
  14. Zhixian Zhao, Shuiyuan Wang, Guojian Li, Hongfei Xue, Chengyou Wang, Shuai Wang, Longshuai Xiao, Zihan Zhang, Hui Bu, Xin Xu, Xinsheng Wang, Hexin Liu, Eng Siong Chng, Hung yi Lee, and Lei Xie. 2026. find
  15. The ICASSP 2026 HumDial challenge: Benchmarking human-like spoken dialogue systems in the LLM era. Preprint, arXiv:2601.05564. — [arXiv:2601.05564](https://arxiv.org/abs/2601.05564) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/miyazawa-2026-evaluation-paralinguistic-aware-spoken-dialogue/