Elamur Research⌕
☰
home / papers / 2026 / Retrospective Speech Recognition for Spoken Dialogue System: Exploiting Subsequent Utterances to Enhance ASR Performance

Retrospective Speech Recognition for Spoken Dialogue System: Exploiting Subsequent Utterances to Enhance ASR Performance

Paper · 2026 · Ryu Takeda and Kazunori Komatani

Summary

Most ASR in spoken dialogue systems recognizes each user turn using only earlier context, so a misrecognition is never revisited. Takeda and Komatani introduce Retrospective Speech Recognition (RSR), which re-decodes a past user utterance once later turns are available. It can also draw on system utterances, whose text is known exactly. The pipeline has two stages:

  • A syllable-based recognizer (S-ASR) converts each user turn into a syllabogram sequence.
  • System turns are mapped to syllables with a pronunciation dictionary or grapheme-to-phoneme conversion.
  • Syllable strings from a window of preceding and following turns are joined with a separator symbol "S". A syllable-to-character translation (SCT) model decodes them, and the target turn is cut out of the output.

Working at the pronunciation level lets different kinds of context be merged. It also sidesteps spelling mismatches in proper nouns. On real Japanese dialogue speech, adding later context lowered character error rate by 0.10 points. Because RSR can run in parallel with ordinary sequential ASR, it gives dialogue systems a practical route to self-correction.

Key points

  • RSR is framed as estimating an utterance's characters from a truncated window of a preceding and b following utterances. Standard sequential ASR is a special case of this formulation, with no future context.
  • S-ASR was trained on paired audio and text from 10 Japanese corpora, totalling over 12,500 hours of audio.
  • SCT was trained on over 400 million characters. The text comes from the same corpora plus unpaired text from BCCWJ and Wiki40b-ja.
  • SCT robustness to S-ASR errors is improved with data augmentation from an S-ASR error simulator. Both forward and backward inference variants of SCT are described.
  • The authors identify two main limitations: computational cost and dialogue-management design. Proposed future work includes N-best-aware SCT, knowledge distillation, and confidence-based selection of the utterances that need RSR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech & spoken language · 480 works

Cite

Cite as Ryu Takeda and Kazunori Komatani. 2026. Retrospective Speech Recognition for Spoken Dialogue System: Exploiting Subsequent Utterances to Enhance ASR Performance. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{takeda2026retrospective,
  title = {{Retrospective Speech Recognition for Spoken Dialogue System: Exploiting Subsequent Utterances to Enhance ASR Performance}},
  author = {Ryu Takeda and Kazunori Komatani},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.20/}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maximilian Bisani and Hermann Ney. 2004. Bootstrap estimates for confidence intervals in ASR performance evaluation. In Proc. of ICASSP, pages 409– 409. find
  2. Maximilian Bisani and Hermann Ney. 2008. Joint-sequence models for grapheme-to-phoneme conversion. Speech Commun., 50(5):434–451. find
  3. Robert Flynn and Anton Ragni. 2023. Leveraging cross-utterance context for asr decoding. In Proc. of Interspeech, pages 1359–1363. find
  4. Xun Gong, Yu Wu, Jinyu Li, Shujie Liu, Rui Zhao, Xie Chen, and Yanmin Qian. 2023. LongFNT: Long-form speech recognition with factorized neural transducer. In Proc. of ICASSP, pages 1–5. find
  5. Xun Gong, Yu Wu, Jinyu Li, Shujie Liu, Rui Zhao, Xie Chen, and Yanmin Qian. 2024. Advanced long-content speech recognition with factorized neural transducer. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:1803–1815. find
  6. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. Int. J. Comput. Vision, 129(6):1789–1819. find
  7. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proc. of LREC, pages 2431–2439. find
  8. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In Proc. of ICLR. find
  9. Kikuo Maekawa. 2003. Corpus of spontaneous Japanese: Its design and evaluation. In Proc. of ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition. find
  10. Ryo Masumura, Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, and Shota Orihashi. 2021. Hierarchical transformer-based large-context end-to-end asr with large-context knowledge distillation. In Proc. of ICASSP, pages 5879–5883. find
  11. Toshinobu Ogiso, Mamoru Komachi, Yasuharu Den, and Yuji Matsumoto. 2012. UniDic for early middle Japanese: a dictionary for morphological analysis of classical Japanese. In Proc. of LREC, pages 911–915. find
  12. Daniel Povey, Arnab Ghoshal, et al. 2011. The kaldi speech recognition toolkit. In Proc. of SLT. find
  13. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.
  14. Alec Radford, Jong Wook Kim, et al. 2023. Robust speech recognition via large-scale weak supervision. In Proc. of ICML.
  15. Yukiya Hono, Koh Mitsuda, et al. Rinna/Nue-ASR. find
  16. Yukiya Hono, Koh Mitsuda, et al. 2024. Integrating pre-trained speech and language models for end-to-end speech recognition. In Proc. of Findings of ACL, pages 13289–13305. find
  17. Takaaki Hori, Niko Moritz, Chiori Hori, and Jonathan Le Roux. 2020. Transformer-based long-context end-to-end speech recognition. In Proc. of Interspeech, pages 5011–5015. find
  18. Maekawa Kikuo, Makoto Yamazaki, et al. 2014. Balanced corpus of contemporary written Japanese. Language Resources and Evaluation, (48):345–371. find
  19. Kazunori Komatani, Ryu Takeda, and Shogo Okada. 2023. Analyzing differences in subjective annotations by participants and third-party annotators in multimodal dialogue corpus. In Proc. of SIGDIAL, pages 104–113. find
  20. Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In Proc. of EMNLP, pages 230–237.
  21. Seongmin Lee, Kohki Tamura, Tomoaki Nakamura, and Naoki Yoshinaga. 2024a. Can noisy cross-utterance contexts help speech-recognition error correction? In Proc. of IWSDS. find
  22. Wonjun Lee, San Kim, and Gary Geunbae Lee. 2024b. Enhancing dialogue speech recognition with robust contextual awareness via noise representation learning. In Proc. of SIGDIAL, pages 333–343, Kyoto, Japan. Association for Computational Linguistics. find
  23. Colin Raffel, Noam Shazeer, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1).
  24. Toshinori Sato, Taiichi Hashimoto, and Manabu Okumura. 2017. Implementation of a word segmentation dictionary called mecab-ipadic-NEologd and study on how to use it effectively for information retrieval (in Japanese). In Proc. of Annual Meeting of the Association for NLP, pages NLP2017–B6–1. find
  25. David Snyder, Guoguo Chen, and Daniel Povey. 2015. MUSAN: A Music, Speech, and Noise Corpus. ArXiv:1510.08484v1. — [arXiv:1510.08484](https://arxiv.org/abs/1510.08484) find
  26. G. Sun, C. Zhang, and P. C. Woodland. 2020. Cross-utterance language models with acoustic error sampling. Preprint, arXiv:2009.01008. — [arXiv:2009.01008](https://arxiv.org/abs/2009.01008) find
  27. Shinnosuke Takamichi, Kentaro Mitsui, et al. 2019. JVS corpus: free Japanese multi-speaker voice corpus. Preprint, arXiv:1908.06248. — [arXiv:1908.06248](https://arxiv.org/abs/1908.06248) find
  28. Ryu Takeda and Kazunori Komatani. 2025. Reducing orthographic dependency on paired data by probabilistic integration via syllabogram for japanese dialogue speech recognition. In Proc. of APSIPA ASC, pages 549–554. find
  29. Shinji Watanabe, Takaaki Hori, Shigeki Karita, et al. 2018. ESPnet: End-to-end speech processing toolkit. In Proc. of Interspeech, pages 2207–2211. find
  30. Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth. 2019. Transformer based grapheme-to-phoneme conversion. In Proc. of Interspeech, pages 2095–2099. *Table 6: CER (↓) in % for Hazumi corpus* Condition Reazon v2 Open Whisper large-v3 ASR Nue model ESPnet (CSJ recipe) find
  31. Character-ASR (Utt.-wise) S-ASR + SCT Ours Utt.-wise: (a, b) = (0, 0) Sequential: (a, b) = (9, 0) RSR: (a, b) = (9, 9) CER 13.06 16.41 22.29 22.46 12.20 *Figure 10: CER (↓) vs. window length under sequential condition: a = 0 or b = 0.* 11.57 11.12 11.03 No. of characters 228,242 exploit preceding ut… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/takeda-2026-retrospective-speech-recognition-spoken-dialogue/