Elamur Research⌕
☰
home / papers / 2020 / End-to-end speech-to-dialog-act recognition

End-to-end speech-to-dialog-act recognition

Paper · 2020 · Viet-Trung Dang, Tianyu Zhao, Sei Ueno et al.

Summary

Recognizing dialog acts (DAs) from speech usually works as a pipeline. An ASR system writes a transcript, and a separate classifier, trained on clean human transcripts, labels it. Recognition mistakes then spread downstream, and acoustic cues such as pauses are discarded. The authors replace this chain with a single network. An attention-based acoustic-to-word ASR model hands its decoder output vector from just before the softmax layer (the "ASR feature") to an LSTM-based DA classifier. The classifier encodes words and then a history of utterances. They train this coupling step by step and then fine-tune it end to end with a weighted multi-task loss. A hybrid variant adds ordinary word embeddings of the hypothesis to the ASR feature. A final extension adds a DA-boundary token to the ASR vocabulary, so the recognizer segments utterances while transcribing. Experiments on Switchboard with SwDA tags show that the unified approach beats the pipeline. With n-best beam search, DA accuracy rises by 2.41 points absolute. This makes a case for passing soft internal representations between stages instead of committed outputs.

Key points

  • Data: Switchboard audio aligned with the Switchboard Dialog Act corpus (SwDA, built on Penn Treebank 3 annotations). It has 1,126 conversations, 213.5k utterances, a 27.2k-word vocabulary and 43 DA tags. Of these, 20 conversations are held out for test and 40 for validation.
  • Pipeline baseline: the classifier reaches an error rate of 26.79% (history 1) and 25.06% (history 5) on reference transcripts. On ASR output with 27.76% WER, the error rate degrades to 31.85% and 31.10%.
  • Retraining the classifier on ASR transcripts did not reliably help. Feeding the ASR feature did, and beam search and n-best hypotheses helped more in that setting.
  • End-to-end fine-tuning gave the best 1-best results, which were statistically significant. Its benefit disappeared with beam search, because the joint loss only sees the 1-best hypothesis. The hybrid input added a small further gain.
  • Segmentation is folded into recognition by appending an utterance-end symbol after every DA in the training targets. The DA tag is then predicted at each such symbol.

Interesting ideas

  • I1. Hand the downstream module the upstream model's pre-softmax vector rather than its chosen symbol, so a wrong pick still carries a representation near the right one.
  • I2. Make boundary detection part of the vocabulary by emitting a special end-of-unit token, so segmenting and labelling happen in one pass using the same evidence.
  • I3. Sum the embedding of the committed hypothesis with the soft feature, because the discrete form is sharper when upstream is right and the soft form is safer when it is wrong.
  • I4. When fine-tuning jointly, keep the downstream loss weight small so the upstream model's accuracy is not sacrificed for the downstream task.
  • I5. Compare three training regimes for the downstream model: gold input, noisy upstream output, and upstream latent features. In this paper, naive training on noisy output was not enough.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): the parser could read the tagger's internal tag/lemma vectors alongside its decided tag, while still logging the discrete decision for explainability.
  • I2 (tokens): sentence and clause splitting in crate en could be emitted as a boundary token by the tagger instead of running as a separate rule pass.
  • I3 (frames): speech-act classification in prag could combine the chosen UD tree with the parser's uncertainty signal, falling back on the soft signal where the arcs are shaky.
  • I4 (learn): when gparse is tuned with a frame-level objective, a low weight on the frame loss would protect LAS; parsing scores should be checked before and after.
  • I5 (learn): evaluate frame and answer modules on gold trees, silver/witness-parser trees and soft parser features, to see where robustness actually comes from.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kyoto UniversityKyoto · Japan27 works · 8,049 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech & spoken language · 233 worksSpeech act theory · 160 works

Cite

Cite as Viet-Trung Dang, Tianyu Zhao, Sei Ueno, Hirofumi Inaguma and Tatsuya Kawahara. 2020. End-to-end speech-to-dialog-act recognition. arXiv:2004.11419.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.11419 · arXiv 2004.11419

@misc{dang2020end,
  title = {{End-to-end speech-to-dialog-act recognition}},
  author = {Viet-Trung Dang, Tianyu Zhao, Sei Ueno, Hirofumi Inaguma and Tatsuya Kawahara},
  year = {2020},
  doi = {10.48550/arxiv.2004.11419},
  eprint = {2004.11419},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.11419}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C. V. Ess-Dykema, and M. Meteer, “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Computational linguistics, vol. 26, no. 3, pp. 339–373, 2000. ↗
  2. K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct acoustics-to-word models for english conversational speech recognition,” Proc. Interspeech 2017, pp. 959–963, 2017.
  3. S. Ueno, H. Inaguma, M. Mimura, and T. Kawahara, “Acoustic-to-word attention-based model complemented with character-level ctc-based model,” in Proc. IEEE-ICASSP, 2018. find
  4. J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end continuous speech recognition using attention-based recurrent nn: First results,” in NIPS 2014 Workshop on Deep Learning, December 2014, 2014. ↗
  5. W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 4960–4964. find
  6. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 4945– 4949. find
  7. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Advances in neural information processing systems, 2015, pp. 577– 585. ↗
  8. S. Joty and E. Hoque, “Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured models,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, 2016, pp. 1746–1756.
  9. S.-s. Shen and H.-Y. Lee, “Neural attention models for sequence classification: Analysis and application to key term extraction and dialogue act detection,” Interspeech 2016, pp. 2716–2720, 2016. find
  10. J. Ang, Y. Liu, and E. Shriberg, “Automatic dialog act segmentation and classification in multiparty meetings,” in Acoustics, Speech, and Signal Processing, 2005. Proceedings.(ICASSP’05). IEEE International Conference on, vol. 1. IEEE, 2005, pp. I– 1061. find
  11. S. Quarteroni, A. V. Ivanov, and G. Riccardi, “Simultaneous dialog act segmentation and classification from human-human spoken conversations,” in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5596–5599. find
  12. M. Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields,” in Tenth Annual Conference of the International Speech Communication Association, 2009. find
  13. T. Zhao and T. Kawahara, “Joint learning of dialog act segmentation and recognition in spoken dialog using neural networks,” in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), vol. 1, 2017, pp. 704–712.
  14. E. Shriberg, A. Stolcke, D. Jurafsky, N. Coccaro, M. Meteer, R. Bates, P. Taylor, K. Ries, R. Martin, and C. Van Ess-Dykema, “Can prosody aid the automatic classification of dialog acts in conversational speech?” Language and speech, vol. 41, no. 3-4, pp. 443–492, 1998. ↗
  15. H. Arsikere, A. Sen, A. Prathosh, and V. Tyagi, “Novel acoustic features for automatic dialog-act tagging,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 6105–6109. find
  16. D. Hakkani-Tür, F. Béchet, G. Riccardi, and G. Tur, “Beyond asr 1-best: Using word confusion networks in spoken language understanding,” Computer Speech & Language, vol. 20, no. 4, pp. 495–514, 2006. ↗
  17. R. Schumann and P. Angkititrakul, “Incorporating asr errors with attention-based, jointly trained rnn for intent detection and slot filling,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 6059– 6063. find
  18. Y. Chen, R. Price, and S. Bangalore, “Spoken language understanding without speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 6189–6193. find
  19. D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5754–5758. find
  20. S. Bhosale, I. Sheikh, S. H. Dumpala, and S. K. Kopparapu, “End-to-end spoken language understanding: Bootstrapping in low resource scenarios,” Proc. Interspeech 2019, pp. 1188–1192, 2019. find
  21. P. Haghani, A. Narayanan, M. Bacchiani, G. Chuang, N. Gaur, P. Moreno, R. Prabhavalkar, Z. Qu, and A. Waters, “From audio to semantics: Approaches to end-to-end spoken language understanding,” in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 720–726. find
  22. L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech model pre-training for end-to-end spoken language understanding,” arXiv preprint arXiv:1904.03670, 2019. ↗
  23. A. Caubrière, N. Tomashenko, A. Laurent, E. Morin, N. Camelin, and Y. Estève, “Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability,” arXiv preprint arXiv:1906.07601, 2019. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dang-2020-end-end-speech-dialog-act/