Elamur Research⌕
☰
home / papers / 2013 / Sub-lexical Dialogue Act Classification in a Spoken Dialogue System Support for the Elderly with Cognitive Disabilities

Sub-lexical Dialogue Act Classification in a Spoken Dialogue System Support for the Elderly with Cognitive Disabilities

Paper · 2013 · Ken Sadohara, Hiroaki Kojima, Takuya Narita et al.

Summary

The paper tackles dialogue act (DA) classification for a conversational assistant that gives schedule, date and time information to elderly people with mild dementia. Lexical cues such as word n-grams are known to work well for DA tagging. Getting them from casual elderly speech, though, requires costly and often speaker-specific language modeling, and the resulting transcripts must still be normalized. The authors avoid this step entirely. Their classifier works directly on phoneme strings produced by a phoneme recognizer. Classification is done by an SVM whose kernel compares phoneme strings by scoring all of their possible sub-sequences, tolerating gaps and awarding partial credit for similar phonemes. The DA of the system's preceding utterance is added as context. The approach rests on one hypothesis: if words signal DAs, then word fragments should signal them too, and fragments can survive even when whole words are misrecognized. On a corpus of conversations between elderly users and the system, the sub-lexical classifier held up against weak language modeling and beat a lexical classifier based on word HMMs. This suggests a cheaper route to understanding speech from atypical speakers.

Key points

  • The corpus comes from 20 nursing-home residents (3 men, 17 women) with an average age of 82.9 (range 67–97) and a mean MMSE of 21.4 (range 9–30). It contains 7,123 annotated utterances, 4,080 of them from users, averaging about 1.7 s each.
  • There are 12 application-specific DAs, such as Question, Confirmation Request, Attention Request, Repeat and Affirmative/Negative Backchannel. Two labelers agreed on 81.9% of utterances (κ = 0.782).
  • The system follows a four-stage conversation protocol: catch attention, run a pre-sequence, deliver the information, then close. It can step back to an earlier stage if the user loses track.
  • The baseline is a simplified version of Stolcke et al.'s HMM DA model. Because the system's own previous DA is known rather than hidden, the model reduces to the context prior P(U|U_R) times the likelihood of the user's utterance.
  • The proposed kernel combines sub-sequence kernels of lengths up to p with weights γ_ℓ. It penalizes gaps, soft-matches phonemes, is normalized, and remains a valid Mercer kernel.

Interesting ideas

  • I1. If whole words predict a class, their fragments should predict it too, so a classifier over sub-word pieces keeps working when the word itself is garbled or missing from the lexicon.
  • I2. A string similarity that tolerates gaps and gives graded credit to near-identical symbols absorbs insertion, deletion and substitution noise without a separate normalization stage.
  • I3. When the prior turn's act is observable, as with the system's own utterance, it can be used as a direct conditioning prior on the next act rather than inferred through a hidden Markov chain.
  • I4. Classifying straight from the raw surface signal can beat a pipeline that requires faithful transcription followed by a hard normalization step, because errors in the intermediate stage stop propagating.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger and lemmatizer could fall back on scored character sub-sequences for unknown or misspelled words instead of emitting an unknown tag, and still record which fragments drove the decision.
  • I2 (tokens): A soft-matching substring kernel could map noisy spellings or informal contractions to known lemmas, and the matched pieces would serve as an explanation.
  • I3 (frames): Speech-act tagging in prag could condition on the already-resolved act of the previous turn, for example raising the probability of an answer after a question, which keeps dialogue reasoning simple and traceable.
  • I4 (learn): In evaluation runs, Mova could compare each pipeline stage against a bypass classifier that skips that stage, as a control that shows whether the intermediate representation actually helps the downstream task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of Advanced Industrial Science and TechnologyTsukuba · Japan13 works · 2,747 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Ken Sadohara, Hiroaki Kojima, Takuya Narita, Misato Nihei, Minoru Kamata, Shinichi Onaka, Yoshihiro Fujita and Takenobu Inoue. 2013. Sub-lexical Dialogue Act Classification in a Spoken Dialogue System Support for the Elderly with Cognitive Disabilities. Proceedings of the Fourth Workshop on Speech and Language Processing for Assistive Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sadohara2013sub,
  title = {{Sub-lexical Dialogue Act Classification in a Spoken Dialogue System Support for the Elderly with Cognitive Disabilities}},
  author = {Ken Sadohara, Hiroaki Kojima, Takuya Narita, Misato Nihei, Minoru Kamata, Shinichi Onaka, Yoshihiro Fujita and Takenobu Inoue},
  booktitle = {Proceedings of the Fourth Workshop on Speech and Language Processing for Assistive Technologies},
  year = {2013},
  url = {https://aclanthology.org/W13-3915/}
}

References

35 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C. V. Ess-Dykema, and M. Meteer, “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Computational Linguistics, vol. 26, no. 3, pp. 1–34, 2000. ↗
  2. N. Webb, M. Hepple, and Y. Wilks, “Dialogue act classification based on intra-utterance features,” in Proceedings of the AAAI Workshop on Spoken Language Understanding, 2005. find
  3. D. Surendran and G. A. Levow, “Dialog act tagging with support vector machines and hidden Markov models,” in Proceedings of the Annual Conference of the International Speech Communication Association, 2006. find
  4. B. Gambäck, F. Olsson, and O. Täckström, “Active learning for dialogue act classification,” in Proceedings of the Annual Conference of the International Speech Communication Association, 2011. find
  5. S. Furui, “Spontaneous speech recognition and summarization,” in Proceedings of the Baltic Conference on Human Language Technologies, pp. 39–50, 2005. find
  6. E. Shriberg, “Spontaneous speech: how people really talk and why engineers should care,” in Proceedings of the European Conference on Speech Communication and Technology, 2005. find
  7. M. Honal and T. Schultz, “Automatic disfluency removal on recognized spontaneous speech - rapid adaptation to speaker-dependent disfluencies,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. ↗
  8. H. Nanjo and T. Kawahara, “Unsupervised language model adaptation for lecture speech recognition,” in Proceedings of the ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003. find
  9. H. Nanjo and T. Kawahara, “Language model and speaking rate adaptation for spontaneous presentation speech recognition,” IEEE Transactions on Speech and Audio Processing, vol. 12, no. 4, pp. 391–400, 2004. ↗
  10. H. Nanjo and T. Kawahara, “Speaking-rate dependent decoding and adaptation for spontaneous lecture speech recognition,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 725–728, 2002. find
  11. T. Sloboda and A. Waibel, “Dictionary learning for spontaneous speech recognition,” in Proceedings of the International Conference on Spoken Language Processing, pp. 2328–2331, 1996. find
  12. Y. Akita and T. Kawahara, “Statistical transformation of language and pronunciation models for spontaneous speech recognition,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 6, pp.1539–1549, 2010. ↗
  13. E. Shriberg, A. Stolcke, D. Jurafsky, N. Coccaro, M. Meteer, R. Bates, P. Taylor, K. Ries, R. Martin, and C. V. Ess-Dykema, “Can prosody aid the automatic classification of dialog acts in conversational speech?,” in Language and speech, vol. 41, no. 3-4, pp. 443–492, 1998. ↗
  14. T. Inoue, R. Ishiwata, R. Suzuki, T. Narita, M. Kamata, M. Shino, and M. Yaoita, “Development by a field-based method of a daily-plan indicator for persons with dementia,” in Assistive Technology from Adapted Equipment to Inclusive Environments: AATE 2009, pp. 364–368, IOS Press, 2009. ↗
  15. “Cognitive aids”, Online: http://www.abilia.org.uk, accessed on 24 Mar 2013. ↗
  16. “Automatic pill dispenser”, Online: http://www.pivotell.co.uk/, accessed on 24 Mar 2013. ↗
  17. M. Nihei, T. Narita, R. Ishiwata, M. Onoda, M. Shino, H. Kojima, S. Ohnaka, Y. Fujita, M. Kamata, and T. Inoue, “Development of an interactive information support system for persons with dementia,” in Proceedings of the International Technology and Persons with Disabilities Conference, 2011. find
  18. T. Inoue, M. Nihei, T. Narita, M. Onoda, R. Ishiwata, I. Mamiya, M. Shino, H. Kojima, S. Ohnaka, Y. Fujita, and M. Kamata, “Field-based development of an information support robot for persons with dementia,” Technology and Disability, vol. 24, no. 4, pp.263–271, 2012. ↗
  19. M. Core and J. Allen, “Dialogs with the DAMSL annotation scheme,” in Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pp. 28–35, 1997. find
  20. M. F. Folstein, S. E. Folstein, and P. R. McHugh, Mini-mental state: a practical method for grading the cognitive state of patients for the clinician, Pergamon Press, 1975. find
  21. D. Jurafsky, E. Shriberg, B. Fox, and T. Curl, “Lexical, prosodic, and syntactic cues for dialog acts,” in Proceedings of the Workshop on Discourse Relations and Discourse Markers, pp. 114– 120, 1998.
  22. V. K. R. Sridhar, S. Bangalore, and S. Narayanan, “Combining lexical, syntactic and prosodic cues for improved online dialog act tagging,” Computer Speech & Language, vol. 23, no. 4, pp. 407–422, 2009. ↗
  23. N. G. Ward and A. Vega, “Towards empirical dialog-state modeling and its use in language modeling,” in Proceedings of the Annual Conference of the International Speech Communication Association, 2012. find
  24. J. O’Shea, Z. Bandar, and K. Crockett, “A multi-classifier approach to dialogue act classification using function words,” Transactions on Computational Collective Intelligence VII, pp. 119– 143, 2012. find
  25. D. Jurafsky, R. Bates, N. Coccaro, R. Martin, M. Meteer, K. Ries, E. Shriberg, A. Stolcke, P. Taylor, and C. V. Ess-Dykema, “Automatic detection of discourse structure for speech recognition and understanding, ” in Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 88–95, 1997. ↗
  26. K. Samuel, Sandra. Carberry, and K. Vijay-Shanker, “Dialogue act tagging with transformation-based learning,” in Proceedings of the International Conference on Computational Linguistics, pp. 1150–1156, 1998. ↗
  27. S. Quarteroni, A. V. Ivanov, and G. Riccardi, “Simultaneous dialog act segmentation and classification from human-human spoken conversations,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5596–5599, 2011. find
  28. V. Vapnik, The Nature of Statistical Learning Theory. Springer-Verlag, 1995. find
  29. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines. Cambridge Press, 2000. find
  30. H. Lodhi, C. Saunders, J. Shawe-Taylor, N. Cristianini, and C. Watkins, “Text classification using string kernels,” Journal of Machine Learning Research, vol. 2, pp. 419–444, 2002. find
  31. K. Sadohara, “Kernel topic segmentation for informal multi-party meetings and performance degradation caused by insufficient lexicon,” in Proceedings of the IEEE Workshop on Spoken Language Technology, pp. 430–435, 2010. find
  32. A. Lee, T. Kawahara, and K. Shikano, “Julius — an open source real-time large vocabulary recognition engine,” in Proceedings of the European Conference on Speech Communication and Technology, pp.1691–1694, 2001. find
  33. A. Baba, S. Yoshizawa, M. Yamada, A. Lee, and K. Shikano, “Elderly acoustic model for large vocabulary continuous speech recognition,” in Proceedings of the European Conference on Speech Communication and Technology, pp.1657–1660, 2001. find
  34. A. Lee, T. Kawahara, K. Takeda, M. Mimura, A. Yamada, A. Ito, K. Itou, and K. Shikano, “Continuous speech recognition consortium: an open repository for CSR tools and models,” in Proceedings of the International Conference on Language Resources and Evaluation, pp. 1438–1441, 2002. find
  35. C. Leggetter and P. Woodland, “Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models,” Computer Speech and Language, vol. 9, pp. 171–185, 1995. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sadohara-2013-sub-lexical-dialogue-act-classification/