Elamur Research⌕
☰
home / papers / 2016 / Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems

Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems

Paper · 2016 · Ramesh Manuvinakurike, Maike Paetzel, Cheng Qu et al.

Summary

Spoken dialogue systems often treat each pause as the end of a dialogue act (DA). This fails in three ways: some DAs contain pauses, some pause-free stretches hold several DAs, and waiting for silence adds delay. The authors build an incremental, word-by-word DA segmenter and classifier for RDG-Image, a fast, time-limited image-matching game. The system is a sequential pipeline that uses prosodic, lexico-syntactic and contextual cues. It was trained on a newly annotated lab corpus of human-human play with 18 domain DA labels. They measure errors with standard segmentation metrics and with new metrics that ignore boundaries and only ask whether each DA type was found. They also compare pipeline variants to see which processing step causes which errors. Finally, they add the fully automated pipeline (AT-AS-AD) to an existing game agent so that only target-description text reaches its understanding module. In simulated play the agent performed no worse than before. This shows the segmenter is accurate enough to support richer conversational behaviour later.

Key points

  • Corpus: 64 participants in 32 pairs produced 1,906 target images and 8,792 speech segments. The data total 4.27 hours of audio, 67,125 tokens and 12,241 annotated DA segments.
    • Speech was pre-split into IPUs at pauses of at least 300 ms.
    • Inter-annotator kappa was 0.92 for boundaries and 0.83 for word-level DA labels.
  • Why pause-based splitting fails:
    • 23% of speech segments start two or more DAs.
    • 29% of DAs begin after the first word of a segment.
    • 4% of DAs span more than one segment.
  • Label set: 18 DA labels, including D-T (describe target), Q-YN, PFB, EC, ST (self-talk), As-I, As-S and a filled-pause act P.
  • New evaluation metrics: precision and recall for each DA type, computed independently of where the boundaries fall. These suit noisy ASR output and dialogue policies that only react to whether a given act is present.
  • Eavesdropper simulations: 18 image sets, using the 32 lab games plus 98 web games, with tagging every 100 ms.
    • The agent filtered to D-T text matched the baseline agent on time, points, points per second and NLU accuracy.
    • A Wilcoxon-Mann-Whitney test found no significant differences (e.g. p = .956 for points per second).

Interesting ideas

  • I1. Instead of trusting pauses, the authors segment the continuous word stream into acts, allowing several acts inside one stretch of speech and letting one act cross a pause. This matters wherever surface delimiters disagree with meaning units.
  • I2. The new metrics score each act type by whether it is detected anywhere in the input, regardless of exact boundaries. This is fair to downstream consumers that only care whether an act is present.
  • I3. The authors attribute errors to individual pipeline stages by mixing manual and automatic inputs, showing how much each automatic step degrades the result.
  • I4. They validate a new component by checking that end-task performance stays the same, using a significance test, rather than relying only on intrinsic scores.
  • I5. Filler, self-talk and feedback acts are kept as separate labels so they can be filtered out before content understanding.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Sentence and clause splitting could stop relying only on punctuation and let a learned decision split run-on text or join fragments split by ellipses.
  • I2 (frames): Speech-act detection in crate prag could be scored per type and per utterance, so small boundary shifts are not counted as failures.
  • I3 (learn): Mova could swap gold for predicted output one stage at a time (tokens, tree, frames) to find which layer is responsible for wrong answers.
  • I4 (learn): Before enabling any new layer or repair rule, Mova could run it as a "no-harm" check against the answer-level benchmarks.
  • I5 (world): Asides, hedges and self-talk could be tagged and kept out of the world-of-text state, so they do not create spurious entities or events.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Uppsala UniversityUppsala · Sweden56 works · 13,046 citations🎓Bielefeld UniversityBielefeld · Germany19 works · 968 citations🔬USC Institute for Creative TechnologiesPlaya Vista · United States4 works · 278 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Ramesh Manuvinakurike, Maike Paetzel, Cheng Qu, David Schlangen and David DeVault. 2016. Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-3632

@inproceedings{manuvinakurike2016toward,
  title = {{Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems}},
  author = {Ramesh Manuvinakurike, Maike Paetzel, Cheng Qu, David Schlangen and David DeVault},
  booktitle = {Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2016},
  doi = {10.18653/v1/w16-3632},
  url = {https://doi.org/10.18653/v1/w16-3632}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeremy Ang, Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In ICASSP, pages 1061– 1064. find
  2. Timo Baumann and David Schlangen. 2012. The in-protk 2012 release. In NAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data, pages 29–32. find
  3. Linda Bell, Johan Boye, and Joakim Gustafson. 2001. Real-time handling of fragmented utterances. In The NAACL Workshop on Adaption in Dialogue Systems, pages 2–8. find
  4. Ananlada Chotimongkol and Alexander I Rudnicky. 2001. N-best speech hypotheses reordering using linear regression. find
  5. David DeVault, Kenji Sagae, and David Traum. 2011. Incremental interpretation and prediction of utterance meaning for interactive dialogue. Dialogue and Discourse, 2(1):143–70. ↗
  6. Raul Fernandez and Rosalind W Picard. 2002. Dialog act classification from prosodic features using support vector machines. In Speech Prosody 2002, International Conference. find
  7. Luciana Ferrer, Elizabeth Shriberg, and Andreas Stolcke. 2003. A prosody-based approach to end-of-utterance detection that does not require speech. In Proc. IEEE ICASSP, pages 608–611. find
  8. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10– 18. ↗
  9. Hanae Koiso, Yasuo Horiuchi, Syun Tutiya, Akira Ichikawa, and Yasuharu Den. 1998. An analysis of turn-taking and backchannels based on prosodic and syntactic features in japanese map task dialogs. Language and Speech, 41(3-4):295–321. ↗
  10. Jáchym Kolář, Elizabeth Shriberg, and Yang Liu. 2006. On speaker-specific prosodic models for automatic dialog act segmentation of multi-party meetings. In Interspeech, volume 1. find
  11. Kazunori Komatani, Naoki Hotta, Satoshi Sato, and Mikio Nakano. 2015. User adaptive restoration for incorrectly-segmented utterances in spoken dialogue systems. In SIGDIAL. find
  12. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pages 282–289. find
  13. Kornel Laskowski and Elizabeth Shriberg. 2010. Comparing the contributions of context and prosody in text-independent dialog act recognition. In ICASSP, pages 5374–5377. IEEE. find
  14. Piroska Lendvai and Jeroen Geertzen. 2007. Token-based chunking of turn-internal dialogue act sequences. In SIGDIAL, pages 174–181.
  15. Diane J Litman, Mihai Rotaru, and Greg Nicholas. 2009. Classifying turn-level uncertainty using word-level prosody. In INTERSPEECH, pages 2003– 2006. find
  16. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In ACL: System Demonstrations, pages 55–60.
  17. Ramesh Manuvinakurike and David DeVault, 2015. Natural Language Dialog Systems and Intelligent Assistants, chapter Pair Me Up: A Web Framework for Crowd-Sourced Spoken Dialogue Collection, pages 189–201. find
  18. Ramesh Manuvinakurike, Casey Kennington, David DeVault, and David Schlangen. 2016. Real-time understanding of complex discriminative scene descriptions. In SIGDIAL. find
  19. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
  20. Fabrizio Morbini and Kenji Sagae. 2011. Joint identification and segmentation of domain-specific dialogue acts for conversational dialogue systems. In Proceedings of ACL: Human Language Technologies: short papers, pages 95–100.
  21. Mikio Nakano, Noboru Miyazaki, Jun-ichi Hirasawa, Kohji Dohsaka, and Takeshi Kawabata. 1999. Understanding unsegmented user utterances in real-time spoken dialogue systems. In ACL, pages 200– 207. find
  22. Maike Paetzel, David Nicolas Racca, and David De-Vault. 2014. A multimodal corpus of rapid dialogue games. In LREC, May. find
  23. Maike Paetzel, Ramesh Manuvinakurike, and David DeVault. 2015. “So, which one is it?” The effect of alternative incremental architectures in a high-performance game-playing agent. In SIGDIAL. find
  24. Volha Petukhova and Harry Bunt. 2014. Incremental recognition and prediction of dialogue acts. In Computing Meaning, pages 235–256. Springer. ↗
  25. Ondřej Plátek and Filip Jurčı́ček. 2014. Free on-line speech recogniser based on Kaldi ASR toolkit producing word posterior lattices. In SIGDIAL. find
  26. Antoine Raux and Maxine Eskenazi. 2008. Optimizing endpointing thresholds using dialogue features in a spoken dialogue system. In SIGDIAL, pages 1– 10. find
  27. Ryo Sato, Ryuichiro Higashinaka, Masafumi Tamoto, Mikio Nakano, and Kiyoaki Aikawa. 2002. Learning decision trees to determine turntaking by spoken dialogue systems. In Proceedings of ICSLP-02, pages 861–864. find
  28. David Schlangen and Gabriel Skantze. 2011. A general, abstract model of incremental dialogue processing. dialogue and discourse. Dialogue and Discourse, 2(1):83–111. ↗
  29. Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-Tür, and Gökhan Tür. 2000. Prosody-based automatic segmentation of speech into sentences and topics. Speech communication, 32(1):127–154. ↗
  30. Gabriel Skantze and David Schlangen. 2009. Incremental dialogue processing in a micro-domain. In EACL, pages 745–753. find
  31. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗
  32. Masashi Takeuchi, Norihide Kitaoka, and Seiichi Nakagawa. 2004. Timing detection for realtime dialog systems using prosodic and linguistic information. In Speech Prosody 2004. find
  33. Nigel G Ward and David DeVault. 2015. Ten challenges in highly interactive dialog systems. In AAAI 2015 Spring Symposium. find
  34. V. Warnke, R. Kompe, H. Niemann, and E. Nth. 1997. Integrated dialog act segmentation and classification using prosodic features and language models. In Proc. 5th Europ. Conf. on Speech, Communication, and Technology, volume 1. find
  35. Matthias Zimmermann, Yang Liu, Elizabeth Shriberg, and Andreas Stolcke, 2006. find
  36. Second International Workshop, MLMI 2005, chapter Toward Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings, pages 187–193. find
  37. Matthias Zimmermann. 2009. Joint segmentation and classification of dialog acts using conditional random fields. In Interspeech. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/manuvinakurike-2016-incremental-dialogue-act-segmentation-fast/