Elamur Research⌕
☰
home / papers / 2014 / Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models

Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models

Paper · 2014 · Aysu Ezen-Can and Kristy Boyer

Summary

Unsupervised dialogue act classifiers avoid hand-built tagsets and costly annotation, but they have trailed supervised systems by a wide margin. This paper asks whether, in task-oriented dialogue, a parallel stream of task events can narrow that gap. The data is a corpus of text-based human tutoring in introductory Java with 42 novice students. Alongside the chat, the system logged every keystroke-level coding, compiling and execution event. Student utterances are clustered using three kinds of features:

  • word unigrams, compared through a longest common sub-sequence (LCS) distance;
  • dialogue context features;
  • task features, such as the latest programming action and its status.

Flat clustering is compared with a hierarchical variant that first splits utterances by the preceding tutor dialogue act. Manual tags are used only for evaluation. The best hierarchical model, using all three feature groups, reaches 65% accuracy (kappa 0.48). This compares with a 39.85% majority baseline and a 69% supervised CRF from earlier work. The results suggest that non-linguistic context can partly substitute for supervision.

Key points

  • The corpus has 5,705 annotated utterances (4,065 tutor, 1,640 student) from the first lesson of each pair, with κ=0.80. The task is to classify only student acts: Answer, Acknowledgement, Statement, Question, Request for Feedback, Clarification and Other.
  • Evaluation is held-out, leave-one-student-out testing. A separate set of 462 unannotated utterances is used only to choose the number of clusters.
  • In the hierarchical model, branching on the previous tutor act beat flat clustering for every feature set. The scores were T 64.2%, D 63.2%, L 60.7% and T+D+L 65%.
  • Inside the hierarchical model, task features did not significantly improve on D alone or L alone. Only T+L versus L was significant.
  • Per-act results:
    • Task-only features found 61.8% of statements, versus 51.3% for the full model.
    • T+L was best on acknowledgements (63.6%).
    • Questions stayed poor, at 14% or less.
    • With flat clustering, LCS reached 33%, while bigrams reached 25%.

Interesting ideas

  • I1. Use the immediately preceding partner move as a top-level branch, then cluster separately within each branch; conditioning on that one move sharpens otherwise ambiguous groups.
  • I2. Measure utterance similarity with longest common sub-sequence over words, so that gapped word patterns count without explicit n-gram features.
  • I3. Treat a non-verbal event stream (actions taken in the shared world) as evidence for what an utterance does; this resolves cases like "yes" serving as either an answer or an acknowledgement.
  • I4. Report results per category, not only overall, because a feature set can win overall yet lose on a specific class (task-only beat all features on statements).
  • I5. Fit model-size choices (here, the number of clusters) on a separate unlabeled development set, so that the labeled data stays purely for testing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act classifier in prag could first condition on the previous turn's act (question vs. directive) before reading the current utterance, which keeps the decision explainable.
  • I2 (frames): An LCS-based similarity over lemma sequences could match new utterances to known speech-act templates while tolerating inserted words.
  • I3 (world): Events and states already recorded in the world of the text could be passed back as features when deciding the act or meaning of short, context-dependent replies.
  • I4 (learn): Evaluation reports should break scores down by frame or act type, so that a feature which hurts one class is not hidden by the average.
  • I5 (learn): When inducing clusters or repair rules from silver data, any tuning should happen on a disjoint pool so that gold evaluation remains clean.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓North Carolina State UniversityRaleigh · United States7 works · 376 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Aysu Ezen-Can and Kristy Boyer. 2014. Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-4316

@inproceedings{ezencan2014combining,
  title = {{Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models}},
  author = {Aysu Ezen-Can and Kristy Boyer},
  booktitle = {Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)},
  year = {2014},
  doi = {10.3115/v1/w14-4316},
  url = {https://doi.org/10.3115/v1/w14-4316}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F. Allen, Lenhart K. Schubert, George Ferguson, Peter Heeman, Chung Hee Hwang, Tsuneaki Kato, Marc Light, Nathaniel Martin, Bradford Miller, Massimo Poesio, et al. 1995. The TRAINS project: A case study in building a conversational planning agent. Journal of Experimental & Theoretical Artificial Intelligence, 7(1):7–48. ↗
  2. David Arthur and Sergei Vassilvitskii. 2007. kmeans++: The advantages of careful seeding. In Proceedings of the 18th ACM-SIAM Symposium on Discrete Algorithms, pages 1027–1035. Society for Industrial and Applied Mathematics. find
  3. John Langshaw Austin. 1975. How To Do Things with Words, volume 1955. Oxford University Press. ↗
  4. Srinivas Bangalore, Giuseppe Di Fabbrizio, and Amanda Stent. 2008. Learning the structure of task-driven human–human dialogs. IEEE Transactions on Audio, Speech, and Language Processing, 16(7):1249–1259. ↗
  5. Kristy Elizabeth Boyer, Eun Young Ha, Robert Phillips, Michael D. Wallis, Mladen A. Vouk, and James C. Lester. 2010. Dialogue act modeling in a complex task-oriented domain. In Proceedings of SIGDIAL, pages 297–305. Association for Computational Linguistics.
  6. Kristy Elizabeth Boyer, Eun Young Ha, Robert Phillips, and James Lester. 2011. The impact of task-oriented feature sets on HMMs for dialogue modeling. In Proceedings of SIGDIAL, pages 49– 58. Association for Computational Linguistics. find
  7. Lin Chen and Barbara Di Eugenio. 2013. Multimodality and dialogue act classification in the RoboHelper project. In Proceedings of SIGDIAL, pages 183– 192.
  8. Mark G. Core and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Proceedings of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35. find
  9. Nigel Crook, Ramon Granell, and Stephen Pulman. 2009. Unsupervised classification of dialogue acts using a Dirichlet process mixture model. In Proceedings of SIGDIAL, pages 341–348. Association for Computational Linguistics. find
  10. Barbara Di Eugenio, Zhuli Xie, and Riccardo Serafin. 2010. Dialogue act classification, higher order dialogue structure, and instance-based learning. Dialogue & Discourse, 1(2):1–24. find
  11. Myroslava O. Dzikovska, Elaine Farrow, and Johanna D. Moore. 2013. Combining semantic interpretation and statistical classification for improved explanation processing in a tutorial dialogue system. In Artificial Intelligence in Education, pages 279– 288. find
  12. Aysu Ezen-Can and Kristy Elizabeth Boyer. 2013a. In-context evaluation of unsupervised dialogue act models for tutorial dialogue. In Proceedings of SIG-DIAL, pages 324–328.
  13. Aysu Ezen-Can and Kristy Elizabeth Boyer. 2013b. Unsupervised classification of student dialogue acts with query-likelihood clustering. In International Conference on Educational Data Mining, pages 20– 27. find
  14. Oliver Ferschke, Iryna Gurevych, and Yevgen Chebotar. 2012. Behind the article: Recognizing dialog acts in Wikipedia talk pages. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 777– 786.
  15. Kate Forbes-Riley and Diane J. Litman. 2005. Using bigrams to identify relationships between student certainness states and tutor responses in a spoken dialogue corpus. In Proceedings of the SIG-DIAL Workshop, pages 87–96. find
  16. Kate Forbes-Riley, Mihai Rotaru, Diane J. Litman, and Joel Tetreault. 2007. Exploring affect-context dependencies for adaptive system development. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics, pages 41–44. find
  17. Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175–204. ↗
  18. Eun Young Ha, Joseph F. Grafsgaard, Christopher M. Mitchell, Kristy Elizabeth Boyer, and James C. Lester. 2012. Combining verbal and nonverbal features to overcome the ‘information gap’ in task-oriented dialogue. In Proceedings of SIGDIAL, pages 247–256. find
  19. Daniel S. Hirschberg. 1975. A linear space algorithm for computing maximal common subsequences. Communications of the ACM, 18(6):341– 343. ↗
  20. Shafiq Joty, Giuseppe Carenini, and Chin-Yew Lin. 2011. Unsupervised modeling of dialog acts in asynchronous conversations. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, pages 1807–1813. find
  21. Daniel Jurafsky, Elizabeth Shriberg, Barbara Fox, and Traci Curl. 1998. Lexical, prosodic, and syntactic cues for dialog acts. In Proceedings of the ACL/COLING-98 Workshop on Discourse Relations and Discourse Markers, pages 114–120.
  22. Simon Keizer, Rieks op den Akker, and Anton Nijholt. 2002. Dialogue act recognition with Bayesian networks for Dutch dialogues. In Proceedings of the SIGDIAL Workshop, pages 88–94.
  23. Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871.
  24. Donghyeon Lee, Minwoo Jeong, Kyungduk Kim, and Seonghan Ryu. 2013. Unsupervised spoken language understanding for a multi-domain dialog system. IEEE Transactions On Audio, Speech, and Language Processing, 21(11):2451–2464. ↗
  25. Johanna Marineau, Peter Wiemer-Hastings, Derek Harter, Brent Olde, Patrick Chipman, Ashish Karnavat, Victoria Pomeroy, Sonya Rajan, Art Graesser, Tutoring Research Group, et al. 2000. Classification of speech acts in tutorial dialog. In Proceedings of the Workshop on Modeling Human Teaching Tactics and Strategies at the Intelligent Tutoring Systems Conference, pages 65–71. find
  26. T. Daniel Midgley, Shelly Harrison, and Cara Mac-Nish. 2009. Empirical verification of adjacency pairs using dialogue segmentation. In Proceedings of SIGDIAL, pages 104–108. find
  27. Raymond Ng and Jiawei Han. 1994. Efficient and effective clustering methods for spatial data mining. In Proceedings of the 20th International Conference on Very Large Data Bases, pages 144–155. find
  28. Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In Proceedings of EuroSpeech, pages 2235–2238. find
  29. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In Proceedings of the Association for Computational Linguistics, pages 172–180.
  30. Vasile Rus, Cristian Moldovan, Nobal Niraula, and Arthur C. Graesser. 2012. Automated discovery of speech act categories in educational games. In International Conference on Educational Data Mining, pages 25–32. find
  31. Emanuel A. Schegloff and Harvey Sacks. 1973. Opening up closings. Semiotica, 8(4):289–327. ↗
  32. John R. Searle. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press. ↗
  33. Riccardo Serafin and Barbara Di Eugenio. 2004. FLSA: Extending latent semantic analysis with features for dialogue act classification. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 692–699. Association for Computational Linguistics.
  34. Rangarajan Sridhar, Vivek Kumar, Srinivas Bangalore, and Shrikanth Narayanan. 2009. Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language, 23(4):407–422. ↗
  35. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. ↗
  36. Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2013. Dialogue act recognition in synchronous and asynchronous conversations. In Proceedings of SIGDIAL, pages 117– 121. ↗
  37. David R. Traum. 1999. Speech acts for dialogue agents. In Foundations of Rational Agency, pages 169–201. Springer. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/can-2014-combining-task-dialogue-streams-unsupervised/