Elamur Research⌕
☰
home / papers / 2010 / Dialogue Act Modeling in a Complex Task-Oriented Domain

Dialogue Act Modeling in a Complex Task-Oriented Domain

Summary

The paper addresses automatic recognition of student dialogue acts in tutoring for introductory Java programming. This setting is harder than typical task-oriented dialogue because the student builds a program in a separate activity stream that runs alongside the textual conversation. The authors collected a human-human tutoring corpus and annotated it manually with dialogue act tags and with a hierarchical task/subtask scheme that also marks correctness. They encode lexical, syntactic and structural information in one feature vector. The structural features include speaker, dialogue act history, subtask labels and the hidden dialogue state predicted by a separately trained HMM. This vector feeds a maximum likelihood classifier that labels utterances greedily, left to right. The full classifier reaches 62.8% accuracy in ten-fold cross-validation, and a model using only lexical/syntactic features comes close at 60.2%. HMM and subtask features gave small gains for some feedback acts, but these gains were not statistically reliable. The work matters as a step toward data-driven tutorial dialogue managers that can reason jointly about conversation and user-driven task actions.

Key points

  • The corpus contains 48 tutoring sessions (about 48 hours), with 1,468 student and 3,338 tutor utterances. A parallel log of 97,509 keystroke-level events was manually grouped into 3,793 subtask actions, and the 66 leaves of the task hierarchy were used as features.
  • The student tagset has 13 dialogue acts, including ACKNOWLEDGMENT, REQUEST FOR FEEDBACK, QUESTION, STATEMENT and several positive, negative and uncertain feedback types. Overall inter-annotator agreement was κ=0.80.
  • Agreement on task annotation was κ=0.58 at the leaves and weighted κ=0.86 when ordinal distance between subtasks was taken into account. Agreement on correctness labels was κ=0.80.
  • The full model achieved 62.8% accuracy and the lexical/syntactic-only model 60.2%. Feature selection kept no HMM features for the overall classifier.
  • For binary classifiers of POSITIVE, NEGATIVE and UNCERTAIN ELABORATED feedback, adding HMM and task/subtask features improved results, though not significantly at p=0.05.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 11 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{boyer2010dialogue,
  title = {{Dialogue Act Modeling in a Complex Task-Oriented Domain}},
  author = {Boyer, Kristy and Ha, Eun Y. and Phillips, Robert and Wallis, Michael and Vouk, Mladen and Lester, James},
  booktitle = {Proceedings of the SIGDIAL 2010 Conference},
  year = {2010},
  url = {https://aclanthology.org/W10-4356/}
}

References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, J., Ferguson, G., & Stent, A. (2001). An architecture for more realistic conversational systems. Proceedings of the IUI, 1-8. find
  2. Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555-596. ↗
  3. Austin, J. L. (1962). How to do things with words. Oxford: Oxford University Press. find
  4. Bangalore, S., Di Fabbrizio, G., & Stent, A. (2008). Learning the structure of task-driven human-human dialogs. IEEE Transactions on Audio, Speech, and Language Processing, 16(7), 1249-1259. ↗
  5. Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996). A maximum entropy approach to natural language processing. Comp. Ling., 22(1), 71. ↗ our card
  6. Boyer, K. E., Phillips, R., Ha, E. Y., Wallis, M. D., Vouk, M. A., & Lester, J. C. (2009). Modeling dialogue structure with adjacency pair analysis and hidden markov models. Proceedings of NAACL-HLT, Short Papers, 49-52. find
  7. Boyer, K. E., Phillips, R., Ha, E. Y., Wallis, M. D., Vouk, M. A., & Lester, J. C. (2010). Leveraging hidden dialogue state to select tutorial moves. Proceedings of the 5th NAACL HLT Workshop on Innovative use of NLP for Building Educational Applications, Los Angeles, California. find
  8. Chotimongkol, A. (2008). Learning the structure of task-oriented conversations from the corpus of in-domain dialogs. (Unpublished Ph.D. Dissertation). Carnegie Mellon University School of Computer Science. find
  9. Core, M., & Allen, J. (1997). Coding dialogs with the DAMSL annotation scheme. AAAI Fall Symposium on Communicative Action in Humans and Machines, 28–35. find
  10. De Marneffe, M. C., MacCartney, B., & Manning, C. D. (2006). Generating typed dependency parses from phrase structure parses. Proceedings of LREC, Genoa, Italy. find our card
  11. Forbes-Riley, K., & Litman, D. (2009). Adapting to student uncertainty improves tutoring dialogues. Proceedings of AIED, 33-40. ↗
  12. Frampton, M., & Lemon, O. (2009). Recent research advances in reinforcement learning in spoken dialogue systems. The Knowledge Engineering Review, 24(4), 375-408. ↗
  13. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. (2009). The WEKA data mining software: An update. SIGKDD Explorations, 11(1) ↗
  14. Hardy, H., Biermann, A., Inouye, R. B., McKenzie, A., Strzalkowski, T., Ursu, C., Webb, N., & Wu, M. (2006). The Amitiés system: Data-driven techniques for automated dialogue. Speech Comm., 48(3-4), 354-373. ↗
  15. Litman, D., & Forbes-Riley, K. (2006). Correlations between dialogue acts and learning in spoken tutoring dialogues. Natural Language Engineering, 12(2), 161-176. ↗
  16. Loper, E., & Bird, S. (2004). NLTK: The natural language toolkit. Proceedings of the ACL Demonstration Session, Barcelona, Spain. 214-217. find
  17. Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130-137. ↗
  18. Purver, M., Kording, K. P., Griffiths, T. L., & Tenenbaum, J. B. (2006). Unsupervised topic modelling for multi-party spoken discourse. Proceedings of the ACL, Sydney, Australia. , 44(1) 17. find
  19. Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257-286. ↗
  20. Sridhar, V. K. R., Bangalore, S., & Narayanan, S. (2009). Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language, 23(4), 407-422. ↗
  21. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C., & Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. Comp. Ling., 26(3), 339-373. ↗ our card
  22. Wright Hastie, H., Poesio, M., & Isard, S. (2002). Automatically predicting dialogue structure using prosodic features. Speech Communication, 36(1-2), 63-79. ↗
  23. Young, S., Gasic, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., & Yu, K. (2009). The hidden information state model: A practical framework for POMDP-based spoken dialogue management. Computer Speech and Language, 24(2), 150-174. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boyer-2010-dialogue-act-modeling-complex-task/