Elamur Research⌕
☰
home / papers / 2011 / An Affect-Enriched Dialogue Act Classification Model for Task-Oriented Dialogue

An Affect-Enriched Dialogue Act Classification Model for Task-Oriented Dialogue

Summary

Most dialogue act classifiers ignore the user's emotional state. This paper asks whether facial cues of confusion can help label student moves in textual, task-oriented tutoring dialogue. The data come from a human-human tutoring study in introductory computer programming. Students typed to tutors while webcams recorded their faces, and the tutors never saw this footage. The 14 best-quality videos were manually coded with Ekman's FACS for Action Unit 4 (AU4), a brow-lowering movement that prior work links to confusion. AU4 presence was encoded over several time windows, both before the target utterance and across earlier turns. These features were incorporated into dialogue act classifiers whose vector-based framework closely resembles the one Sridhar et al. (2009) used for their maximum-entropy modeling. Separate classifiers were trained per act, with feature selection. The affect-enriched models were significantly better at recognizing GROUNDING and REQUEST FOR FEEDBACK acts. The work is an early demonstration that a nonverbal affective channel can improve dialogue act recognition. This matters because webcams are now widespread.

Key points

  • Corpus: 48 tutoring dialogues in total, of which 14 high-quality videos were used. These run 11 h 55 min and cover 3 female and 11 male students working with 2 tutors.
  • Annotation reliability: Dialogue act annotation reached κ=0.80. Task annotation scored a simple κ of .56 and a weighted κ of .80. GROUNDING was the most frequent student act, at about 21%.
  • Features: AU4 presence was measured in 1-, 5- and 10-second windows, using a dialogue history of up to three utterances back.
  • Grounding: When AU4 was absent just before an utterance, a grounding act such as "Ok" was more likely.
  • Requests for feedback: These were predicted by AU4 three turns earlier and again just before the message, suggesting a delayed and recurring display of confusion.
  • Limitations: Gains for QUESTION, POSITIVE FEEDBACK and EXTRA-DOMAIN acts were not statistically significant. The sample was small because manual tagging is costly. The manual coding likely represents an upper bound for what automatic facial analysis could achieve.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{boyer2011affect,
  title = {{An Affect-Enriched Dialogue Act Classification Model for Task-Oriented Dialogue}},
  author = {Boyer, Kristy and Grafsgaard, Joseph and Ha, Eun Young and Phillips, Robert and Lester, James},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1119/}
}

References

43 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Andreevskaia and S. Bergler. 2008. When specialists and generalists work together: Overcoming domain dependence in sentiment tagging. Proceedings of the Annual Meeting of the Association for Computational Linguistics and Human Language Technologies (ACL HLT), 290-298. find
  2. M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan. 2006. Fully Automatic Facial Action Recognition in Spontaneous Behavior. 7th International Conference on Automatic Face and Gesture Recognition (FGR06), 223-230. find
  3. K.E. Boyer, M. Vouk, and J.C. Lester. 2007. The influence of learner characteristics on task-oriented tutorial dialogue. Proceedings of the International Conference on Artificial Intelligence in Education, 365–372. find
  4. K.E. Boyer, E.Y. Ha, R. Phillips, M.D. Wallis, M. Vouk, and J.C. Lester. 2010. Dialogue act modeling in a complex task-oriented domain. Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 297-305. find our card
  5. K.E. Boyer, R. Phillips, E.Y. Ha, M.D. Wallis, M.A. Vouk, and J.C. Lester. 2009. Modeling dialogue structure with adjacency pair analysis and hidden Markov models. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Short Papers, 49-52. find
  6. K.E. Boyer, R. Phillips, E.Y. Ha, M.D. Wallis, M.A. Vouk, and J.C. Lester. 2010. Leveraging hidden dialogue state to select tutorial moves. Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, 66-73. find
  7. R.A. Calvo and S. D’Mello. 2010. Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing, 1(1): 18-37. ↗
  8. M. Cavazza, R.S.D.L. Cámara, M. Turunen, J. Gil, J. Hakulinen, N. Crook, et al. 2010. How was your day? An affective companion ECA prototype. Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 277-280. find
  9. F. Cavicchio. 2009. The modulation of cooperation and emotion in dialogue: the REC Corpus. Proceedings of the ACL-IJCNLP 2009 Student Research Workshop, 43-48. find
  10. J.F. Cohn, L.I. Reed, Z. Ambadar, J. Xiao, and T. Moriyama. 2004. Automatic Analysis and Recognition of Brow Actions and Head Motion in Spontaneous Facial Behavior. IEEE International Conference on Systems, Man and Cybernetics, 610-616. find
  11. S.D. Craig, S. D’Mello, A. Witherspoon, J. Sullins, and A.C. Graesser. 2004. Emotions during learning: The first steps toward an affect sensitive intelligent tutoring system. In J. Nall and R. Robson (Eds.), E-learn 2004: World conference on Elearning in Corporate, Government, Healthcare, & Higher Education, 241-250. find
  12. D. Das and S. Bandyopadhyay. 2009. Word to sentence level emotion tagging for Bengali blogs. Proceedings of the ACL-IJCNLP Conference, Short Papers, 149-152. find
  13. S. Dasgupta and V. Ng. 2009. Mine the easy, classify the hard: a semi-supervised approach to automatic sentiment classification. Proceedings of the 46th Annual Meeting of the ACL and the 4th IJCNLP, 701-709. find
  14. B. Di Eugenio, Z. Xie, and R. Serafin. 2010. Dialogue Act Classification, Higher Order Dialogue Structure, and Instance-Based Learning. Dialogue & Discourse, 1(2): 1-24. find
  15. M. Dzikovska, J.D. Moore, N. Steinhauser, and G. Campbell. 2010. The impact of interpretation problems on tutorial dialogue. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Short Papers, 43-48. find
  16. S. D’Mello, S.D. Craig, J. Sullins, and A.C. Graesser. 2006. Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor’s Mixed- Initiative Dialogue. International Journal of Artificial Intelligence in Education, 16(1): 3-28. ↗
  17. P. Ekman. 1999. Basic Emotions. In T. Dalgleish and M. J. Power (Eds.), Handbook of Cognition and Emotion. New York: Wiley. find
  18. P. Ekman, W.V. Friesen. 1978. Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press. find
  19. P. Ekman, W.V. Friesen, and J.C. Hager. 2002. Facial Action Coding System: Investigator’s Guide. Salt Lake City, USA: A Human Face. find
  20. P. Ekman and E.L. Rosenberg (Eds.). 2005. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) (2nd ed.). New York: Oxford University Press. find
  21. K. Forbes-Riley, M. Rotaru, D.J. Litman, and J. Tetreault. 2007. Exploring affect-context dependencies for adaptive system development. The Conference of the North American Chapter of the Association for Computational Linguistics and Human Language Technologies (NAACL HLT), Short Papers, 41-44. find
  22. K. Forbes-Riley, M. Rotaru, D.J. Litman, and J. Tetreault. 2009. Adapting to student uncertainty improves tutoring dialogues. Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED), 33-40. ↗
  23. A.C. Graesser, S. Lu, B. Olde, E. Cooper-Pye, and S. Whitten. 2005. Question asking and eye tracking during cognitive disequilibrium: comprehending illustrated texts on devices when the devices break down. Memory & Cognition, 33(7): 1235-1247. ↗
  24. S. Greene and P. Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. Proceedings of the 2009 Annual Conference of the North American Chapter of the ACL and Human Language Technologies (NAACL HLT), 503-511. find
  25. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I.H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1): 10–18. ↗
  26. R. Iida, S. Kobayashi, and T. Tokunaga. 2010. Incorporating extra-linguistic information into reference resolution in collaborative task dialogue. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 1259-1267. find
  27. M.L. Knapp and J.A. Hall. 2006. Nonverbal Communication in Human Interaction (6th ed.). Belmont, CA: Wadsworth/Thomson Learning. find
  28. C.M. Lee, S.S. Narayanan. 2005. Toward detecting emotions in spoken dialogs. IEEE Transactions on Speech and Audio Processing, 13(2): 293-303. ↗
  29. R. López-Cózar, J. Silovsky, and D. Griol. 2010. F2– New Technique for Recognition of User Emotional States in Spoken Dialogue Systems. Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 281-288. find
  30. B.T. McDaniel, S. D’Mello, B.G. King, P. Chipman, K. Tapp, and A.C. Graesser. 2007. Facial Features for Affective State Detection in Learning Environments. Proceedings of the 29th Annual Cognitive Science Society, 467-472. find
  31. D. McNeill. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press. find
  32. A. Mehrabian. 2007. Nonverbal Communication. New Brunswick, NJ: Aldine Transaction. find
  33. T. Nguyen. 2010. Mood patterns and affective lexicon access in weblogs. Proceedings of the ACL 2010 Student Research Workshop, 43-48. find
  34. M. Pantic and M.S. Bartlett. 2007. Machine Analysis of Facial Expressions. In K. Delac and M. Grgic (Eds.), Face Recognition, 377-416. Vienna, Austria: I-Tech Education and Publishing. ↗
  35. J.A. Russell. 2003. Core affect and the psychological construction of emotion. Psychological Review, 110(1): 145-172. ↗
  36. J.A. Russell, J.A. Bachorowski, and J.M. Fernandez-Dols. 2003. Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329-49. ↗
  37. K.L. Schmidt and J.F. Cohn. 2001. Human Facial Expressions as Adaptations: Evolutionary Questions in Facial Expression Research. Am J Phys Anthropol, 33: 3-24. ↗
  38. V.J. Shute. 2008. Focus on Formative Feedback. Review of Educational Research, 78(1): 153-189. ↗
  39. V.K.R Sridar, S. Bangalore, and S.S. Narayanan. 2009. Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language, 23(4): 407-422. Elsevier Ltd. find
  40. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, et al. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics, 26(3): 339-373. ↗ our card
  41. C. Toprak, N. Jakob, and I. Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 575-584. find
  42. T. Wilson, J. Wiebe, and P. Hoffmann. 2009. Recognizing Contextual Polarity: An Exploration of Features for Phrase-Level Sentiment Analysis. Computational Linguistics, 35(3): 399-433. ↗
  43. Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang. 2009. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1): 39-58. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/boyer-2011-affect-enriched-dialogue-act-classification/