Elamur Research⌕
☰
home / papers / 2013 / Multimodality and Dialogue Act Classification in the RoboHelper Project

Multimodality and Dialogue Act Classification in the RoboHelper Project

Summary

The paper examines how to recognize Dialogue Acts (DAs) in task-oriented conversations where people mix speech with pointing and physical manipulation of objects. Such interactions are hard to classify because partners may respond with gestures or actions instead of words, and spoken turns become short and less informative. The work is part of the RoboHelper project, which aims to build robotic assistants for elderly people living at home. The authors annotated the Find subcorpus of the ELDERLY-AT-HOME corpus, where a helper and an elderly person search for kitchen items. The annotation covers pointing gestures, Haptic-Ostensive (H-O) actions (contact actions that can also refer to objects), their links to utterances, and DAs. They then trained CRF, MaxEnt, Naive Bayes and decision tree classifiers on textual, multimodal, location, dialogue game and dialogue history features. They tested two settings: offline, using gold previous tags, and online, using the system's own earlier predictions. CRF performed best offline but dropped sharply online, while MaxEnt stayed stable and was the best online model. Combined non-verbal cues and dialogue structure both improved accuracy. This matters for building realistic multimodal dialogue managers.

Key points

  • ELDERLY-AT-HOME corpus: 20 human-human dialogues with 482 minutes of video, of which 301 minutes are task-relevant, containing 4782 spoken turns. Haptic data was recorded with instrumented gloves.
  • Find subcorpus: 137 Find tasks from 19 subject pairs, annotated in Anvil. It contains 1516 utterances, 332 pointing gestures and 597 H-O actions, nearly all of the H-O actions performed by the helper.
  • H-O action types: Five were defined: Touch, MANIP-HOLD, MANIP-NO-HOLD, Open and Close.
  • MaxEnt results: With the full feature set, MaxEnt reached .757 accuracy offline and .742 online. CRF fell from .847 offline to .475 online.
  • Feature effects: Pointing, H-O and location features helped significantly only when combined, and leaving any single one out made no significant difference. Dialogue game and dialogue history features also yielded significant gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chen2013multimodality,
  title = {{Multimodality and Dialogue Act Classification in the RoboHelper Project}},
  author = {Chen, Lin and Eugenio, Barbara Di},
  booktitle = {Proceedings of the SIGDIAL 2013 Conference},
  year = {2013},
  url = {https://aclanthology.org/W13-4031/}
}

References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, and Henry S. 1991. The HCRC Map Task corpus. Language and Speech, 34(4):351. ↗
  2. Srinivas Bangalore, Giuseppe Di Fabbrizio, and Amanda Stent. 2008. Learning the structure of task-driven human–human dialogs. IEEE Transactions on Audio, Speech, and Language Processing, 16(7):1249–1259. ↗
  3. K.E. Boyer, J.F. Grafsgaard, E.Y. Ha, R. Phillips, and J.C. Lester. 2011. An affect-enriched dialogue act classification model for task-oriented dialogue. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1190–1199. Association for Computational Linguistics. find our card
  4. Jean Carletta. 2007. Unleashing the killer corpus: experiences in creating the multi-everything AMI meeting corpus. Language Resources and Evaluation, 41(2):181–190. ↗
  5. Lauri Carlson. 1983. Dialogue games: An approach to discourse analysis. D. Reidel Publishing Company. find
  6. Lin Chen and Barbara Di Eugenio. 2012. Co-reference via pointing and haptics in multi-modal dialogues. In The 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. The Association for Computational Linguistics. find
  7. Lin Chen, Anruo Wang, and Barbara Di Eugenio. 2011. Improving pronominal and deictic coreference resolution with multi-modal features. In Proceedings of SIGdial 2011, the 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 307–311, Portland, Oregon, June. Association for Computational Linguistics. find
  8. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  9. Marie-Catherine De Marneffe and Christopher D. Manning. 2008. The Stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pages 1–8. Association for Computational Linguistics. find our card
  10. Barbara Di Eugenio, Zhuli Xie, and Riccardo Serafin. 2010a. Dialogue act classification, higher order dialogue structure, and instance-based learning. Dialogue & Discourse, 1(2):1–24. find
  11. Barbara Di Eugenio, Miloš Žefran, Jezekiel Ben-Arie, Mark Foreman, Lin Chen, Simone Franzini, Shankaranand Jagadeesan, Maria Javaid, and Kai Ma. 2010b. Towards Effective Communication with Robotic Assistants for the Elderly: Integrating Speech, Vision and Haptics. In Dialog with Robots, AAAI 2010 Fall Symposium, Arlington, VA, USA, November. find
  12. M.E. Foster, E.G. Bard, M. Guhe, R.L. Hill, J. Oberlander, and A. Knoll. 2008. The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction, pages 295–302. ACM. find
  13. Simone Franzini and Jezekiel Ben-Arie. 2012. Speech recognition by indexing and sequencing. International Journal of Computer Information Systems and Industrial Management Applications, 4:358–365. find
  14. David Graff, Alexandra Canavan, and George Zipperlen. 1998. Switchboard-2 Phase I. find
  15. Eun Young Ha, Joseph F. Grafsgaard, Christopher Mitchell, Kristy Elizabeth Boyer, and James C. Lester. 2012. Combining verbal and nonverbal features to overcome the “information gap” in task-oriented dialogue. In Proceedings of SIGdial 2012, the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 247–256, Seoul, South Korea, July. Association for Computational Linguistics. find
  16. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1). ↗
  17. Helen Wright Hastie, Massimo Poesio, and Stephen Isard. 2002. Automatically predicting dialogue structure using prosodic features. Speech Communication, 36(1–2):63–79. ↗
  18. Edward Ivanovic. 2008. Automatic instant messaging dialogue using statistical models and dialogue acts. Master’s thesis, University of Melbourne. find
  19. Maria Javaid and Miloš Žefran. 2012. Interpreting communication through physical interaction during collaborative manipulation. Draft, October. find
  20. Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2010. Classifying dialogue acts in one-on-one live chats. In Proceedings of EMNLP 2010, the Conference on Empirical Methods in Natural Language Processing, pages 862–871. Association for Computational Linguistics. find our card
  21. Su Nam Kim, Lawrence Cavedon, and Timothy Baldwin. 2012. Classifying dialogue acts in multi-party live chats. In Proceedings of the 26th Pacific Asia Conference on Language, Information, and Computation, pages 463–472, Bali, Indonesia, November. Faculty of Computer Science, Universitas Indonesia. find our card
  22. Michael Kipp. 2001. Anvil-a generic annotation tool for multimodal dialogue. In Proceedings of the 7th European Conference on Speech Communication and Technology, pages 1367–1370. find
  23. Kristine M. Krapp. 2002. The Gale Encyclopedia of Nursing & Allied Health. Gale Group, Inc. Chapter Activities of Daily Living Evaluation. find
  24. John D. Lafferty, Andrew McCallum, and Fernando C.N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 282–289. Morgan Kaufmann Publishers Inc. find
  25. L. Levin, A. Thymé-Gobbel, A. Lavie, K. Ries, and K. Zechner. 1998. A discourse coding scheme for conversational Spanish. In Fifth International Conference on Spoken Language Processing. find
  26. K. Ma and J. Ben-Arie. 2012. Multi-view multi-class object detection via exemplar compounding. In IEEE-IAPR 21st International Conference on Pattern Recognition (ICPR 2012), Tsukuba, Japan, November. find
  27. Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. find
  28. E. Shriberg, R. Dhillon, S.V. Bhagat, J. Ang, and H. Carvey. 2004. The ICSI Meeting Recorder Dialog Act (MRDA) Corpus. In Proceedings of 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100, Cambridge, MA, April 30-May 1. ↗ our card
  29. V.K.R. Sridhar, S. Bangalore, and S. Narayanan. 2009. Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language, 23(4):407–422. ↗
  30. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C.V. Ess-Dykema, and M. Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
  31. Matthias Zimmermann, Andreas Stolcke, and Elizabeth Shriberg. 2006. Joint segmentation and classification of dialog acts in multiparty meetings. In ICASSP 2006, the IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1. IEEE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2013-multimodality-dialogue-act-classification-robohelper/