Elamur Research⌕
☰
home / papers / 2024 / Common Ground Tracking in Multimodal Dialogue

Common Ground Tracking in Multimodal Dialogue

Summary

The paper presents common ground tracking (CGT), a counterpart to dialogue state tracking. Where DST follows one user's needs, CGT follows what a whole group jointly accepts as true during a task-oriented, multimodal interaction. The authors model this with a Common Ground Structure (CGS) that has three parts: FBank (propositions the group treats as facts), EBank (propositions with supporting evidence) and QBank (open questions under discussion, QUDs). Belief is graded rather than binary. The formalism draws on evidence-based Dynamic Epistemic Logic and a Dialogue Game Board.

The authors extend the Weights Task Dataset with annotations for gesture, action and common ground. A deep neural classifier uses speech transcripts, prosody, gestures, actions and collaboration indicators to predict moves toward shared understanding. Its outputs then pass through formal closure and update rules that derive entailed beliefs. The study measures how much each feature type contributes against ground truth and sets a first benchmark for this difficult task. This matters for AI agents that need to know what a collaborating group currently agrees on.

Key points

  • Dataset: the Weights Task Dataset (WTD) records triads using a balance scale to work out the weights of five colored blocks, whose weights follow a Fibonacci pattern.
  • Existing annotations: speech transcripts (automatic and manual), Gesture-AMR (GAMR) and CPS indicators.
  • New annotations: gesture, action and common ground labels, made mainly in ELAN.
  • Move classifier evaluation: hold-one-group-out over 10 groups. Per-group accuracy ranges from 0.429 to 0.833.
  • Inter-annotator agreement:
    • Common ground: mean F1 0.541, Cohen's κ 0.500.
    • Actions: mean F1 0.671, Cohen's κ 0.597.
    • GAMR: mean F1 about 0.75.
  • Code: available at github.com/csu-signal/Common-Ground-detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2403.17284

@inproceedings{khebour2024common,
  title = {{Common Ground Tracking in Multimodal Dialogue}},
  author = {Khebour, Ibrahim Khalil and Lai, Kenneth and Bradford, Mariah and Zhu, Yifan and Brutti, Richard A. and Tam, Christopher and Tu, Jingxuan and Ibarra, Benjamin A. and Blanchard, Nathaniel and Krishnaswamy, Nikhil and Pustejovsky, James},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/4kmqkvuo7jy3},
  eprint = {2403.17284},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/4kmqkvuo7jy3}
}

References

77 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Asher. 1998. Common ground, corrections and coordination. Journal of Semantics. find
  2. Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. 2016. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc, volume 1, page 3. find
  3. Alexandru Baltag, Lawrence S Moss, and Sławomir Solecki. 2016. The logic of public announcements, common knowledge, and private suspicions. Springer. Leonardo Bonacci. 1202. Liber Abaci. Unpublished lost manuscript. ↗
  4. Mariah Bradford, Ibrahim Khebour, Nathaniel Blanchard, and Nikhil Krishnaswamy. 2023. Automatic detection of collaborative states in small groups using multimodal features. In Proceedings of the 24th International Conference on Artificial Intelligence in Education. find
  5. Hennie Brugman and Albert Russel. 2004. Annotating multi-media/multi-modal resources with elan. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). find
  6. Richard Brutti, Lucia Donatelli, Kenneth Lai, and James Pustejovsky. 2022. Abstract Meaning Representation for gesture. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1576–1583, Marseille, France. European Language Resources Association. find
  7. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multi-WOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics. find
  8. Justine Cassell, Joseph Sullivan, Elizabeth Churchill, and Scott Prevost. 2000. Embodied conversational agents. MIT press. find
  9. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357. ↗ our card
  10. Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In AAAI Conference on Artificial Intelligence. find
  11. Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid Ó Séaghdha, and Anders Johannsen. 2020. Conversational semantic parsing for dialog state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8107– 8117, Online. Association for Computational Linguistics. ↗
  12. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Lauren Resnick, Levine B., M. John, Stephanie Teasley, and D, editors, Perspectives on Socially Shared Cognition, pages 13–1991. American Psychological Association. ↗
  13. Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2022. From rewriting to remembering: Common ground for conversational qa models. arXiv preprint arXiv:2204.03930. ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. Maria Di Maro, Antonio Origlia, and Francesco Cutugno. 2021. Cutting melted butter? common ground inconsistencies management in dialogue systems using graph databases. IJCoL. Italian Journal of Computational Linguistics, 7(7-1, 2):157–190. ↗
  16. Lee R Dice. 1945. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302. ↗
  17. Pierre Dillenbourg and David Traum. 2006. Sharing solutions: Persistence and grounding in multimodal collaborative problem solving. The Journal of the Learning Sciences, 15(1):121–151. ↗
  18. Eleni Efthimiou and Georgios Kouroupetroglou. 2011. Gestures in Embodied Communication and Human–Computer Interaction. find
  19. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. find
  20. Kerstin Fischer. 2011. How people talk with robots: Designing dialog to reduce user uncertainty. AI Magazine, 32(4):31–38. find
  21. Mary Ellen Foster. 2007. Enhancing human-computer interaction with embodied conversational agents. In International Conference on Universal Access in Human-Computer Interaction, pages 828–837. Springer. find
  22. Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 264–273, Stockholm, Sweden. Association for Computational Linguistics. find our card
  23. Jonathan Ginzburg. 1996. Interrogatives: Questions, facts and dialogue. The handbook of contemporary semantic theory. Blackwell, Oxford, pages 359–423. find
  24. Jonathan Ginzburg. 2012. The interactive stance: Meaning for conversation. Oxford University Press. find
  25. Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking. In Proc. Interspeech 2019, pages 1458–1462. find
  26. Susan Goldin-Meadow. 2003. Hearing Gesture: How Our Hands Help Us Think, volume 14. find
  27. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  28. Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. find
  29. Lauren V. Hadley, Graham Naylor, and Antonia F. de C. Hamilton. 2022. A review of theories and methods in the science of face-to-face social interaction. Nature Reviews Psychology, 1(1):42– 54. Number: 1 Publisher: Nature Publishing Group. ↗
  30. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034. find
  31. Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35–44, 1st virtual meeting. Association for Computational Linguistics. ↗
  32. Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIG-DIAL), pages 292–299, Philadelphia, PA, U.S.A. Association for Computational Linguistics. find our card
  33. Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2022. A simple language model for task-oriented dialogue. find
  34. Léo Jacqmin, Lina M Rojas-Barahona, and Benoit Favre. 2022. ” do you follow me?”: A survey of recent approaches in dialogue state tracking. arXiv preprint arXiv:2207.14627. ↗
  35. Adam Kendon. 1997. Gesture. Annual Review of Anthropology, 26(1):109–128. find
  36. Adam Kendon. 2004. Gesture: Visible action as utterance. ↗
  37. Ibrahim Khebour, Richard Brutti, Indrani Dey, Rachel Dickler, Kelsey Sikes, Kenneth Lai, Mariah Bradford, Brittany Cates, Paige Hansen, Changsoo Jung, Brett Wisniewski, Corbyn Terpstra, Leanne Hirshfield, Sadhana Puntambekar, Nathaniel Blanchard, James Pustejovsky, and Nikhil Krishnaswamy. 2023. The Weights Task Dataset: A Multimodal Dataset of Collaboration in a Situated Task. ↗
  38. Anthony Pak-Hin Kong, Sam Po Law, Connie Kwan, Christy Lai, and Vivian Lam. 2015. A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a database of speech and gesture (dosage). Journal of Nonverbal Behavior, 39. ↗
  39. Stefan Kopp and Ipke Wachsmuth. 2010. Gesture in embodied communication and human-computer interaction, volume 5934. Springer. find
  40. Nikhil Krishnaswamy and James Pustejovsky. 2019. Generating a novel dataset of multimodal referring expressions. In Proceedings of the 13th International Conference on Computational Semantics-Short Papers, pages 44–51. find
  41. Nikhil Krishnaswamy and James Pustejovsky. 2020. A formal analysis of multimodal referring strategies under common ground. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5919–5927. ↗ our card
  42. Geert-Jan M Kruijff, Pierre Lison, Trevor Benjamin, Henrik Jacobsson, Hendrik Zender, Ivana Kruijff-Korbayová, and Nick Hawes. 2010. Situated dialogue processing for human-robot interaction. In Cognitive systems, pages 311– 364. Springer. ↗
  43. Ang Li, Meghana Thotakuri, David A. Ross, João Carreira, Alexander Vostrikov, and Andrew Zisserman. 2020. The ava-kinetics localized human actions video dataset. ↗
  44. Xinmeng Li, Qian Li, Wansen Wu, and Quanjun Yin. 2021. Generation and extraction combined dialogue state tracking with hierarchical ontology integration. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2241–2249, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  45. Lizi Liao, Le Hong Long, Yunshan Ma, Wenqiang Lei, and Tat-Seng Chua. 2021. Dialogue state tracking with incremental reasoning. Transactions of the Association for Computational Linguistics, 9:557–569. ↗
  46. Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne. 2021. Knowledge-aware graph-enhanced GPT-2 for dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7871– 7881, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Magdalena Markowska, Adil Soubki, Gary Mar, Seyed Abolghasem Mirroshandel, Owen Rambow, and Anita Wasilewska. Formal representation of common ground in dialogue. find
  47. Paul Marshall and Eva Hornecker. 2013. Theories of embodiment in hci. The SAGE handbook of digital technology research, 1:144–158. ↗
  48. David Mcneill. 2005. Gesture and Thought. ↗
  49. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777–1788, Vancouver, Canada. Association for Computational Linguistics. find
  50. Xenia Ohmer, Marko Duda, and Elia Bruni. 2022. Emergence of hierarchical reference systems in multi-agent communication. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5689–5706. ↗
  51. Eric Pacuit. 2017. Neighborhood semantics for modal logic. Springer. ↗
  52. Jan Plaza. 1989. Logics of public communications. In Proceedings 4th International Symposium on Methodologies for Intelligent Systems, pages 201–216. find
  53. James Pustejovsky and Nikhil Krishnaswamy. 2016. Voxml: A visualization modeling language. Proceedings of LREC. find
  54. James Pustejovsky and Nikhil Krishnaswamy. 2021. Embodied human computer interaction. KI-Künstliche Intelligenz, 35(3-4):307–327. ↗
  55. Patrick Rohrer, Ingrid Vilà-Giménez, Júlia Florit-Pons, Núria Esteve-Gibert, Ada Ren, Stefanie Shattuck-Hufnagel, and Pilar Prieto. 2020. The multimodal multidimensional (m3d) labelling scheme for the annotation of audiovisual corpora. find
  56. Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. find
  57. Stefan Schaffer and Norbert Reithinger. 2019. Conversation is multimodal: thus conversational user interfaces should be as well. In Proceedings of the 1st International Conference on Conversational User Interfaces, pages 1–3. find
  58. Matthias Scheutz, Rehj Cantrell, and Paul Schermerhorn. 2011. Toward humanlike task-based dialogue processing for human robot interaction. Ai Magazine, 32(4):77–84. ↗
  59. David Schlangen and Gabriel Skantze. 2011. A general, abstract model of incremental dialogue processing. Dialogue & Discourse, 2(1):83– 111. ↗ our card
  60. LE Sigler. 2002. Fibonacci’s Liber Abaci. Leonardo Pisano’s Book of Calculation. New York: Springer. ↗
  61. Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. ↗
  62. Thorvald Sørensen. 1948. A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biologiske skrifter, 5:1–34. find
  63. Robert Stalnaker. 2002. Common ground. Linguistics and philosophy, 25(5-6):701–721. find
  64. Angela E. B. Stewart, Zachary Keirn, and Sidney K. D’Mello. 2021. Multimodal modeling of collaborative problem-solving facets in triads. User Modeling and User-Adapted Interaction, 31(4):713–751. ↗
  65. Chen Sun, Valerie J Shute, Angela Stewart, Jade Yonehiro, Nicholas Duran, and Sidney D’Mello. 2020. Towards a generalized competency model of collaborative problem solving. Computers & Education, 143:103672. ↗
  66. Christopher Tam, Richard Brutti, Kenneth Lai, and James Pustejovsky. 2023. Annotating situated actions in dialogue. In Proceedings of the 4th International Workshop on Designing Meaning Representation. find
  67. Corbyn Terpstra, Ibrahim Khebour, Mariah Bradford, Brett Wisniewski, Nikhil Krishnaswamy, and Nathaniel Blanchard. 2023. How good is automatic segmentation as a multimodal discourse annotation aid? In Workshop on Interoperable Semantic Annotation (ISA-19), page 75. ↗
  68. David Traum. 1994. A computational theory of grounding in natural language conversation. PhD thesis, University of Rochester. find
  69. David R Traum and Staffan Larsson. 2003. The information state approach to dialogue management. Current and new directions in discourse and dialogue, pages 325–353. ↗
  70. Jingxuan Tu, Kyeongmin Rim, Eben Holderness, Bingyang Ye, and James Pustejovsky. 2023. Dense paraphrasing for textual enrichment. In Proceedings of the 15th International Conference on Computational Semantics (IWCS), Nancy, France. Association for Computational Linguistics. ↗
  71. Jingxuan Tu, Kyeongmin Rim, and James Pustejovsky. 2022. Competence-based question generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1521–1533, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  72. Johan van Benthem, David Fernández-Duque, and Eric Pacuit. 2014. Evidence and plausibility in neighborhood structures. Annals of Pure and Applied Logic, 165(1):106–133. ↗
  73. Johan van Benthem and Eric Pacuit. 2011. Dynamic logics of evidence-based beliefs. Studia Logica, 99:61–92. ↗
  74. Wolfgang Wahlster. 2006. Dialogue systems go multimodal: The smartkom experience. In SmartKom: foundations of multimodal dialogue systems, pages 3–27. Springer. ↗
  75. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence, Italy. Association for Computational Linguistics. find
  76. Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020a. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 154–167, Barcelona, Spain (Online). Association for Computational Linguistics. ↗ our card
  77. Zheng Zhang, Ryuichi Takanobu, Qi Zhu, Min-Lie Huang, and XiaoYan Zhu. 2020b. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027. ↗

↑ back to top