Elamur Research⌕
☰
home / papers / 2020 / A Formal Analysis of Multimodal Referring Strategies Under Common Ground

A Formal Analysis of Multimodal Referring Strategies Under Common Ground

Summary

The paper examines how gesture and speech combine when a computer agent singles out an object for a human, and how each modality changes the knowledge the two parties share. The authors revisit their Embodied Multimodal Referring Expressions (EMRE) dataset. In it, a virtual avatar in a simulated tabletop scene identifies a target object in one of three ways: pointing, verbal description, or both together (an "ensemble"). Crowdworkers on Amazon Mechanical Turk rated how natural each strategy seemed. The authors formalize common ground as a state monad with four parts: agents, shared beliefs, jointly perceived objects and relations, and the embedding space. They then treat each stage of a video (showing the scene, marking the target, pointing, describing) as an epistemic update in the spirit of public announcement logic. They argue that these formal update properties are useful features for predicting how viewers judge referring expressions. They also suggest the properties could guide the generation of more natural and informative multimodal references. This is relevant for human–computer and human–robot interaction.

Key points

  • EMRE is public (GitHub, VoxML/public-data). Each video was rated by eight annotators on a 1–5 naturalness scale. Scene and expression parameters are stored in a SQL database, including target identity, distance to the agent, modality, utterance, relational descriptors, object coordinates, and the full relation set.
  • The common ground model has four components: A (avatar and human observer), B (shared beliefs), P (perceived items) and E (embedding space). P includes a table, cup, knife, plate, and two blocks each of purple, red and green.
  • The updates unfold in three steps. At t0, the displayed objects enter P through transitive closure of co-perception. At t1, a circle marks the target, so the human knows it but not that the avatar knows it. At t2, pointing and describing publicly reveal the avatar's knowledge of the target, the meaning of her terms, and her spatial frame of reference.
  • The semantics of a referring expression include a separate continuation for each modality (speech k_s, gesture k_g), which carries the referent into later dialogue moves.
  • The provided excerpt omits the analysis and results sections, so no quantitative model scores are available for this summary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2003.07385

@inproceedings{krishnaswamy2020formal,
  title = {{A Formal Analysis of Multimodal Referring Strategies Under Common Ground}},
  author = {Krishnaswamy, Nikhil and Pustejovsky, James},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  eprint = {2003.07385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.07385}
}

References

49 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alahverdzhieva, K., Flickinger, D., and Lascarides, A. (2012). Multimodal grammar implementation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 582–586, Montréal, Canada, June. Association for… find
  2. Alahverdzhieva, K., Lascarides, A., and Flickinger, D. (2017). Aligning speech and co-speech gesture in a constraint-based grammar. Journal of Language Modelling, 5. find
  3. Asher, N. (1998). Common ground, corrections and coordination. Journal of Semantics. find
  4. Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., and Frank, S. (2018). Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304–323, Belgium, Brussels, October. Association for Comp… find our card
  5. Beinborn, L., Botschen, T., and Gurevych, I. (2018). Multimodal grounding for language processing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2325–2339, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. find
  6. Bortfeld, H. and Brennan, S. E. (1997). Use and acquisition of idiomatic expressions in referring by native and non-native speakers. Discourse Processes, 23(2):119– 147. find
  7. Claassen, W. (1992). Generating referring expressions in a multimodal environment. In Aspects of automated natural language generation, pages 247–262. find
  8. Springer. Clark, H. H. and Brennan, S. E. (1991). Grounding in communication. In Lauren Resnick, et al., editors, Perspectives on Socially Shared Cognition, pages 13–1991. find
  9. American Psychological Association. Cohen, V., Burchfiel, B., Nguyen, T., Gopalan, N., Tellex, S., and Konidaris, G. (2019). Grounding language attributes to objects using bayesian eigenobjects. arXiv preprint arXiv:1905.13153. — [arXiv:1905.13153](https://arxiv.org/abs/1905.13153) find
  10. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314. find
  11. Dale, R. and Reiter, E. (1995). Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive science, 19(2):233–263. find our card
  12. Davis, E. and Marcus, G. (2016). The scope and limits of simulation in automated reasoning. Artificial Intelligence, 233:60–72. Drijvers, L., Özyürek, A., and Jensen, O. (2018). Hearing and seeing meaning in noise: Alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech… find
  13. Funakoshi, K., Watanabe, S., Kuriyama, N., and Tokunaga, T. (2004). Generating referring expressions using perceptual groups. In International Conference on Natural Language Generation, pages 51–60. find
  14. Springer. Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017). The” something something” video database for learning and evaluating visual common sense. In ICCV, volume 1, page 3. find
  15. Grice, H. P., Cole, P., Morgan, J., et al. (1975). Logic and conversation. 1975, pages 41–58. find
  16. Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J. A., and Smith, I. (1997). Unification-based multimodal integration. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, pages 281–288. Association for Computational Linguistics… find
  17. Kasenberg, D., Roque, A., Thielstrom, R., Chita-Tegmark, M., and Scheutz, M. (2019). Generating justifications for norm-related agent decisions. In 12th International Conference on Natural Language Generation. find
  18. Kim, J.-H., Jo, S., and Lattimer, B. Y. (2016). Feature selection for intelligent firefighting robot classification of fire, smoke, and thermal reflections using thermal infrared images. Journal of Sensors, 2016. find
  19. Krahmer, E. and Van Deemter, K. (2012). Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218. find
  20. Krahmer, E. and van der Sluis, I. (2003). A new model for generating multimodal referring expressions. In Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003. find
  21. Krishnaswamy, N. and Pustejovsky, J. (2016a). Multimodal semantic simulations of linguistically underspecified motion events. In Spatial Cognition X: International Conference on Spatial Cognition. find
  22. Springer. Krishnaswamy, N. and Pustejovsky, J. (2016b). VoxSim: A visual platform for modeling motion language. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. find
  23. ACL. Krishnaswamy, N. and Pustejovsky, J. (2019a). Generating a novel dataset of multimodal referring expressions. In Proceedings of the 13th International Conference on Computational Semantics-Short Papers, pages 44–51. find
  24. Krishnaswamy, N. and Pustejovsky, J. (2019b). Multimodal continuation-style architectures for human-robot interaction. arXiv preprint arXiv:1909.08161. — [arXiv:1909.08161](https://arxiv.org/abs/1909.08161) find
  25. Lascarides, A. and Stone, M. (2009). A formal semantic analysis of gesture. Journal of Semantics, page ffp004. find
  26. Magassouba, A., Sugiura, K., and Kawai, H. (2019). Multimodal attention branch network for perspective-free sentence generation. arXiv preprint arXiv:1909.05664. — [arXiv:1909.05664](https://arxiv.org/abs/1909.05664) find
  27. Matuszek, C., Bo, L., Zettlemoyer, L., and Fox, D. (2014). Learning from unscripted deictic gesture and language for human-robot interactions. In AAAI, pages 2556– 2563. find
  28. McNeill, D. (2000). Language and gesture, volume 2. find
  29. Cambridge University Press. Paraboni, I., Van Deemter, K., and Masthoff, J. (2007). Generating referring expressions: Making referents easy to identify. Computational linguistics, 33(2):229–254. Petit, M., Lallée, S., Boucher, J.-D., Pointeau, G., Chem-inade, P., Ognibene, D., Chinellato, E., Patta… find
  30. Plaza, J. (2007). Logics of public communications. Synthese, 158(2):165–179. find
  31. Pustejovsky, J. and Krishnaswamy, N. (2014). Generating simulations of motion events from verbal descriptions. Lexical and Computational Semantics (* SEM 2014), page 99. find
  32. Pustejovsky, J. and Krishnaswamy, N. (2016). VoxML: A visualization modeling language. Proceedings of LREC. find
  33. Pustejovsky, J., Krishnaswamy, N., Draper, B., Narayana, P., and Bangar, R. (2017). Creating common ground through multimodal simulations. In Proceedings of the IWCS workshop on Foundations of Situated and Multimodal Communication. find
  34. Pustejovsky, J. (2018). From actions to events. Interaction Studies, 19(1-2):289–317. find
  35. Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K. E., and Ansari, R. (2002). Multimodal human discourse: gesture and speech. ACM Transactions on Computer-Human Interaction (TOCHI), 9(3):171–193. find
  36. Schlenker, P. (2018). Gesture projection and cosupposi-tions. Linguistics and Philosophy, 41(3):295–365. find
  37. Shridhar, M. and Hsu, D. (2018). Interactive visual grounding of referring expressions for human-robot interaction. arXiv preprint arXiv:1806.03831. — [arXiv:1806.03831](https://arxiv.org/abs/1806.03831) find
  38. Stalnaker, R. (2002). Common ground. Linguistics and philosophy, 25(5-6):701–721. find
  39. Tomasello, M. and Carpenter, M. (2007). Shared intentionality. Developmental science, 10(1):121–125. find
  40. Van Benthem, J. (2011). Logical dynamics of information and interaction. find
  41. Cambridge University Press. Van Der Sluis, I. and Krahmer, E. (2001). Generating referring expressions in a multimodal context: An empirically oriented approach. In Computational Linguistics in the Netherlands 2000, pages 158–176. find
  42. Brill Rodopi. Van Ditmarsch, H., van Der Hoek, W., and Kooi, B. (2007). Dynamic epistemic logic, volume 337. find
  43. Springer Science & Business Media. Van Eijck, J. and Unger, C. (2010). Computational semantics with functional programming. find
  44. Cambridge University Press. Viethen, J. and Dale, R. (2008). The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference, pages 59–67. Association for Computational Linguistics. find
  45. Wallbridge, C. D., Lemaignan, S., Senft, E., and Belpaeme, T. (2019). Generating spatial referring expressions in a social robot: Dynamic vs non-ambiguous. Frontiers in Robotics and AI, 6:67. find
  46. Whitney, D., Eldon, M., Oberlin, J., and Tellex, S. (2016). Interpreting multimodal referring expressions in real time. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pages 3331–3338. find
  47. IEEE. Yatskar, M., Zettlemoyer, L., and Farhadi, A. (2016). Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5534–5542. find
  48. Zhang, X., Pacheco, M. L., Li, C., and Goldwasser, D. (2016). Introducing drail–a step towards declarative deep relational learning. In Proceedings of the Workshop on Structured Prediction for NLP, pages 54–62. find
  49. Zhou, M., Cheng, R., Lee, Y. J., and Yu, Z. (2018). A visual attention grounding neural model for multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3643–3653, Brussels, Belgium, October-November. Association for Computati… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/krishnaswamy-2020-formal-analysis-multimodal-referring-strategies/