Elamur Research⌕
☰
home / papers / 2021 / Maintaining Common Ground in Dynamic Environments

Maintaining Common Ground in Dynamic Environments

Summary

Most grounded dialogue benchmarks test whether agents can establish shared understanding of a fixed scene. They largely overlook keeping that understanding up to date as the world changes. Udagawa and Aizawa (TACL, 2021) address this gap with a sequential collaborative reference task that extends the OneCommon setup over time. Two agents each see part of a 2-D plane with dots whose color, size and position are continuous values. The dots move, and the agents' views shift between turns. At every turn the partners must agree on one shared dot. The game ends at the first mismatch, so the number of consecutive successful turns measures how well common ground is maintained. Using crowdsourcing, the authors built Dynamic-OneCommon Corpus (D-OCC). Their analysis shows people rely on rich spatio-temporal descriptions and refer back to earlier agreements. A neural baseline handles spatial cues and keeps a previously shared referent, but makes poor use of temporal information and performs worse with real humans than in selfplay. The resource offers a controlled testbed for dialogue systems in changing environments.

Key points

  • D-OCC contains 5,617 dialogues with more than 65K utterances, collected via Amazon Mechanical Turk. Code and data are at github.com/Alab-NII/dynamic-onecommon.
  • Evaluation separates two skills. Success at the first selection reflects creating common ground. The length of successful timesteps (LST) reflects maintaining it.
  • Entity movements are synthetic and fully controllable. This is meant to reduce biases, increase diversity and allow causal probing of model behavior.
  • The baseline builds on Udagawa and Aizawa (2020). It retains common ground best when given an explicit meta feature. It struggles with temporal reasoning and drops notably when paired with human partners.
  • Proposed future work includes:
    • reference and spatio-temporal expression annotation;
    • reinforcement learning through selfplay;
    • pragmatic reasoning (RSA);
    • video and vision-language pretraining.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00409 · arXiv 2105.14207

@article{udagawa2021maintaining,
  title = {{Maintaining Common Ground in Dynamic Environments}},
  author = {Udagawa, Takuma and Aizawa, Akiko},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00409},
  eprint = {2105.14207},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00409}
}

References

59 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nayyer Aafaq, Ajmal Mian, Wei Liu, Syed Zulqarnain Gilani, and Mubarak Shah. 2019. Video description: A survey of methods, datasets, and evaluation metrics. ACM Computing Surveys, 52(6):1–37. https://doi.org/10.1145/3355390 ↗
  2. Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, and Devi Parikh. 2019. Audio visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7558–7567. https://doi.org/10.1109/CVPR.2019 .00774 ↗
  3. Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. PHYRE: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems, pages 5082–5093. ↗
  4. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198. https://doi.org/10 .18653/v1/2020.acl-main.463 ↗ our card
  5. Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixé. 2019. Tracking without bells and whistles. In International Conference on Computer Vision. https://doi.org/10.1109/ICCV.2019.00103 ↗
  6. Pierre Bézier. 1974. Mathematical and practical possibilities of UNISURF. In Computer Aided Geometric Design, pages 127–152. Elsevier. https://doi.org/10.1016/B978-0-12 -079050-0.50012-6 ↗
  7. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22:1482–93. https://doi.org/10.1037/0278-7393 .22.6.1482 ↗
  8. Susan E. Brennan, Alexia Galati, and Anna K. Kuhlen. 2010. Two minds, one dialog: Coordinating speaking and understanding. In Psychology of Learning and Motivation, volume 53, pages 301–344. Elsevier. https://doi.org/10.1016/S0079-7421(10)53008-1 ↗
  9. J. Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4724–4733. find
  10. Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, and Rada Mihalcea. 2020. LifeQA: A real-life dataset for video question answering. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4352–4358. find
  11. Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee Kenneth Wong. 2019. Weakly-supervised spatio-temporally grounding natural sentence in video. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1884–1894. https://doi.org/10.18653/v1/P19-1183 ↗
  12. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111. find our card
  13. Herbert H. Clark. 1996. Using Language. Cambridge University Press. https://doi.org/10.1037/10096-006 ↗
  14. Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Perspectives on Socially Shared Cognition, pages 127–149. American Psychological Association. ↗
  15. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213. https://doi.org/10.1037/h0026256, Pubmed: 19673146 ↗
  16. Harm De Vries, Dzmitry Bahdanau, and Christopher Manning. 2020. Towards ecologi-cally valid research on language user interfaces. arXiv preprint arXiv:2007.14435. for asymmetries in visual perspective. Cognitive Science, 45 3:e12926. https://doi.org/10.1111/cogs.12926, Pubmed: 33686646 ↗
  17. Harm De Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367. ↗
  18. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1766–1776. https://doi.org/10.18653/v1/P17 -1162 ↗
  19. Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! Visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503–5512. https://doi.org/10 .1109/CVPR.2017.475 ↗
  20. Rui Fang, Malcolm Doering, and Joyce Y. Chai. 2015. Embodied collaborative referring expression generation in situated human-robot interaction. In Proceedings of the Tenth Annual ACM/ IEEE International Conference on Human-Robot Interaction, HRI ’15, pages 271–278. https://doi.org/10.1145/2696454 .2696467 ↗
  21. Rohit Girdhar and Deva Ramanan. 2020. CATER: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations. ↗
  22. Noah D. Goodman and Michael C. Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20:818–829. https://doi.org/10 .1016/j.tics.2016.08.005, Pubmed: 27692852 ↗
  23. H. Paul Grice. 1975. Logic and conversation. Syntax and Semantics, 3:41–58. ↗
  24. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910. ↗ our card
  25. Robert X. D. Hawkins, H. Gweon, and Noah D. Goodman. 2021. The division of labor in communication: Speakers help listeners account Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  26. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017a. Dense-captioning events in videos. In International Conference on Computer Vision, pages 706–715. https://doi.org/10.1109/ICCV.2017.83 ↗
  27. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017b. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73. https://doi.org/10.1007/s11263-016-0981-7 ↗ our card
  28. Hung Le, Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2020. BiST: Bi-directional spatio-temporal reasoning for video-grounded dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1846–1859. find
  29. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369–1379. https://doi.org/10 .18653/v1/D18-1167 ↗ our card
  30. David Lewis. 1969. Convention: A Philosophical Study. Harvard University Press. find
  31. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453. https://doi.org/10 .18653/v1/D17-1259 ↗
  32. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pages 13–23. find our card
  33. Will Monroe, Robert X. D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. https://doi.org/10.1162/tacl_a_00064 ↗ our card
  34. Seungwhan Moon, Satwik Kottur, Paul Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, and Alborz Geramifard. 2020. Situated and interactive multimodal conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1103–1121. find
  35. Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405–5415. https://doi.org/10.18653/v1/P19 -1537 ↗
  36. Carita Paradis. 2008. Configurations, construals and change: Expressions of DEGREE. English Language and Linguistics, 12(2): 317–343. https://doi.org/10.1017/S1360674308002645 ↗
  37. Ramakanth Pasunuru and Mohit Bansal. 2018. Game-based video-context dialogue. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 125–136. https://doi.org/10 .18653/v1/D18-1012 ↗
  38. Massimo Poesio and Hannes Rieser. 2010. Completions, coordination, and alignment in dialogue. Dialogue and Discourse, 1:1–89. https://doi.org/10.5087/dad.2010 .001 ↗
  39. Arka Sadhu, Kan Chen, and Ram Nevatia. 2020. Video object grounding using semantic roles in language description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10417–10427. https://doi.org/10.1109/CVPR42600 .2020.01043 ↗
  40. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pages 4967–4976. ↗ our card
  41. David Schlangen. 2019. Grounded agreement games: Emphasizing conversational grounding in visual dialogue settings. arXiv preprint arXiv:1908.11279. ↗
  42. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2556–2565. https://doi.org/10 .18653/v1/P18-1238 ↗ our card
  43. Robert C Stalnaker. 1978. Assertion. Syntax and Semantics, 9:315–332. find
  44. Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2119–2130. https://doi.org/10.18653/v1/D19 -1218 ↗
  45. Ece Takmaz, Mario Giulianelli, Sandro Pezzelle, Arabella Sinclair, and Raquel Fernández. 2020. Refer, reuse, reduce: Generating subsequent references in visual and conversational contexts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4350–4368. https://doi.org/10.18653/v1/2020.emnlp-main .353 ↗
  46. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394–406. find
  47. David R. Traum. 1994. A Computational Theory of Grounding in Natural Language Conversation. Ph.D. thesis, Department of Computer Science, University of Rochester. find
  48. Takuma Udagawa and Akiko Aizawa. 2019. A natural language corpus of common grounding under continuous and partially-observable context. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7120–7127. https://doi.org/10.1609/aaai.v33i01 .33017120 ↗
  49. Takuma Udagawa and Akiko Aizawa. 2020. An annotated corpus of reference resolution for interpreting common grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9081–9089. https://doi.org/10.1609/aaai.v34i05.6442 ↗
  50. Takuma Udagawa, Takato Yamazaki, and Akiko Aizawa. 2020. A linguistic analysis of visually grounded dialogues based on spatial expressions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 750–765. https://doi.org/10 .18653/v1/2020.findings-emnlp.67 ↗
  51. Hans Van Ditmarsch, Wiebe van Der Hoek, and Barteld Kooi. 2007. Dynamic Epistemic Logic, volume 337. Springer Science & Business Media. https://doi.org/10.1007/978 -1-4020-5839-4 ↗
  52. Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803. https://doi.org/10.1109/CVPR.2018.00813 ↗
  53. Zhongdao Wang, Liang Zheng, Yixuan Liu, and Shengjin Wang. 2020. Towards real-time multi-object tracking. In European Conference on Computer Vision. https://doi.org/10.1007/978-3-030-58621-8_7 ↗
  54. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pages 2048–2057. ↗ our card
  55. Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dialogue. In Proceedings of the International Conference on Machine Learning, pages 5587–5595. ↗
  56. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2020. CLEVRER: Collision events for video representation and reasoning. In International Conference on Learning Representations. ↗
  57. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. ActivityNet-QA: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134. https://doi.org/10.1609/aaai.v33i01.33019127 ↗ our card
  58. Sina Zarrieß, Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David DeVault, Raquel Fernández, and David Schlangen. 2016. PentoRef: A corpus of spoken references in task-oriented dialogues. In Proceedings of the 10th Language Resources and Evaluation Conference, pages 125–131. find
  59. Luowei Zhou, Nathan Louis, and Jason J. Corso. 2018. Weakly-supervised video object grounding from text by loss weighting and object interaction. In British Machine Vision Conference. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/udagawa-2021-maintaining-common-ground-dynamic-environments/