Elamur Research⌕
☰
home / papers / 2019 / Executing Instructions in Situated Collaborative Interactions

Executing Instructions in Situated Collaborative Interactions

Paper · 2019 · Alane Suhr, Claudia Yan, Charlotte Schluger et al.

Summary

The paper looks at following natural language instructions in a setting where the person giving them also acts in the shared world. The authors build CEREALBAR, a turn-based 3D card game. A leader, who sees the whole map but has few moves, writes instructions for a follower, who has more moves but only a first-person view. Together they collect sets of three cards that differ in color, shape, and count. From 1,202 human-human games, they learn a follower policy.

The model extends the Visitation Prediction Network (VPN). It predicts several maps over positions: where to go, which cards to pick, what to avoid, and what is blocked. A recurrent action generator then turns these maps into moves. During training, the authors add examples where the agent starts from a wrong position caused by an earlier mistake, which teaches it to recover. They also propose cascaded evaluation, a way to score agents on recorded games that still captures how errors carry forward. In live games, human leaders adjusted their language and used the agent productively, though it remains far below human followers.

Key points

  • Dataset: 1,202 recorded human-human CEREALBAR games, released together with code and demo videos.
  • On test data, the full model reaches 58.4% card-state accuracy and 32.1% environment-state accuracy. The SEQ2SEQ + ATTN baseline reaches 37.3% and 10.8%.
  • Ablations show that the trajectory distribution, action recurrence, and joint fine-tuning matter most. Recovery-example aggregation improves results only when combined with the implicit reasoning discriminator.
  • With human leaders, the agent averaged 6.2 points (maximum 14), compared with 12.7 (maximum 20) for human followers.
  • Leaders simplified their instructions for the agent: 8.5 tokens on average versus 12.3 with humans, and 578 word types versus 1037.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🏢IntelSanta Clara · United States15 works · 41,257 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Alane Suhr, Claudia Yan, Charlotte Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang and Yoav Artzi. 2019. Executing Instructions in Situated Collaborative Interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1218 · arXiv 1910.03655

@inproceedings{suhr2019executing,
  title = {{Executing Instructions in Situated Collaborative Interactions}},
  author = {Alane Suhr, Claudia Yan, Charlotte Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang and Yoav Artzi},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1218},
  eprint = {1910.03655},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1218}
}

References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry S. Thompson, and Regina Weinert. 1991. The HCRC map task corpus. Language and Speech, 34. find
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In IEEE Conference on Computer Vision and Pattern R…
  3. Jacob Andreas, Anca Dragan, and Dan Klein. 2017. Translating neuralese. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 232–242. find
  4. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182.
  5. Alexander Broad, Jacob Arkin, Nathan Ratliff, Thomas Howard, and Brenna Argall. 2017. Real-time natural language corrections for assistive robotic manipulators. The International Journal of Robotics Research, 36(5-7):684–698. find
  6. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the National Conference on Artificial Intelligence. find
  7. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition. find
  8. Alex Djalali, David Clausen, Sven Lauer, Karl Schultz, and Christopher Potts. 2011. Modeling expert effects and common ground using questions under discussion. In AAAI Fall Symposium: Building Representations of Common Ground with Intelligent Agents. find
  9. Alex Djalali, Sven Lauer, and Christopher Potts. 2012. Corpus evidence for preference-driven interpretation. In Logic, Language and Meaning, pages 150– 159. find
  10. Jacob Arkin, Matthew R. Walter, Adrian Boteanu, Michael E. Napoli, Harel Biggie, Hadas Kress-Gazit, and Thomas M. Howard. 2017. Contextual awareness: Understanding monologic natural language instructions for autonomous robots. In IEEE International Symposium on Robot and Human Interactive Communicat… find
  11. Felix Duvallet, Thomas Kollar, and Anthony Stentz. 2013. Imitation learning for natural language direction following through unknown environments. In IEEE International Conference on Robotics and Automation, pages 1047–1053. find
  12. Yoav Artzi, Dipanjan Das, and Slav Petrov. 2014. Learning compact lexicons for CCG semantic parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1273–1283. find
  13. Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. 2017. Emergent communication in a multi-modal, multi-step referential game. In Proceedings of the International Conference on Learning Representations. find
  14. Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 421–432. find
  15. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1951–1963.
  16. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association of Computational Linguistics, 1:49–62.
  17. Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi. 2018a. Following high-level navigation instructions on a simulated quadcopter with imitation learning. In Proceedings of the Robotics: Science and Systems Conference. find
  18. Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. 2018b. Mapping navigation instructions to continuous control actions with position visitation prediction. In Proceedings of the Conference on Robot Learning. find
  19. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14:179–211. find
  20. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1766–1776. find
  21. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2333–2343.
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9. find
  23. Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. CoRR, abs/906.00744. find
  24. Nikolai Ilinykh, Sina Zarrieß, and David Schlangen. 2019. MeetUp! A corpus of joint activity dialogues in a visual environment. CoRR, abs/1907.05084. — [arXiv:1907.05084](https://arxiv.org/abs/1907.05084) find
  25. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1821–1831.
  26. Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication. CoRR, abs/1704.04517.
  27. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
  28. Theodora Koulouri and Stanislao Lauria. 2009. Exploring miscommunication and collaborative behaviour in human-robot interaction. In Proceedings of the SIGDIAL Conference, pages 111–119. find
  29. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2443–2453.
  30. Claire Liang, Julia Proft, Erik Andersen, and Ross A. Knepper. 2019. Implicit communication of actionable information in human-AI teams. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 95:1–95:13. find
  31. Karen E. Lochbaum. 1998. A collaborative planning model of intentional structure. Computational Linguistics, 24(4):525–572.
  32. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1456–1465. find
  33. Matthew MacMahon, Brian Stankiewics, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, action in route instructions. In Proceedings of the National Conference on Artificial Intelligence. find
  34. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  35. Scott Miller, David Stallard, Robert Bobrow, and Richard Schwartz. 1996. A fully statistical approach to natural language interfaces. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 55–61. find
  36. Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2667–2678. find
  37. Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1004–1015. find
  38. Christopher Potts. 2012. Goal-driven answers in the Cards dialogue corpus. In Proceedings of the West Coast Conference on Formal Linguistics, pages 1– 20. find
  39. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics. find
  40. Candace L. Sidner, Carolyn Boettner, and Charles Rich. 2000. Lessons learned in building spoken language collaborative interface agents. In ANLP-NAACL Workshop: Conversational Systems. find
  41. Alane Suhr and Yoav Artzi. 2018. Situated mapping of sequential instructions to actions with single-step reward observation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2072–2082. find
  42. Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2238–2249. find
  43. Stephanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence. find
  44. Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the International Joint Conference on Artificial Intelligence. find
  45. Takuma Udagawa and Akiko Aizawa. 2019. A natural language corpus of common grounding under continuous and partially-observable context. In Proceedings of the Conference on Artificial Intelligence.
  46. Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. 2017. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4105– 4113. find
  47. Andreas Vlachos and Stephen Clark. 2014. A new corpus and imitation learning framework for context-dependent semantic parsing. Transactions of the Association for Computational Linguistics, 2:547–560. find
  48. Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the Walk: Navigating New York City through grounded dialogue. arXiv preprint arXiv:1807.03367. find
  49. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2368–2378. find
  50. Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL Conference, pages 404–413. find
  51. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1:270–280. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suhr-2019-executing-instructions-situated-collaborative-interactions/