Elamur Research⌕
☰
home / papers / 2018 / Representation Learning for Grounded Spatial Reasoning

Representation Learning for Grounded Spatial Reasoning

Paper · 2018 · Michael Janner, Karthik Narasimhan and Regina Barzilay

Summary

Spatial expressions in instructions only make sense relative to a particular environment, so a system must reason over language and map layout together. The authors study this in simulated 2D grid worlds where an agent is rewarded only when it reaches the place an instruction describes. It receives no demonstrations, object mappings or spatial ontology. Their model encodes the instruction with an LSTM and splits the resulting vector into two parts. One part serves as a convolution kernel over learned state embeddings, tying words to local neighborhoods. The other part weights gradient functions to capture map-wide references such as superlatives. A CNN merges both outputs into a predicted value map, trained by reinforcement learning with a generalized value iteration scheme. The authors also release Puddle World Navigation, a dataset of more than 3,300 tasks on 200 maps with crowdsourced instructions. Against adapted versions of UVFA and the Misra et al. (2017) architecture, the model cuts goal-localization error by 45% and transfers well to unseen layouts. This is relevant to robot navigation and human–robot communication.

Key points

  • New dataset: Puddle World Navigation contains 3,300+ tasks over 200 maps, with instructions collected via Mechanical Turk. Code and data are on GitHub (JannerM/spatial-reasoning).
  • Instruction types: Instructions refer to a single entity, one referent or several referents. Each type can use local relations (e.g., "above") or global ones (e.g., "easternmost").
  • Main result: Manhattan distance error is 45% lower than the strongest baseline. The model reaches 0.90 policy quality on local instructions after only 400 training samples.
  • Bisk et al.'s (2016) ISI benchmark for grounding language: The task was adapted with an oracle choosing the moving block and a 13 × 13 grid. Here the model scores 0.74 policy quality and 3.94 distance, versus 0.17/4.11 for CNN + LSTM and 0.15/4.61 for UVFA (text).
  • Weaknesses and side finding: Instructions with multiple levels of indirection and long, redundant instructions remain hard. On ISI, subgoals that come later in a task are predicted more accurately, which suggests language matters less once the structure is mostly built.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksAutomotive Engineering · 6 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Michael Janner, Karthik Narasimhan and Regina Barzilay. 2018. Representation Learning for Grounded Spatial Reasoning. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00004

@article{janner2018representation,
  title = {{Representation Learning for Grounded Spatial Reasoning}},
  author = {Michael Janner, Karthik Narasimhan and Regina Barzilay},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00004},
  url = {https://doi.org/10.1162/tacl_a_00004}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In EMNLP. find
  2. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. TACL, 1(1):49–62.
  3. Richard Bellman. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1 edition. find
  4. Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016. Natural language communication with robots. In NAACL HLT, pages 751–761. find
  5. S.R.K. Branavan, David Silver, and Regina Barzilay. 2011. Learning to win by reading manuals in a Monte-Carlo framework. In ACL HLT, pages 268–277. find
  6. Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling. 2011. Amazon’s Mechanical Turk. Perspectives on Psychological Science, 6(1):3–5. PMID: 26162106. find
  7. Ruth M.J. Byrne and Philip N. Johnson-Laird. 1989. Spatial reasoning. Journal of memory and language, 28(5):564–575. find
  8. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions fro mobservations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011), pages 859–865, San Francisco, CA, USA, August. find
  9. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. ABC-CNN: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960. — [arXiv:1511.05960](https://arxiv.org/abs/1511.05960) find
  10. William T. Freeman and Edward H. Adelson. 1991. The design and use of steerable filters. IEEE TPAMI, 13(9):891–906. find
  11. Sachithra Hemachandra, Matthew R. Walter, Stefanie Tellex, and Seth Teller. 2014. Learning spatial-semantic representations from natural language descriptions and scene classifications. In ICRA, pages 2623–2630. IEEE. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural comp… find
  12. John D. Kelleher, Geert-Jan M. Kruijff, and Fintan J. Costello. 2006. Proximity in context: an empirically grounded computational model of proximity for processing topological spatial expressions. In ACL, pages 745–752. find
  13. Joohyun Kim and Raymond J. Mooney. 2013. Adapting discriminative reranking to grounded language learning. In ACL (1), pages 218–227. find
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  15. Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. 2010. Toward understanding natural language directions. In Human-Robot Interaction, pages 259– 266. find
  16. IEEE. Stephen C Levinson. 2003. Space in language and cognition: Explorations in cognitive diversity, volume 5. find
  17. Cambridge University Press. Michael Levit and Deb Roy. 2007. Interpretation of spatial language in a map navigation task. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 37(3):667–679. find
  18. Peggy Li and Lila Gleitman. 2002. Turning the tables: Language and spatial reasoning. Cognition, 83(3):265–294. find
  19. Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. AAAI, 2(6):4. find
  20. Daniel J. Mankowitz, Timothy Arthur Mann, and Shie Mannor. 2016. Iterative hierarchical optimization for misspecified problems (IHOMP). CoRR, abs/1602.03348. — [arXiv:1602.03348](https://arxiv.org/abs/1602.03348) find
  21. Dipendra K. Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. find
  22. EMNLP. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and… find
  23. Reinhard Moratz and Thora Tenbrink. 2006. Spatial reference in linguistic human-robot interaction: Iterative, empirically supported development of a model of projective relations. Spatial cognition and computation, 6(1):63–107. find
  24. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. 2015. Universal value function approximators. In ICML, pages 1312–1320. find
  25. M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams, M. Bugajska, and D. Brock. 2004. Spatial language for human-robot dialogs. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 34(2):154–167, May. find
  26. Richard S. Sutton and Andrew G. Barto. 1998. Introduction to reinforcement learning. find
  27. MIT Press. Richard S. Sutton. 1996. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In NIPS. find
  28. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J. Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI. find
  29. Thora Tenbrink. 2007. Space, Time, and the Use of Language: An Investigation of Relationships, volume 36. find
  30. Walter de Gruyter. Jette Viethen and Robert Dale. 2008. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference, pages 59–67. find
  31. ACL. Adam Vogel and Dan Jurafsky. 2010. Learning to follow navigational directions. In ACL, pages 806–814. find
  32. Matthew R. Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. 2013. Learning semantic maps from natural language descriptions. Proceedings of the 2013 Robotics: Science and Systems IX Conference. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/janner-2018-representation-learning-grounded-spatial-reasoning/