Elamur Research
☰
home / papers / 2017 / Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1711.07280

Summary

The authors formulate the Vision-and-Language Navigation (VLN) task. An agent receives an instruction in natural language and must reach a goal in a building it has never seen before, relying only on images from its own camera. To make such a task reproducibly evaluable, they created the Matterport3D Simulator. It is a reinforcement learning environment built on real panoramic images from Matterport3D. In it, the agent moves between discrete viewpoints along a graph of navigable paths. Based on the simulator, the Room-to-Room (R2R) dataset was collected: more than 21 thousand crowdworker instructions, each describing a route through several rooms. The authors took the baseline model from related tasks (VQA, image captioning): it is a sequence-to-sequence network, and it trained best in student-forcing mode. In familiar buildings the model reaches the goal in about 39% of cases, in new ones only 21%, while humans achieve over 86%. This is the first benchmark for navigation in real indoor environments that is rich both visually and linguistically. It showed that the main problem is generalization to new environments.

Key points

  • Matterport3D Simulator uses 10 800 panoramas (assembled from 194 400 RGB-D images) of 90 real buildings: houses, apartments, hotels, offices, churches. The navigation graph was built by ray tracing between viewpoints. Edges longer than 5 m were discarded, and each graph was checked manually.
  • R2R contains 21 567 open-vocabulary instructions, 29 words per instruction on average. The simulator, dataset and baseline models are published at bringmeaspoon.org.
  • Success is defined as stopping within 3 m of the goal. On the test split, seq2seq with student-forcing achieves 20.4% success, the RANDOM baseline 13.2%, and humans from AMT 86.4% (measured on 1 390 instructions).
  • There is a large gap between familiar and new environments: 38.6% versus 21.8% on validation. Even with dropout and weight decay, the model overfits to the training buildings.
  • Student-forcing gives a better result than teacher-forcing, but takes longer to train because the agent explores a larger part of the environment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

190 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 177 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.07280 · arXiv 1711.07280

@misc{anderson2017vision,
  title = {{Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments}},
  author = {Anderson, Peter and Wu, Qi and Teney, Damien and Bruce, Jake and Johnson, Mark and Sünderhauf, Niko and Reid, Ian and Gould, Stephen and Hengel, Anton van den},
  year = {2017},
  doi = {10.48550/arxiv.1711.07280},
  eprint = {1711.07280},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.07280}
}

References

62 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016. 4 find our card
  2. P. Ammirato, P. Poirson, E. Park, J. Kosecka, and A. C. Berg. A dataset for developing and benchmarking active vision. In ICRA, 2017. 3 find
  3. P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. 7 find
  4. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. 2, 4, 5 find
  5. I. Armeni, A. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. arXiv preprint arXiv:1702.01105, 2017. 4 — [arXiv:1702.01105](https://arxiv.org/abs/1702.01105) find
  6. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 6 find our card
  7. C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016. 1, 2, 3 find
  8. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015. 7 find
  9. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016. 4 find
  10. S. Brodeur, E. Perez, A. Anand, F. Golemo, L. Celotti, F. Strub, J. Rouat, H. Larochelle, and A. Courville. HoME: A household multimodal environment. arXiv:1711.11017, 2017. 3 — [arXiv:1711.11017](https://arxiv.org/abs/1711.11017) find
  11. A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017. 1, 2, 3, 5, 6 find
  12. D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov. Gated-attention architectures for task-oriented language grounding. arXiv preprint arXiv:1706.07230, 2017. 1, 2, 3 — [arXiv:1706.07230](https://arxiv.org/abs/1706.07230) find
  13. D. L. Chen and R. J. Mooney. Learning to interpret natural language navigation instructions from observations. In AAAI, 2011. 1, 2 find
  14. X. Chen, T.-Y. L. Hao Fang, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv preprint arXiv:1504.00325, 2015. 2, 4 find our card
  15. A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 3 find
  16. A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied Question Answering. In CVPR, 2018. 2, 4 find
  17. A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra. Visual dialog. In CVPR, 2017. 2 find
  18. D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. IQA: Visual question answering in interactive environments. In CVPR, 2018. 2, 4 find
  19. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR, 2017. 2 find our card
  20. S. Guadarrama, L. Riano, D. Golland, D. Go, Y. Jia, D. Klein, P. Abbeel, T. Darrell, et al. Grounding spatial relations for human-robot interaction. In IROS, 2013. 1, 2 find
  21. S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017. 3 find
  22. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 6 find
  23. S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 1997. 6 find
  24. A. S. Huang, S. Tellex, A. Bachrach, T. Kollar, D. Roy, and N. Roy. Natural language command of an autonomous micro-air vehicle. In IROS, 2010. 2 find
  25. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 4 find our card
  26. S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referit game: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 2 find
  27. M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski. ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, 2016. 1, 2, 3 find
  28. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7 find
  29. T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on, pages 259–266. IEEE, 2010. 2 find
  30. E. Kolve, R. Mottaghi, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. AI2-THOR: An interactive 3d environment for visual AI. arXiv:1712.05474, 2017. 3 find
  31. T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016. 3 — [arXiv:1606.02396](https://arxiv.org/abs/1606.02396) find
  32. A. M. Lamb, A. G. A. P. GOYAL, Y. Zhang, S. Zhang, A. C. Courville, and Y. Bengio. Professor forcing: A new algorithm for training recurrent networks. In NIPS, 2016. 7 find
  33. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016. 7 find
  34. M.-T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2014. 6 find
  35. M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006. 2 find
  36. J. Mao, H. Jonathan, A. Toshev, O. Camburu, A. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 2 find our card
  37. Matterport. Press release, October 2017. 8 find
  38. H. Mei, M. Bansal, and M. R. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In AAAI, 2016. 1, 2 find
  39. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476, 2017. 4 find our card
  40. P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. In ICLR, 2017. 6 find
  41. D. K. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017. 1, 3, 6 find
  42. P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 3 find
  43. D. A. Norman. The Design of Everyday Things. Basic Books, Inc., New York, NY, USA, 2002. 5 find
  44. S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In CVPR, 2017. 7 find our card
  45. S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011. 7 find
  46. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015. 6 find our card
  47. M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun. MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931, 2017. 3, 4 — [arXiv:1712.03931](https://arxiv.org/abs/1712.03931) find
  48. S. Song, S. P. Lichtenberg, and J. Xiao. SUN RGB-D: A rgb-d scene understanding benchmark suite. In CVPR, 2015. 3 find
  49. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. 6 find
  50. L. Tai and M. Liu. Towards cognitive exploration through deep reinforcement learning for mobile robots. arXiv preprint arXiv:1610.01733, 2016. 3 — [arXiv:1610.01733](https://arxiv.org/abs/1610.01733) find
  51. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. MovieQA: Understanding stories in movies through question-answering. In CVPR, 2016. 2 find
  52. S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011. 1, 2 find
  53. C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and S. Mannor. A deep hierarchical approach to lifelong learning in minecraft. In AAAI, pages 1553–1561, 2017. 3 find
  54. A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, 2011. 4 find
  55. A. Vogel and D. Jurafsky. Learning to follow navigational directions. In ACL, 2010. 2 find
  56. D. Wierstra, A. Foerster, J. Peters, and J. Schmidhuber. Solving deep memory pomdps with recurrent policy gradients. In International Conference on Artificial Neural Networks, 2007. 6 find
  57. T. Winograd. Procedures as a representation for data in a computer program for understanding natural language. Technical report, Massachusetts Institute of Technology, 1971. 2 find
  58. Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv:1801.02209, 2018. 3 — [arXiv:1801.02209](https://arxiv.org/abs/1801.02209) find
  59. C. Yan, D. Misra, A. Bennnett, A. Walsman, Y. Bisk, and Y. Artzi. CHALET: Cornell house agent learning environment. arXiv:1801.07357, 2018. 3 — [arXiv:1801.07357](https://arxiv.org/abs/1801.07357) find
  60. Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola. Stacked attention networks for image question answering. In CVPR, 2016. 7 find
  61. A. R. Zamir, F. Xia, J. He, S. Sax, J. Malik, and S. Savarese. Gibson Env: Real-world perception for embodied agents. In CVPR, 2018. 3 find
  62. Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017. 1, 2, 3, 4, 6 Figure 9. Snapshot of the visual diversity in the Matterport3D dataset, illustrating one randomly selected… find