Elamur Research⌕
☰
home / papers / 2019 / Habitat: A Platform for Embodied AI Research

Habitat: A Platform for Embodied AI Research

Paper · 2019 · Manolis Savva, Abhishek Kadian, Oleksandr Maksymets et al.

Summary

Embodied AI studies agents, such as virtual robots, that must perceive, move and act. Training them in the physical world is slow, risky, costly and hard to reproduce. Existing simulators were fragmented: each tied one task, one 3D dataset and a fixed agent design together, and most ran at only 10–100 fps.

The authors present Habitat, an open-source stack with two layers:

  • Habitat-Sim is a C++ simulator with a Python interface. It loads generic 3D scene datasets (Matterport3D, Gibson, Replica) and lets users configure agents and sensors.
  • Habitat-API is a library for defining tasks, training agents (reinforcement or imitation learning, or classical SLAM) and scoring them with standard metrics.

Rendering reaches thousands of frames per second on one thread and over 10,000 fps across processes on a single GPU. This speed made it possible to train point-goal navigation agents on 75 million steps. At that scale, learned agents overtook SLAM baselines, which reverses earlier conclusions. Cross-dataset tests showed that only agents with depth input transfer well between Matterport3D and Gibson. The platform aims to serve as a shared benchmarking base for the field.

Key points

  • Habitat-Sim renders Matterport3D scenes at several thousand fps single-threaded and above 10,000 fps multi-process on one GPU. MINOS, used in prior work, runs at roughly 100 fps.
  • Learned point-goal navigation agents trained on 75M steps, versus 5M in Mishkin et al. [20], match and exceed classical SLAM. This contradicts the conclusion of [20, 16].
  • The first cross-dataset experiments used {train, test} × {Matterport3D, Gibson} with Blind, RGB, RGBD and D sensors plus GPS+Compass. Only depth-equipped agents generalized well.
  • Habitat Challenge (CVPR 2019) asked participants to submit agents as Docker containers through EvalAI. The agents were then run on unseen environments on AWS GPU instances.
  • The planned challenge changes are:
    • giving only a fixed relative goal coordinate instead of continuously updated GPS+Compass;
    • adding an actuation noise model measured on a real robot;
    • adding realistic RGB and depth sensor noise.

Interesting ideas

  • I1. Decouple task, simulator and dataset. A dataset-agnostic ingestion API and a separate task layer let one experiment run on many corpora. The same separation would help NLP pipelines evaluate a model across several treebanks or corpora without rewriting code.
  • I2. Treat throughput as a scientific variable. The reversed learning-versus-SLAM result appeared only once training experience grew by an order of magnitude. Before declaring one method weaker, check whether it was trained to convergence.
  • I3. Collect code, not predictions. When evaluation is interactive or must stay hidden, participants submit a containerized agent behind a fixed interface. This prevents tuning on test data and suits any shared task with a private test set.
  • I4. Run cross-dataset generalization grids. Crossing train/test datasets with input modalities shows which signals carry over between domains. This design transfers directly to cross-corpus or cross-genre evaluation.
  • I5. Remove idealized assumptions step by step. Each benchmark iteration adds noise and drops perfect localization to make the task more realistic. This keeps benchmarks challenging while tracking progress across versions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations🎓Simon Fraser UniversityBurnaby · Canada19 works · 6,205 citations🏢IntelSanta Clara · United States18 works · 43,058 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksAerospace Engineering · 4 worksAutomotive Engineering · 9 works

Cite

Cite as Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh and Dhruv Batra. 2019. Habitat: A Platform for Embodied AI Research. arXiv:1904.01201.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.01201 · arXiv 1904.01201

@misc{savva2019habitat,
  title = {{Habitat: A Platform for Embodied AI Research}},
  author = {Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh and Dhruv Batra},
  year = {2019},
  doi = {10.48550/arxiv.1904.01201},
  eprint = {1904.01201},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.01201}
}

References

31 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Košecká, and Alexander C Berg. A dataset for developing and benchmarking active vision. In ICRA, 2017. find
  2. Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents. arXiv:1807.06757, 2018. — [arXiv:1807.06757](https://arxiv.org/abs/1807.06757) find
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
  4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. find
  5. Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3D semantic parsing of large-scale indoor spaces. In CVPR, 2016. find
  6. Alex Bewley, Jessica Rigley, Yuxuan Liu, Jeffrey Hawke, Richard Shen, Vinh-Dieu Lam, and Alex Kendall. Learning to drive from simulation without real world labels. In ICRA, 2019. find
  7. Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron C. Courville. HoME: A household multimodal environment. arXiv:1711.11017, 2017. — [arXiv:1711.11017](https://arxiv.org/abs/1711.11017) find
  8. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. In International Conference on 3D Vision (3DV), 2017. find
  9. Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR, 2018. find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018.
  12. David Donoho. 50 years of data science. In Tukey Centennial Workshop, 2015. find
  13. Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In CVPR, 2017. find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  15. Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019. find
  16. Noriyuki Kojima and Jia Deng. To learn or not to learn: Analyzing the role of learning for navigation in virtual environments. arXiv:1907.11770, 2019. — [arXiv:1907.11770](https://arxiv.org/abs/1907.11770) find
  17. Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An interactive 3D environment for visual AI. arXiv:1712.05474, 2017.
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
  19. Dirk Merkel. Docker: Lightweight Linux containers for consistent development and deployment. Linux Journal, 2014. find
  20. Dmytro Mishkin, Alexey Dosovitskiy, and Vladlen Koltun. Benchmarking classic and learned navigation in complex 3D environments. arXiv:1901.10915, 2019. — [arXiv:1901.10915](https://arxiv.org/abs/1901.10915) find
  21. Raúl Mur-Artal and Juan D. Tardós. ORB-SLAM2: An open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE Transactions on Robotics, 33(5), 2017. find
  22. Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta. Pyrobot: An open-source robotics framework for research and benchmarking. arXiv preprint arXiv:1906.08236, 2019. — [arXiv:1906.08236](https://arxiv.org/abs/1906.08236) find
  23. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating household activities via programs. In CVPR, 2018. find
  24. Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931, 2017. — [arXiv:1712.03931](https://arxiv.org/abs/1712.03931) find
  25. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017. find
  26. Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial Life, 11(1-2), 2005. find
  27. Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In CVPR, 2017. find
  28. Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggle… — [arXiv:1906.05797](https://arxiv.org/abs/1906.05797) find
  29. Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3D environment. arXiv:1801.02209, 2018. — [arXiv:1801.02209](https://arxiv.org/abs/1801.02209) find
  30. Fei Xia, Amir R. Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018. find
  31. Claudia Yan, Dipendra Misra, Andrew Bennnett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. CHALET: Cornell house agent learning environment. arXiv:1801.07357, 2018. — [arXiv:1801.07357](https://arxiv.org/abs/1801.07357) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/savva-2019-habitat-platform-embodied-ai-research/