Elamur Research
☰
home / papers / 2019 / ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1912.01734

Summary

ALFRED addresses a gap in embodied AI. Most earlier vision-and-language benchmarks cover navigation or question answering in static settings. Real household robots must also manipulate objects, track how those objects change, and follow long multi-step directions. The authors built a benchmark in the AI2-THOR 2.0 simulator spanning 120 indoor scenes, with seven household task types such as Clean & Place and Heat & Place. A PDDL-based planner generated expert demonstrations, and crowdworkers then described each one with a high-level goal and step-by-step instructions. To interact with an object, agents must output a pixelwise mask rather than just pick an object class, which makes the setting closer to real robotic grasping. A sequence-to-sequence baseline with progress monitoring, adapted from navigation work, completes fewer than 5% of full tasks. It does much better on isolated sub-goals, and humans reach about 91% success. This wide gap shows that long-horizon, compositional grounding is still largely unsolved. ALFRED therefore offers a demanding testbed for hierarchical, modular, and planning-based models.

Key points

  • The dataset contains 25,743 English directives for 8,055 expert demonstrations, averaging about 50 steps each and totaling 428,322 image-action pairs. A held-out test fold uses unseen environments.
  • Tasks are parameterized by 58 object classes and 26 receptacle classes across 120 scenes, split evenly so that every room type (kitchen, living room, bedroom, and bathroom) has 30 scenes. Each of the 2,685 parameter combinations has three demonstrations.
  • On full tasks, the Seq2Seq baseline succeeds less than 5% of the time and satisfies only about 8% of goal conditions. Progress monitoring raises full-task success only marginally, from about 1% to about 2%.
  • In sub-goal evaluation with Seq2Seq+PM on seen scenes, path-weighted success is about 51% for Goto and 32% for Pickup. Cool and Heat, which depend little on the specific object, reach roughly 90%. Performance drops further in unseen scenes.
  • Five human participants completed 100 sampled unseen-test directives with 91% success and 86% path-length-weighted success. This indicates the language annotations align well with the demonstrations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

195 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 179 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.01734 · arXiv 1912.01734

@misc{zettlemoyer2019alfred,
  title = {{ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks}},
  author = {Zettlemoyer;, Mohit Shridhar; Jesse Thomason; Daniel Gordon; Yonatan Bisk; Winson Han; Roozbeh Mottaghi; Luke and Fox, Dieter},
  year = {2019},
  doi = {10.48550/arxiv.1912.01734},
  eprint = {1912.01734},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.01734}
}

References

59 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016. 3 find
  2. Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. 6 — [arXiv:1807.06757](https://arxiv.org/abs/1807.06757) find
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018. 1, 2, 4, 13 find our card
  4. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, June 2016. 8 find our card
  5. Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. TACL, 2013. 2 find our card
  6. Masataro Asai and Alex Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary. In AAAI, 2018. 8 find
  7. Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mösenlechner, Dejan Pangercic, Thomas Rühr, and Moritz Tenorth. Robotic roommates making pancakes. In IEEE-RAS, 2011. 3 find
  8. Yonatan Bisk, Daniel Marcu, and William Wong. Towards a dataset for human computer communication via grounded language acquisition. In AAAI Workshop on Symbiotic Cognitive Systems, 2016. 8 find
  9. Yonatan Bisk, Deniz Yuret, and Daniel Marcu. Natural language communication with robots. In NAACL, 2016. 2 find
  10. Mario Bollini, Stefanie Tellex, Tyler Thompson, Nicholas Roy, and Daniela Rus. Interpreting and executing recipes with a cooking robot. In ISER, 2012. 3 find
  11. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. 3DV, 2017. 1 find
  12. Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2017. 1, 2 find
  13. David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instructions from observations. In AAAI, 2011. 2 find
  14. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019. 1, 2, 4 find
  15. Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR, 2018. 2 find
  16. Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural Modular Control for Embodied Question Answering. In CoRL, 2018. 8 find
  17. Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation. In NeurIPS, 2018. 1, 2 find
  18. Malik Ghallab, Adele Howe, Craig Knoblock, Drew Mc-Dermott, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl the planning domain definition language. 1998. 3 find
  19. Daniel Gordon, Dieter Fox, and Ali Farhadi. What should i do now? marrying reinforcement learning and symbolic planning. arXiv preprint arXiv:1901.01492, 2018. 2 — [arXiv:1901.01492](https://arxiv.org/abs/1901.01492) find
  20. Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In CVPR, 2017. 2 find
  21. Stevan Harnad. The symbol grounding problem. Physica D, 42:335–346, 1990. 1 find
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 5, 12 find
  23. Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. JAIR, 2001. 3 find
  24. Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In CVPR, 2019. 1, 2 find
  25. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 13 find
  26. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vander-Bilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv preprint arXiv:1712.05474, 2017. 1, 18 find
  27. Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS, pages 3675–3683, 2016. 8 find
  28. Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In ICLR, 2019. 2, 6, 7, 8 find
  29. Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR, 2019. 1, 2 find
  30. James MacGlashan, Monica Babes-Vroman, Marie des-Jardins, Michael L. Littman, Smaranda Muresan, Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. Grounding english commands to reward functions. In RSS, 2015. 3 find
  31. Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006. 2 find
  32. Jonathan Malmaud, Earl Wagner, Nancy Chang, and Kevin Murphy. Cooking with semantics. In ACL Workshop on Semantic Parsing, 2014. 2 find
  33. Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3D environments with visual goal prediction. In EMNLP, 2018. 2 find
  34. Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017. 2 find
  35. Dipendra Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me daved: Context-sensitive grounding of natural language to manipulation instructions. In RSS, 2014. 3 find
  36. Dipendra Misra, Kejia Tao, Percy Liang, and Ashutosh Saxena. Environment-driven lexicon induction for high-level instructions. In ACL, 2015. 2 find
  37. Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019. 1, 8 find
  38. Khanh Nguyen and Hal Daumé III. Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning. In EMNLP, 2019. 2 find
  39. Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In CVPR, 2019. 2 find
  40. Daniel Nyga, Subhro Roy, Rohan Paul, Daehyung Park, Mihai Pomarlan, Michael Beetz, and Nicholas Roy. Grounding robot plans from natural language instructions with incomplete world knowledge. In CoRL, 2018. 3 find
  41. Rohan Paul, Jacob Arkin, Derya Aksaray, Nicholas Roy, and Thomas M. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. IJRR, 2018. 3 find
  42. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018. 1, 2, 4 find
  43. Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. TACL, 2013. 2, 3 find
  44. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011. 4 find
  45. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In ICCV, 2019. 1 find
  46. Ozan Sener, Amir R. Zamir, Silvio Savarese, and Ashutosh Saxena. Unsupervised semantic parsing of video collections. In ICCV, 2015. 3 find
  47. Mohit Shridhar and David Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In RSS, 2018. 3 find
  48. Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACL, 2019. 1 find
  49. Moritz Tenorth, Daniel Nyga, and Michael Beetz. Understanding and executing instructions for everyday manipulation tasks from the world wide web. In ICRA, 2010. 2 find
  50. Jesse Thomason, Daniel Gordon, and Yonatan Bisk. Shifting the baseline: Single modality performance on visual navigation & qa. In NAACL, 2019. 7 find
  51. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL, 2019. 2 find
  52. Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In IJCAI, 2015. 3 find
  53. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR, 2019. 1, 2 find
  54. Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In ECCV, 2018. 2 find
  55. Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic environments with point cloud perception. In CVPR, 2019. 2 find
  56. Haonan Yu and Jeffrey Mark Siskind. Grounded language learning from video described with sentences. In ACL, 2013. 3 find
  57. Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, and Dhruv Batra. Multi-target embodied question answering. In CVPR, 2019. 2 find
  58. Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In ICCV, 2017. 2 find
  59. Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In CVPR, 2019. 3 find