Elamur Research⌕
☰
home / papers / 2023 / Distilling Internet-Scale Vision-Language Models into Embodied Agents

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Paper · 2023 · Theodore Sumers, Kenneth Marino, Arun Ahuja et al.

Summary

Language-conditioned embodied agents must link words to what they perceive and do. The paired trajectory–language data needed for this is costly to collect, often ambiguous, and tied to one agent's design. The authors propose using a pretrained generative vision-language model as an automatic annotator.

Building on hindsight experience replay (HER), the Flamingo VLM inspects the final observation of an agent's episode and writes an instruction that the episode actually satisfied. The relabeled trajectories then train the agent through imitation learning. Because the method works purely as data augmentation, it does not depend on the agent architecture, and its labels are human-readable.

Varying the prompt, or adding few-shot examples, controls which aspect of the scene gets described. In a 3D-rendered environment this taught object names, colours, categories such as food versus toys, and arbitrary user preferences. An analysis of noisy labels found that precision matters far more than accuracy. The work offers an inexpensive route for transferring internet-scale grounding into agents without heavy human annotation.

Key points

  • The relabeling function is Flamingo, used zero-shot or few-shot. Swapping in the OWL-ViT detector as relabeler performed substantially worse.
  • Few-shot prompting used 32 in-context examples (3–4 for each of 10 objects). It raised the share of labels containing the correct object name from 54.7% to 77.2%. However, agent task success fell from 64.4% (Zeroshot) to 52.9% (Fewshot).
  • A mixed-effects regression over 40 label sets (10 tasks × 4 labeling variants) gave precision an effect about ten times that of accuracy (β = 1.44 vs β = 0.16).
  • Discarding the least confident 50% of labels gave Fewshot agents a large gain but barely changed Zeroshot agents.
  • Limitations:
    • The work covers English only and relabels only the final observation.
    • A low-level motor policy was first learned from the Interactive Agents Team (2021) dataset.
    • Flamingo is not public; GPT-4 or Prismer are suggested as substitutes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 worksEngineering · 206 worksControl and Systems Engineering · 72 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 827 works

Cite

Cite as Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus and Ishita Dasgupta. 2023. Distilling Internet-Scale Vision-Language Models into Embodied Agents. arXiv:2301.12507.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.12507 · arXiv 2301.12507

@misc{sumers2023distilling,
  title = {{Distilling Internet-Scale Vision-Language Models into Embodied Agents}},
  author = {Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus and Ishita Dasgupta},
  year = {2023},
  doi = {10.48550/arxiv.2301.12507},
  eprint = {2301.12507},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12507}
}

References

81 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F., Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik, A., et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020. — [arXiv:2012.05672](https://arxiv.org/abs/2012.05672) find
  2. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  3. Future work may experiment with other ways to leverage pretrained generative VLMs for embodied agents. Following our offline supervision approach, VLMs could provide a reward signal during training on the basis of vision and text alignment. Alternatively, VLMs could be used directly 4 Our VLM (Alayr… find
  4. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… — Flamingo: a Visual Language Model for Few-Shot Learning
  5. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.,… — Language Models are Few-Shot Learners
  6. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and van den Hengel, A. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pp. 3674–3683, 2018.
  7. Bucker, A. F. C., Figueredo, L. F. C., Haddadin, S., Kapoor, A., Ma, S., Vemprala, S., and Bonatti, R. LaTTe: Language trajectory transformEr. ArXiv, abs/2208.02918, 2022. — [arXiv:2208.02918](https://arxiv.org/abs/2208.02918) find
  8. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
  9. Chai, J. Y., Gao, Q., She, L., Yang, S., Saba-Sadiya, S., and Xu, G. Language to action: Towards interactive task learning with physical agents. In IJCAI, pp. 2–9, 2018. find
  10. Chan, H., Wu, Y., Kiros, J., Fidler, S., and Ba, J. Actrce: Augmenting experience via teacher’s advice for multi-goal reinforcement learning. arXiv preprint arXiv:1902.04546, 2019. find
  11. Artzi, Y. and Zettlemoyer, L. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49–62, 2013.
  12. Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., and Salakhutdinov, R. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018. find
  13. Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, A., Kohli, P., and Grefenstette, E. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018. find
  14. Chen, B., Xia, F., Ichter, B., Rao, K., Gopalakrishnan, K., Ryoo, M. S., Stone, A., and Kappler, D. Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874, 2022a. — [arXiv:2209.09874](https://arxiv.org/abs/2209.09874) find
  15. Barsalou, L. W. Ad hoc categories. Memory & cognition, 11:211–227, 1983. find
  16. Bigazzi, R., Landi, F., Cornia, M., Cascianelli, S., Baraldi, L., and Cucchiara, R. Explore and explain: Self-supervised navigation and recounting. In 25th International Conference on Pattern Recognition (ICPR), pp. 1152–1159, 2021. find
  17. Chen, D. and Mooney, R. Learning to interpret natural language navigation instructions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 859–865, 2011. find
  18. Blukis, V., Terme, Y., Niklasson, E., Knepper, R. A., and Artzi, Y. Learning to map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2019. find
  19. Chen, S., Guhur, P.-L., Tapaswi, M., Schmid, C., and Laptev, I. Learning from unlabeled 3d environments for vision-and-language navigation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX, pp. 638–655. find
  20. Blukis, V., Knepper, R. A., and Artzi, Y. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:2011.07384, abs/2011.07384, 2020. — [arXiv:2011.07384](https://arxiv.org/abs/2011.07384) find
  21. Chen, V., Gupta, A. K., and Marino, K. Ask your humans: Using human instructions to improve generalization in reinforcement learning. ICLR, 2020. find
  22. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  23. Cideron, G., Seurin, M., Strub, F., and Pietquin, O. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 225– 232. find
  24. Branavan, S., Silver, D., and Barzilay, R. Learning to win by reading manuals in a Monte-Carlo framework. Journal of Artificial Intelligence Research, 43:661–704, 2012. find
  25. Co-Reyes, J. D., Gupta, A., Sanjeev, S., Altieri, N., Andreas, J., DeNero, J., Abbeel, P., and Levine, S. Guiding policies with language via meta-learning. In International Conference on Learning Representations, 2018. find
  26. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L., Levine, S., Hausman, K., and Ichter, B. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conferen… find
  27. Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R. Collaborating with language models for embodied reasoning. In Second Workshop on Language and Reinforcement Learning, 2022.
  28. Interactive Agents Team, D. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint arXiv:2112.03763, 2021. — [arXiv:2112.03763](https://arxiv.org/abs/2112.03763) find
  29. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. N. BERT: Pre-training of deep bidirectional transformers for language understanding. 2018. URL https://arxiv.org/abs/1810.04805. ↗
  30. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  31. Fan, L. J., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. MineDojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 2022.
  32. Kollar, T., Tellex, S., Roy, D., and Roy, N. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 259–266. find
  33. Kulick, J., Toussaint, M., Lang, T., and Lopes, M. Active learning for teaching a robot grounded relational symbols. In IJCAI, pp. 1451–1457. Citeseer, 2013. find
  34. Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., and Darrell, T. Speaker-follower models for vision-and-language navigation. Advances in Neural Information Processing Systems, 31, 2018. find
  35. Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., and Angelova, A. F-vlm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15639, 2022. — [arXiv:2209.15639](https://arxiv.org/abs/2209.15639) find
  36. Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., and Schmid, C. Airbert: In-domain pretraining for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1634– 1643, 2021. find
  37. Li, C., Xia, F., Martı́n-Martı́n, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272, 2021. — [arXiv:2108.03272](https://arxiv.org/abs/2108.03272) find
  38. Hao, W., Li, C., Li, X., Carin, L., and Gao, J. Towards learning a generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13137–13146, 2020. find
  39. Li, X., Li, C., Xia, Q., Bisk, Y., Celikyilmaz, A., Gao, J., Smith, N. A., and Choi, Y. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  40. Harnad, S. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990. find
  41. Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., and Clark, S. Grounded language learning fast and slow. In International Conference on Learning Representations, 2021. find
  42. Lin, J., Fried, D., Klein, D., and Dragan, A. Inferring rewards from language in context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8546– 8560, Dublin, Ireland, May 2022. Association for Computational Linguistics. find
  43. Hsu, K., Lum, T. G. W., Gao, R., Gu, S. S., Wu, J., and Finn, C. What makes certain pre-trained visual representations better for robotic learning? In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. find
  44. Liu, S., Fan, L., Johns, E., Yu, Z., Xiao, C., and Anandkumar, A. Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02506, 2023. — [arXiv:2303.02506](https://arxiv.org/abs/2303.02506) find
  45. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ArXiv, abs/2201.07207, 2022a.
  46. Lynch, C. and Sermanet, P. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
  47. Majumdar, A., Aggarwal, G., Devnani, B. S., Hoffman, J., and Batra, D. ZSON: Zero-shot object-goal navigation using multimodal goal embeddings. In Advances in Neural Information Processing Systems, 2022. find
  48. Quine, W. V. O. Word & Object. MIT Press, 1960. find
  49. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
  50. Mei, H., Bansal, M., and Walter, M. R. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In AAAI, 2016. find
  51. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
  52. Mikolov, T., Yih, W.-t., and Zweig, G. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746–751, 2013.
  53. Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339–9347, 2019.
  54. Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., and Houlsby, N. Simple open-vocabulary object detection. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel,… find
  55. Shah, D., Osiński, B., Levine, S., et al. LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action. In 6th Annual Conference on Robot Learning, 2022. find
  56. Misra, D., Langford, J., and Artzi, Y. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1004–1015, 2017.
  57. Sharma, P., Torralba, A., and Andreas, J. Skill induction and planning with latent language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1713–1726, 2022.
  58. Mohan, S. and Laird, J. Learning goal-oriented hierarchical tasks from situated interactive instruction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014. find
  59. She, L., Cheng, Y., Chai, J. Y., Jia, Y., Yang, S., and Xi, N. Teaching robots new actions through natural language instructions. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 868–873. IEEE, 2014. find
  60. Mooney, R. J. Learning to connect language and perception. In AAAI, pp. 1598–1601. Chicago, 2008. find
  61. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740–10749, 2020.
  62. Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3M: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022. find
  63. Narasimhan, K., Barzilay, R., and Jaakkola, T. Grounding language for transfer in deep reinforcement learning. J. Artif. Int. Res., 63(1):849–874, sep 2018. ISSN 1076-9757. find
  64. Shridhar, M., Manuelli, L., and Fox, D. Cliport: What and where pathways for robotic manipulation. CoRL, 2021.
  65. Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022.
  66. Nguyen, K. X., Misra, D., Schapire, R., Dudı́k, M., and Shafto, P. Interactive learning from activity description. In International Conference on Machine Learning, pp. 8096–8108. PMLR, 2021. find
  67. OpenAI. Gpt-4 technical report, 2023.
  68. Sumers, T. R., Ho, M. K., Hawkins, R. D., Narasimhan, K., and Griffiths, T. L. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6002–6010, 2021. find
  69. Pinto, L. and Gupta, A. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3406–3413, 2016. doi: 10.1109/ICRA.2016.7487517. find
  70. Tam, A., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C., Strouse, D., Wang, J. X., Banino, A., and Hill, F. Semantic exploration from language abstractions and pretrained representations. In Advances in Neural Information Processing Systems, 2022. find
  71. Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., and Roy, N. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507–1514, 2011. find
  72. Tellex, S., Gopalan, N., Kress-Gazit, H., and Matuszek, C. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1), 2020. find
  73. Thomason, J., Padmakumar, A., Sinapov, J., Hart, J., Stone, P., and Mooney, R. Opportunistic active learning for grounding natural language descriptions. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), 2017. find
  74. Winograd, T. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. find
  75. Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., and Tompson, J. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736, 2022. — [arXiv:2211.11736](https://arxiv.org/abs/2211.11736) find
  76. Yan, C., Carnevale, F., Georgiev, P., Santoro, A., Guy, A., Muldal, A., Hung, C.-C., Abramson, J. S., Lillicrap, T. P., and Wayne, G. Intra-agent speech permits zero-shot task acquisition. In Advances in Neural Information Processing Systems, 2022. find
  77. Yu, H., Zhang, H., and Xu, W. Interactive grounded language acquisition and generalization in a 2D world. In ICLR, 2018. find
  78. Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
  79. Zhong, V., Rocktäschel, T., and Grefenstette, E. RTFM: Generalising to new environment dynamics via reading. In ICLR, 2020. find
  80. Zhong, V., Hanjie, A. W., Wang, S., Narasimhan, K., and Zettlemoyer, L. SILG: The multi-domain symbolic interactive language grounding benchmark. Advances in Neural Information Processing Systems, 34:21505–21519, 2021. find
  81. Zhong, V., Mu, J., Zettlemoyer, L., Grefenstette, E., and Rocktäschel, T. Improving policy learning via language dynamics distillation. In Advances in Neural Information Processing Systems, 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sumers-2023-distilling-internet-scale-vision-language/