Elamur Research⌕
☰
home / papers / 2020 / Language Conditioned Imitation Learning over Unstructured Data

Language Conditioned Imitation Learning over Unstructured Data

Paper · 2020 · Corey Lynch and Pierre Sermanet

Summary

Imitation-learned robots usually need each task identified by a task ID or a goal image, which everyday users cannot easily provide. Lynch and Sermanet let people direct a manipulation agent with free-form typed instructions instead. One end-to-end network maps raw pixels and text to 8-DOF continuous actions in a simulated 3D tabletop environment. The core technique, Multicontext Imitation Learning (MCIL), trains a single policy on a shared latent goal space. Each type of goal description has its own encoder that feeds into this space. Large amounts of unlabeled teleoperated "play" data, relabeled with hindsight goal images, can then be combined with a small set of play windows annotated with language. These annotations make up under 1% of the data. The resulting LangLfP policy carries out long sequences of text-specified tasks and outperforms imitation baselines trained on structured demonstrations. Placing the policy on top of pretrained language embeddings (TransferLangLfP) lets it handle unseen synonyms and other languages without new robot demonstrations. This makes language-driven robot instruction far cheaper to scale.

Key points

  • MCIL uses one latent-goal-conditioned policy with one encoder per dataset (goal image, task id, natural language). Most training data has no task or language labels, and language annotation is below 1% of the total.
  • The policy is built on latent motor plans (LMP). Perception, the plan posterior and prior, and the action decoder are shared between the image-goal and language-goal forward passes.
  • Evaluation spans 18 manipulation tasks (Multi-18) plus an out-of-distribution synonym set (OOD-syn). OOD-16-lang is a cross-lingual benchmark of about 240k instructions in 16 languages, produced with the Google Translate API. On it, TransferLangLfP degrades only slightly, while LangLfP drops materially.
  • With equal data (LangBC vs. Restricted LangLfP), models trained on play keep improving as model size grows. Models trained on conventional predefined demonstrations peak and then decline.
  • The authors varied the number of language pairs from 0.25k to 10k. For state-input models, going from 5k to 10k gives little gain, while pixel-input models have not yet converged. This suggests that extra language data mainly helps with perceptual grounding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 worksEngineering · 206 worksControl and Systems Engineering · 72 works

Cite

Cite as Corey Lynch and Pierre Sermanet. 2020. Language Conditioned Imitation Learning over Unstructured Data. arXiv:2005.07648.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.07648 · arXiv 2005.07648

@misc{lynch2020language,
  title = {{Language Conditioned Imitation Learning over Unstructured Data}},
  author = {Corey Lynch and Pierre Sermanet},
  year = {2020},
  doi = {10.48550/arxiv.2005.07648},
  eprint = {2005.07648},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.07648}
}

References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics, 4(4):343–355, 2012. find
  2. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pages 5048–5058, 2017.
  3. Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009. find
  4. Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Survey: Robot programming by demonstration. Technical report, Springrer, 2008. find
  5. Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997. find
  6. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2018.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  8. Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems, pages 15298–15309, 2019. find
  9. Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018. — [arXiv:1812.00568](https://arxiv.org/abs/1812.00568) find
  10. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49–58. PMLR, 2016. find
  11. Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self-supervised learning of image embedding for continuous control. arXiv preprint arXiv:1901.00943, 2019. — [arXiv:1901.00943](https://arxiv.org/abs/1901.00943) find
  12. Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. Using natural language for reward shaping in reinforcement learning, 2019. find
  13. Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE, 2017. find
  14. Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019. find
  15. Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990. find
  16. Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. arXiv preprint arXiv:1705.10479, 2017. — [arXiv:1705.10479](https://arxiv.org/abs/1705.10479) find
  17. Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017.
  18. Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro. Emergent systematic generalization in a situated agent. arXiv preprint arXiv:1910.00571, 2019. — [arXiv:1910.00571](https://arxiv.org/abs/1910.00571) find
  19. Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint arXiv:1606.03476, 2016. find
  20. Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning, 2019. find
  21. Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pages 1094–1099. Citeseer, 1993. find
  22. Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018. find
  23. Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013. find
  24. Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 259–266. IEEE, 2010. find
  25. Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926, 2019.
  26. Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1903.01973. — [arXiv:1903.01973](https://arxiv.org/abs/1903.01973) ↗
  27. Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795, 2017.
  28. Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. In Conference on Robot Learning, pages 530–539. PMLR, 2020. find
  29. Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191–9200, 2018. find
  30. Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000. find
  31. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, pages 2863–2871, 2015.
  32. Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2661–2670. JMLR. org, 2017. find
  33. Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pages 15578–15588, 2019. find
  34. Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online object representations with contrastive learning. arXiv preprint arXiv:1906.04312, 2019. — [arXiv:1906.04312](https://arxiv.org/abs/1906.04312) find
  35. Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88–97, 1991. find
  36. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. find
  37. Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758–3765. IEEE, 2018. find
  38. Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017. find
  39. Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233–242, 1999. find
  40. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312–1320, 2015. find
  41. Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. Proceedings of International Conference in Robotics and Automation (ICRA), 2018. URL http://arxiv.org/abs/1704.06888. — [arXiv:1704.06888](https://arxiv.org/abs/1704.06888) ↗
  42. Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854, 2019. — [arXiv:1904.07854](https://arxiv.org/abs/1904.07854) find
  43. Avi Singh, Eric Jang, Alexander Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levine, Mohi Khansari, and Chelsea Finn. Scalable multi-task imitation learning with autonomous improvement. arXiv preprint arXiv:2003.02636, 2020. — [arXiv:2003.02636](https://arxiv.org/abs/2003.02636) find
  44. Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. arXiv preprint arXiv:2010.12083, 2020. — [arXiv:2010.12083](https://arxiv.org/abs/2010.12083) find
  45. Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270–279. Springer, 2018. find
  46. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pages 4496–4506, 2017. find
  47. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pages 5026–5033. IEEE, 2012. ISBN 978-1-4673-1737-5. URL http://dblp.uni-trier.de/db/conf/iros/iros2012. html\#TodorovET12. ↗
  48. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni… find
  49. Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. find
  50. Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015. — [arXiv:1507.04888](https://arxiv.org/abs/1507.04888) find
  51. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307, 2019.
  52. Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in a 2d world. arXiv preprint arXiv:1802.01433, 2018. — [arXiv:1802.01433](https://arxiv.org/abs/1802.01433) find
  53. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018. find
  54. Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint arXiv:1910.08210, 2019. A PPENDIX — [arXiv:1910.08210](https://arxiv.org/abs/1910.08210) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lynch-2020-language-conditioned-imitation-learning-over/