Elamur Research⌕
☰
home / papers / 2021 / CLIPort: What and Where Pathways for Robotic Manipulation

CLIPort: What and Where Pathways for Robotic Manipulation

Paper · 2021 · Mohit Shridhar, Lucas Manuelli and Dieter Fox

Summary

Robots that learn manipulation end-to-end can act with high spatial accuracy, but they rarely understand task semantics. Every new goal usually needs fresh demonstrations or a goal image. Large vision-language models trained on web data capture semantics well but have no sense of how to physically handle objects. CLIPort brings the two together in an imitation-learning agent guided by natural-language instructions.

The architecture has two streams:

  • Semantic ("what") stream: frozen CLIP encoders for the image and the instruction.
  • Spatial ("where") stream: a Transporter-style network trained on RGB-D input.

The streams are fused to predict pixel-level pick-and-place affordances. The approach needs no object poses, segmentation masks or symbolic state.

The authors extend the Ravens simulator with 10 language-conditioned tabletop tasks, each with thousands of variants. A single multi-task policy matches or beats per-task policies, and it can carry attributes such as new colours from one task to another. A real Franka Panda robot learned 9 tasks from very little data.

The work suggests that web-scale semantic priors can make precise robot policies more general and more data-efficient.

Key points

  • Architecture: a frozen CLIP ResNet50 encodes RGB images, and the CLIP sentence encoder conditions the decoder. A Transporter ResNet trained from scratch handles RGB-D input and is fused laterally with the semantic stream.
  • Benchmark: 10 new language-grounding tasks in Ravens with a simulated suction gripper, including packing unseen objects, block stacking and rope alignment. The benchmark, code and pre-trained models are released at cliport.github.io.
  • Cross-task transfer: training on the colour splits of other tasks lifted the multi-task model on put-blocks-in-bowls-unseen-colors (1000 demos) from 45.8 to 75.7 success.
  • Real-robot results: one multi-task model on a Franka Panda covered 9 tasks using only 179 image-action pairs. Simple block tasks succeeded about 70% of the time, and the authors estimate that 50–100 demonstrations per task are needed for robust performance.
  • Limitations: the method is restricted to a two-step pick-and-place primitive. It cannot do dexterous 6-DOF or multi-fingered control, and it cannot detect when a task is finished. On the real robot it sometimes exploited biases in the training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🏢NvidiaSanta Clara · United States37 works · 12,656 citations🏢Nvidia (United Kingdom)Reading · United Kingdom27 works · 11,141 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksControl and Systems Engineering · 62 works

Cite

Cite as Mohit Shridhar, Lucas Manuelli and Dieter Fox. 2021. CLIPort: What and Where Pathways for Robotic Manipulation. arXiv:2109.12098.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.12098 · arXiv 2109.12098

@misc{shridhar2021cliport,
  title = {{CLIPort: What and Where Pathways for Robotic Manipulation}},
  author = {Mohit Shridhar, Lucas Manuelli and Dieter Fox},
  year = {2021},
  doi = {10.48550/arxiv.2109.12098},
  eprint = {2109.12098},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.12098}
}

References

69 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs], Feb. 2021.
  2. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020. find
  3. I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019. find
  4. D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL), 2018. find
  5. D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021. — [arXiv:2104.08212](https://arxiv.org/abs/2104.08212) find
  6. D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In IEEE International Conference on Robotics and Automation (ICRA), 2021. find
  7. M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems (RSS), 2018. find
  8. C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014. find
  9. M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481–495. Springer, 2013. find
  10. D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research (IJRR), 35(1-3):281–300, 2016. find
  11. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  12. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729–9738, 2020.
  13. J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NeuRIPS), 2019.
  14. Y. C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pages 104–120. Springer, 2020.
  15. H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
  16. D. H. Hubel and T. N. Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology, 28(2):229–289, 1965. find
  17. M. Livingstone and D. Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853):740–749, 1988. find
  18. A. Derrington and P. Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):219–240, 1984. find
  19. J. J. Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014. find
  20. A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr–modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.12763, 2021. find
  21. K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. find
  22. Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Proceedings of Robotics: Science and Systems (RSS), 2018. find
  23. M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3936–3943. IEEE, 2014. find
  24. A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1386–1383. IEEE, 2017. find
  25. X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3665–3671. IEEE, 2020. find
  26. C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL), pages 1369–1378. PMLR, 2020. find
  27. P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning (CoRL), 2018. find
  28. P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492–499, 2019. find
  29. P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9411–941… find
  30. L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In International Symposium on Robotics Research (ISRR), 2019. find
  31. T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems (NeuRIPS), 32:10724–10734, 2019. find
  32. X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11602–11610, 2020. find
  33. K. Zakka, A. Zeng, J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9404–9410. IEEE, 2020. find
  34. S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978–4985, 2020. find
  35. Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020. find
  36. L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2020. find
  37. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeuRIPS), 2017.
  38. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
  39. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
  40. F. Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, and H. Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934, 2020.
  41. Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 751–761, 2016. find
  42. J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), 2015. find
  43. J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In Proceedings of International Conference on Robotics and Automation (ICRA), 2018. find
  44. Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A Joint Network for Grasp Detection Conditioned on Natural Language Commands. arXiv:2104.00492 [cs], Apr. 2021. — [arXiv:2104.00492](https://arxiv.org/abs/2104.00492) find
  45. V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL), 2020. find
  46. C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Automation (ICRA), pages 6942–6948. IEEE, 2019. find
  47. S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2011. find
  48. C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020. find
  49. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199, 2014. find
  50. C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1933–1941, 2016. find
  51. C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 6202–6211, 2019. find
  52. E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 855–859. IEEE, 2021. find
  53. F. Xiao, Y. J. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020. find
  54. A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 37… find
  55. E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine. End-to-end learning of semantic grasping. In Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research. PMLR, 2017. find
  56. R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning (ICML), 2018. find
  57. T. Cohen and M. Welling. Group equivariant convolutional networks. In International conference on machine learning (ICML), 2016. find
  58. D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. find
  59. G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi:10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. ↗
  60. E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016. find
  61. Google scanned objects dataset, 2020. URL https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects. ↗
  62. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  63. J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. find
  64. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
  65. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  66. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. find
  67. J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584, 2019.
  68. D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508, 2020. — [arXiv:2012.08508](https://arxiv.org/abs/2012.08508) find
  69. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shridhar-2021-cliport-where-pathways-robotic-manipulation/