Elamur Research⌕
☰
home / papers / 2023 / Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Paper · 2023 · Cheng Chi, Zhenjia Xu, Siyuan Feng et al.

Summary

Learning robot control from human demonstrations can be framed as supervised regression, but robot actions are hard targets. Several different motions may be equally valid, consecutive actions are strongly correlated, and execution must be precise. Earlier work tried Gaussian mixtures, discretized actions, or energy-based implicit policies, and the implicit policies tend to train unstably.

This paper proposes Diffusion Policy, which treats a visuomotor policy as a conditional denoising diffusion model (DDPM). Starting from Gaussian noise, a network repeatedly estimates and removes noise to produce a whole sequence of future actions, conditioned on recent camera images and robot state. To make this practical on physical robots, the authors add:

  • receding-horizon execution,
  • conditioning on vision instead of modelling observations jointly,
  • a transformer-based denoiser alongside a CNN-based one.

Across 15 simulated and real tasks drawn from 4 manipulation benchmarks, the method beats prior state-of-the-art approaches by 46.9% on average. The authors argue this shows that the policy's form, not only the demonstration data, is a major bottleneck in behavior cloning.

Key points

  • Evaluation spans 15 tasks taken from 4 benchmarks, introduced respectively by Gupta et al. 2019, Shafiullah et al. 2022, Florence et al. 2021, and Mandlekar et al. 2021. The tasks cover 2DoF–6DoF actions, rigid and fluid objects, and single- and multi-user demonstrations. The mean gain over prior SOTA is 46.9%.
  • Training uses the standard DDPM noise-prediction MSE loss. Inference runs K denoising iterations, which can be read as noisy gradient descent on a learned energy landscape. This avoids the negative sampling that destabilizes implicit (energy-based) policies.
  • Three horizons structure each step: T_o is how many past observations are used, T_p is how many actions are predicted, and T_a is how many actions are executed before replanning.
  • Two denoiser backbones are offered:
    • a 1D-CNN with FiLM conditioning on observation features;
    • a time-series diffusion transformer with cross-attention to observations and causal attention over actions, which handles fast action changes and velocity control better.
  • The extended version adds bimanual real-robot tasks (Egg Beater, Mat Unrolling, Shirt Folding), a section linking the method to control theory, and more ablations. Code and data are at diffusion-policy.cs.columbia.edu. Known drawbacks are higher latency and compute than simpler baselines such as LSTM-GMM.

Interesting ideas

  • I1. Diffusion for ambiguous targets: Generate the output with a diffusion model instead of regressing it when several distinct answers are valid. Plain regression averages these answers into something wrong. This applies to any structured prediction task with ambiguous targets.
  • I2. Predict a chunk, execute only part of it: Predict a long sequence but commit to only its first part, then replan with fresh input. This balances coherence against reactivity and suits any sequential decision or streaming generation setup.
  • I3. Encode the input once: Treat the expensive input (here, images) purely as conditioning, so it is encoded once rather than at every denoising step. This simple design choice makes iterative generation fast enough for real-time use.
  • I4. Gradient view of iterative refinement: Reading denoising as gradient steps on a learned energy explains why the method stays expressive without the normalization problems of energy-based training. This framing is useful when choosing between generative formulations.
  • I5. Causal attention over the output sequence: In the transformer denoiser, output tokens attend only to earlier tokens and get cross-attention to the conditioning signal. This reusable pattern reduces the over-smoothing seen with convolutional sequence models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputational Theory and Mathematics · 321 worksEngineering · 188 worksControl and Systems Engineering · 62 works

Cite

Cite as Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake and Shuran Song. 2023. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv:2303.04137.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.04137 · arXiv 2303.04137

@misc{chi2023diffusion,
  title = {{Diffusion Policy: Visuomotor Policy Learning via Action Diffusion}},
  author = {Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake and Shuran Song},
  year = {2023},
  doi = {10.48550/arxiv.2303.04137},
  eprint = {2303.04137},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.04137}
}

References

70 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ajay A, Du Y, Gupta A, Tenenbaum J, Jaakkola T and Agrawal P (2022) Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 . — [arXiv:2211.15657](https://arxiv.org/abs/2211.15657) find
  2. Argall BD, Chernova S, Veloso M and Browning B (2009) A survey of robot learning from demonstration. Robotics and autonomous systems 57(5): 469–483. find
  3. Atkeson CG and Schaal S (1997) Robot learning from demonstration. In: ICML, volume 97. pp. 12–20. find
  4. Avigal Y, Berscheid L, Asfour T, Kröger T and Goldberg K (2022) Speedfolding: Learning efficient bimanual folding of garments. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 1–8. find
  5. Bishop CM (1994) Mixture density networks. Aston University. find
  6. Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J et al. (2016) End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 . — [arXiv:1604.07316](https://arxiv.org/abs/1604.07316) find
  7. Chen T (2023) On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972 . — [arXiv:2301.10972](https://arxiv.org/abs/2301.10972) find
  8. Chi C, Feng S, Du Y, Xu Z, Cousineau E, Burchfiel B and Song S (2023) Diffusion policy: Visuomotor policy learning via action diffusion. In: Proceedings of Robotics: Science and Systems (RSS).
  9. Dai B, Liu Z, Dai H, He N, Gretton A, Song L and Schuurmans D (2019) Exponential family estimation via adversarial dynamics embedding. Advances in Neural Information Processing Systems 32. find
  10. Deng J, Dong W, Socher R, Li LJ, Li K and Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. Ieee, pp. 248–255. find
  11. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .
  12. Du Y, Li S, Tenenbaum J and Mordatch I (2020) Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316 . — [arXiv:2012.01316](https://arxiv.org/abs/2012.01316) find
  13. Du Y and Mordatch I (2019) Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689 . — [arXiv:1903.08689](https://arxiv.org/abs/1903.08689) find
  14. Florence P, Lynch C, Zeng A, Ramirez OA, Wahid A, Downs L, Wong A, Lee J, Mordatch I and Tompson J (2021) Implicit behavioral cloning. In: 5th Annual Conference on Robot Learning. find
  15. Florence P, Manuelli L and Tedrake R (2019) Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters 5(2): 492–499. find
  16. Grathwohl W, Wang KC, Jacobsen JH, Duvenaud D and Zemel R (2020) Learning the stein discrepancy for training and evaluating energy-based models without sampling. In: International Conference on Machine Learning. find
  17. Gupta A, Kumar V, Lynch C, Levine S and Hausman K (2019) Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 . — [arXiv:1910.11956](https://arxiv.org/abs/1910.11956) find
  18. Hansen-Estruch P, Kostrikov I, Janner M, Kuba JG and Levine S (2023) Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573 . — [arXiv:2304.10573](https://arxiv.org/abs/2304.10573) find
  19. He K, Fan H, Wu Y, Xie S and Girshick R (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738.
  20. He K, Zhang X, Ren S and Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778.
  21. Ho J, Jain A and Abbeel P (2020) Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239 . find
  22. Huang S, Wang Z, Li P, Jia B, Liu T, Zhu Y, Liang W and Zhu SC (2023) Diffusion-based generation, optimization, and planning in 3d scenes. arXiv preprint arXiv:2301.06015 . find
  23. Janner M, Du Y, Tenenbaum J and Levine S (2022a) Planning with diffusion for flexible behavior synthesis. In: International Conference on Machine Learning. find
  24. Janner M, Du Y, Tenenbaum J and Levine S (2022b) Planning with diffusion for flexible behavior synthesis. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G and Sabato S (eds.) Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR. find
  25. Jarrett D, Bica I and van der Schaar M (2020) Strictly batch imitation learning by energy-based distribution matching. Advances in Neural Information Processing Systems 33: 7354– 7365. find
  26. Karras T, Aittala M, Aila T and Laine S (2022) Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 . find
  27. Khatib O (1987) A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation 3(1): 43–53. DOI:10. 1109/JRA.1987.1087068. Conference Name: IEEE Journal on Robotics and Automation. find
  28. LeCun Y, Chopra S, Hadsell R, Huang FJ and et al (2006) A tutorial on energy-based learning. In: Predicting Structured Data. MIT Press. find
  29. Liu L, Liu X, Gao J, Chen W and Han J (2020) Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249 .
  30. Liu N, Li S, Du Y, Torralba A and Tenenbaum JB (2022) Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.01714 . — [arXiv:2206.01714](https://arxiv.org/abs/2206.01714) find
  31. Mandlekar A, Ramos F, Boots B, Savarese S, Fei-Fei L, Garg A and Fox D (2020a) Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE. find
  32. Mandlekar A, Xu D, Martı́n-Martı́n R, Savarese S and Fei-Fei L (2020b) Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085 . — [arXiv:2003.06085](https://arxiv.org/abs/2003.06085) find
  33. Mandlekar A, Xu D, Wong J, Nasiriany S, Wang C, Kulkarni R, Fei-Fei L, Savarese S, Zhu Y and Martı́n-Martı́n R (2021) What matters in learning from offline human demonstrations for robot manipulation. In: 5th Annual Conference on Robot Learning. find
  34. Mayne DQ and Michalska H (1988) Receding horizon control of nonlinear systems. In: Proceedings of the 27th IEEE Conference on Decision and Control. IEEE, pp. 464–465. find
  35. Nair S, Rajeswaran A, Kumar V, Finn C and Gupta A (2022) R3m: A universal visual representation for robot manipulation. In: 6th Annual Conference on Robot Learning. find
  36. Neal RM et al. (2011) Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo . find
  37. Nichol AQ and Dhariwal P (2021) Improved denoising diffusion probabilistic models. In: International Conference on Machine Learning. PMLR, pp. 8162–8171. find
  38. Pearce T, Rashid T, Kanervisto A, Bignell D, Sun M, Georgescu R, Macua SV, Tan SZ, Momennejad I, Hofmann K et al. (2023) Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 . — [arXiv:2301.10677](https://arxiv.org/abs/2301.10677) find
  39. Perez E, Strub F, De Vries H, Dumoulin V and Courville A (2018) Film: Visual reasoning with a general conditioning layer. In: Proceedings of the AAAI Conference on Artificial Intelligence.
  40. Pomerleau DA (1988) Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems 1. find
  41. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et al. (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp. 8748–8763.
  42. Rahmatizadeh R, Abolghasemi P, Bölöni L and Levine S (2018) Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In: 2018 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 3758–3765. find
  43. Ravichandar H, Polydoros AS, Chernova S and Billard A (2020) Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems 3: 297– 330. find
  44. Reuss M, Li M, Jia X and Lioutikov R (2023) Goal-conditioned imitation learning using score-based diffusion policies. In: Proceedings of Robotics: Science and Systems (RSS). find
  45. Ridnik T, Ben-Baruch E, Noy A and Zelnik-Manor L (2021) Imagenet-21k pretraining for the masses. find
  46. Ronneberger O, Fischer P and Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention– MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, pp. 234–241.
  47. Ross S, Gordon G and Bagnell D (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, pp. 627–635. find
  48. Shafiullah NMM, Cui ZJ, Altanzaya A and Pinto L (2022) Behavior transformers: Cloning $k$ modes with one stone. In: Oh AH, Agarwal A, Belgrave D and Cho K (eds.) Advances in Neural Information Processing Systems. find
  49. Sharma P, Mohan L, Pinto L and Gupta A (2018) Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In: Conference on robot learning. PMLR. find
  50. Siciliano B, Khatib O and Kröger T (2008) Springer handbook of robotics, volume 200. Springer. find
  51. Sohl-Dickstein J, Weiss E, Maheswaranathan N and Ganguli S (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning.
  52. Song J, Meng C and Ermon S (2021) Denoising diffusion implicit models. In: International Conference on Learning Representations.
  53. Song Y, Dhariwal P, Chen M and Sutskever I (2023) Consistency models. arXiv preprint arXiv:2303.01469 . find
  54. Song Y and Ermon S (2019) Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems 32.
  55. Subramanian J and Mahajan A (2019) Approximate information state for partially observed systems. In: 2019 IEEE 58th Conference on Decision and Control (CDC). IEEE, pp. 1629– 1636. find
  56. Ta DN, Cousineau E, Zhao H and Feng S (2022) Conditional energy-based models for implicit policies: The gap between theory and practice. arXiv preprint arXiv:2207.05824 . — [arXiv:2207.05824](https://arxiv.org/abs/2207.05824) find
  57. Tancik M, Srinivasan P, Mildenhall B, Fridovich-Keil S, Raghavan N, Singhal U, Ramamoorthi R, Barron J and Ng R (2020) Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33: 7537–7547. find
  58. Toyer S, Shah R, Critch A and Russell S (2020) The magical benchmark for robust imitation. Advances in Neural Information Processing Systems 33: 18284–18295. find
  59. Urain J, Funk N, Chalvatzaki G and Peters J (2022) Se (3)-diffusionfields: Learning cost functions for joint grasp and motion optimization through diffusion. arXiv preprint arXiv:2209.03855 . — [arXiv:2209.03855](https://arxiv.org/abs/2209.03855) find
  60. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł and Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30.
  61. Wang Z, Hunt JJ and Zhou M (2022) Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 . — [arXiv:2208.06193](https://arxiv.org/abs/2208.06193) find
  62. Wang Z, Hunt JJ and Zhou M (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. In: The Eleventh International Conference on Learning Representations. URL https://openreview.net/forum?id=AHvFDPi-FA. ↗
  63. Welling M and Teh YW (2011) Bayesian learning via stochastic gradient langevin dynamics. In: Proceedings of the 28th international conference on machine learning (ICML-11). pp. 681–688. find
  64. Wu J, Sun X, Zeng A, Song S, Lee J, Rusinkiewicz S and Funkhouser T (2020) Spatial action maps for mobile manipulation. In: Proceedings of Robotics: Science and Systems (RSS). find
  65. Wu Y and He K (2018) Group normalization. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19. find
  66. Yang J, Zhang J, Settle C, Rai A, Antonova R and Bohg J (2022) Learning periodic tasks from human demonstrations. In: 2022 International Conference on Robotics and Automation (ICRA). IEEE, pp. 8658–8665. find
  67. Zeng A, Florence P, Tompson J, Welker S, Chien J, Attarian M, Armstrong T, Krasin I, Duong D, Sindhwani V et al. (2021) Transporter networks: Rearranging the visual world for robotic manipulation. In: Conference on Robot Learning. PMLR, pp. 726–747. find
  68. Zhang A, McAllister RT, Calandra R, Gal Y and Levine S (2020) Learning invariant representations for reinforcement learning without reconstruction. In: International Conference on Learning Representations. find
  69. Zhang T, McCarthy Z, Jow O, Lee D, Chen X, Goldberg K and Abbeel P (2018) Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 5628–5635. find
  70. Zhou Y, Barnes C, Lu J, Yang J and Li H (2019) On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5745–5753. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chi-2023-diffusion-policy-visuomotor-policy-learning/