Elamur Research⌕
☰
home / papers / 2018 / Learning Latent Dynamics for Planning from Pixels

Learning Latent Dynamics for Planning from Pixels

Paper · 2018 · Danijar Hafner, Timothy Lillicrap, Ian Fischer et al.

Summary

Planning works well when an agent knows how its environment behaves. Learning such dynamics from raw images accurately enough to plan with them has long been difficult. PlaNet (Deep Planning Network) addresses this. It is a fully model-based agent that learns a compact latent dynamics model from 64×64 pixel observations and selects actions by replanning at every step with the cross entropy method (CEM). Planning runs entirely in latent space and never generates images. The central component is a recurrent state-space model (RSSM) whose transitions contain both deterministic and stochastic parts. The authors also introduce latent overshooting, a variational objective that rewards accurate predictions several steps ahead. Training alternates between fitting the model and collecting new episodes with the current planner. On six DeepMind control suite tasks with contacts, partial observability and sparse rewards, PlaNet nearly matches or sometimes beats strong model-free agents while needing far less experience. This shows that learned latent models can make planning from images practical.

Key points

  • PlaNet uses about 1,000 episodes, while A3C and D4PG use 100,000, which is roughly 200× less interaction on average at similar computation time.
  • Final scores against D4PG: Cheetah Run 662 vs 524, Cup Catch 930 vs 980, Walker Walk 951 vs 968. PlaNet beats A3C on every task, even though A3C uses proprioceptive input.
  • Ablations show that removing either path hurts planning: a purely deterministic model (GRU) and a purely stochastic model (SSM) both perform worse than the RSSM.
  • CEM planning with the true simulator serves as an approximate upper bound, for example 850 on Cartpole Swing Up versus PlaNet's 821.
  • The planner uses no policy or value network, and the model is not tied to a specific task. The authors point to value learning, temporal abstraction and multi-task transfer as future directions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓University of TorontoToronto · Canada110 works · 179,061 citations🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations🔬Hasso Plattner InstitutePotsdam · Germany14 works · 5,514 citations

Also on the byline: Southmead Hospital.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee and James Davidson. 2018. Learning Latent Dynamics for Planning from Pixels. arXiv:1811.04551.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.04551 · arXiv 1811.04551

@misc{hafner2018learning,
  title = {{Learning Latent Dynamics for Planning from Pixels}},
  author = {Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee and James Davidson},
  year = {2018},
  doi = {10.48550/arxiv.1811.04551},
  eprint = {1811.04551},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.04551}
}

References

59 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, P., Nair, A. V., Abbeel, P., Malik, J., and Levine, S. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems, pp. 5074–5082, 2016. find
  2. Amos, B., Dinh, L., Cabi, S., Rothörl, T., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. Learning awareness models. In International Conference on Learning Representations, 2018. find
  3. Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252, 2017. — [arXiv:1710.11252](https://arxiv.org/abs/1710.11252) find
  4. Banijamali, E., Shu, R., Ghavamzadeh, M., Bui, H., and Ghodsi, A. Robust locally-linear controllable embedding. arXiv preprint arXiv:1710.05373, 2017. — [arXiv:1710.05373](https://arxiv.org/abs/1710.05373) find
  5. Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Muldal, A., Heess, N., and Lillicrap, T. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018. — [arXiv:1804.08617](https://arxiv.org/abs/1804.08617) find
  6. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171–1179, 2015. find
  7. Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. arXiv preprint arXiv:1807.01675, 2018. — [arXiv:1807.01675](https://arxiv.org/abs/1807.01675) find
  8. Buesing, L., Weber, T., Racaniere, S., Eslami, S., Rezende, D., Reichert, D. P., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006, 2018. — [arXiv:1802.03006](https://arxiv.org/abs/1802.03006) find
  9. Chiappa, S., Racaniere, S., Wierstra, D., and Mohamed, S. Recurrent environment simulators. arXiv preprint arXiv:1704.02254, 2017. — [arXiv:1704.02254](https://arxiv.org/abs/1704.02254) find
  10. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  11. Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114, 2018. find
  12. Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., and Bengio, Y. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980–2988, 2015. find
  13. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
  14. Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465–472, 2011. find
  15. Denton, E. and Fergus, R. Stochastic video generation with a learned prior. arXiv preprint arXiv:1802.07687, 2018. — [arXiv:1802.07687](https://arxiv.org/abs/1802.07687) find
  16. Dillon, J. V., Langmore, I., Tran, D., Brevdo, E., Vasudevan, S., Moore, D., Patton, B., Alemi, A., Hoffman, M., and Saurous, R. A. Tensorflow distributions. arXiv preprint arXiv:1711.10604, 2017. — [arXiv:1711.10604](https://arxiv.org/abs/1711.10604) find
  17. Doerr, A., Daniel, C., Schiegg, M., Nguyen-Tuong, D., Schaal, S., Toussaint, M., and Trimpe, S. Probabilistic recurrent state-space models. arXiv preprint arXiv:1801.10395, 2018. — [arXiv:1801.10395](https://arxiv.org/abs/1801.10395) find
  18. Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., and Levine, S. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568, 2018. — [arXiv:1812.00568](https://arxiv.org/abs/1812.00568) find
  19. Finn, C. and Levine, S. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 2786–2793. find
  20. Gal, Y., McAllister, R., and Rasmussen, C. E. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, ICML, 2016. find
  21. Gemici, M., Hung, C.-C., Santoro, A., Wayne, G., Mohamed, S., Rezende, D. J., Amos, D., and Lillicrap, T. Generative temporal models with memory. arXiv preprint arXiv:1702.04649, 2017. — [arXiv:1702.04649](https://arxiv.org/abs/1702.04649) find
  22. Gregor, K. and Besse, F. Temporal difference variational auto-encoder. arXiv preprint arXiv:1806.03107, 2018. — [arXiv:1806.03107](https://arxiv.org/abs/1806.03107) find
  23. Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122, 2018. find
  24. Henaff, M., Whitney, W. F., and LeCun, Y. Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705.07177, 2018. — [arXiv:1705.07177](https://arxiv.org/abs/1705.07177) find
  25. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016. find
  26. Higuera, J. C. G., Meger, D., and Dudek, G. Synthesizing neural network controllers with probabilistic model based reinforcement learning. arXiv preprint arXiv:1803.02291, 2018. — [arXiv:1803.02291](https://arxiv.org/abs/1803.02291) find
  27. Igl, M., Zintgraf, L., Le, T. A., Wood, F., and Whiteson, S. Deep variational reinforcement learning for pomdps. arXiv preprint arXiv:1806.02426, 2018. — [arXiv:1806.02426](https://arxiv.org/abs/1806.02426) find
  28. Kalchbrenner, N., Oord, A. v. d., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. Video pixel networks. arXiv preprint arXiv:1610.00527, 2016. find
  29. Kalweit, G. and Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, pp. 195–206, 2017. find
  30. Karl, M., Soelch, M., Bayer, J., and van der Smagt, P. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432, 2016. — [arXiv:1605.06432](https://arxiv.org/abs/1605.06432) find
  31. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  32. Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039, 2018. find
  33. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  34. Krishnan, R. G., Shalit, U., and Sontag, D. Deep kalman filters. arXiv preprint arXiv:1511.05121, 2015. — [arXiv:1511.05121](https://arxiv.org/abs/1511.05121) find
  35. Krishnan, R. G., Shalit, U., and Sontag, D. Structured inference networks for nonlinear state space models. In AAAI, pp. 2101–2109, 2017. find
  36. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018. — [arXiv:1802.10592](https://arxiv.org/abs/1802.10592) find
  37. Lamb, A. M., GOYAL, A. G. A. P., Zhang, Y., Zhang, S., Courville, A. C., and Bengio, Y. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp. 4601–4609, 2016. find
  38. Mathieu, M., Couprie, C., and LeCun, Y. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015. — [arXiv:1511.05440](https://arxiv.org/abs/1511.05440) find
  39. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529, 2015. find
  40. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928– 1937, 2016. find
  41. Moerland, T. M., Broekens, J., and Jonker, C. M. Learning multimodal transition dynamics for model-based reinforcement learning. arXiv preprint arXiv:1705.00470, 2017. — [arXiv:1705.00470](https://arxiv.org/abs/1705.00470) find
  42. Moravčík, M., Schmid, M., Burch, N., Lisỳ, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508–513, 2017. find
  43. Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv:1708.02596, 2017. find
  44. Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010. find
  45. Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pp. 2863–2871, 2015.
  46. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
  47. Richards, A. G. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005. find
  48. Rubinstein, R. Y. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89–112, 1997. find
  49. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017. find
  50. Srinivas, A., Jabri, A., Abbeel, P., Levine, S., and Finn, C. Universal planning networks. arXiv preprint arXiv:1804.00645, 2018. — [arXiv:1804.00645](https://arxiv.org/abs/1804.00645) find
  51. Talvitie, E. Model regularization for stable sample rollouts. In UAI, pp. 780–789, 2014. find
  52. Tassa, Y., Erez, T., and Todorov, E. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 4906– 4913. find
  53. Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. van den Oord, A., Vinyals, O., et al. Neural discrete representation learning. In Advances in Neural Inform… find
  54. Venkatraman, A., Hebert, M., and Bagnell, J. A. Improving multi-step prediction of learned time series models. In AAAI, pp. 3024–3030, 2015. find
  55. Vondrick, C., Pirsiavash, H., and Torralba, A. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, 2016. find
  56. Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746–2754, 2015. find
  57. Wayne, G., Hung, C.-C., Amos, D., Mirza, M., Ahuja, A., Grabska-Barwinska, A., Rae, J., Mirowski, P., Leibo, J. Z., Santoro, A., et al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint arXiv:1803.10760, 2018. find
  58. Weber, T., Racanière, S., Reichert, D. P., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203, 2017. find
  59. Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M., and Levine, S. SOLAR: deep structured representations for model-based reinforcement learning. arXiv preprint arXiv:1808.09105, 2018. — [arXiv:1808.09105](https://arxiv.org/abs/1808.09105) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hafner-2018-learning-latent-dynamics-planning-pixels/