Elamur Research⌕
☰
home / papers / 2015 / Action-Conditional Video Prediction using Deep Networks in Atari Games

Action-Conditional Video Prediction using Deep Networks in Atari Games

Paper · 2015 · Junhyuk Oh, Xiaoxiao Guo, Honglak Lee et al.

Summary

The paper studies whether a deep network can forecast future video frames when those frames depend on the agent's chosen actions as well as on past frames. This amounts to learning an environment model for vision-based reinforcement learning. The experiments are run on Atari titles drawn from the ALE (Arcade Learning Environment) benchmark suite. Their screens are high-dimensional, contain many interacting objects, and are partially observable. The authors propose two encoder–transformation–decoder networks. One uses feedforward convolutional encoding of a stacked frame history. The other uses recurrent encoding, with an LSTM placed on top of CNN features. In both, a factored multiplicative layer combines the encoded features with the action, and deconvolution layers reconstruct the next frame. Training uses a multi-step squared-error loss with a curriculum that gradually lengthens the prediction horizon. In several games, the models produce realistic frames for roughly 100 steps, and these frames remain useful for control with DQN. The learned representations also separate agent-controlled objects from the rest of the scene. The authors present this as the first long-horizon, action-conditioned prediction of high-dimensional video, which makes it a step toward model-based RL from pixels.

Key points

  • Architecture. There are two variants: feedforward encoding (early fusion of frames) and recurrent encoding (LSTM without peepholes). Both use a factorized 3-way tensor to condition on actions and deconvolution to decode.
  • Training. A curriculum moves from short-term to longer-term K-step prediction and was needed for stable training. Optimization uses SGD with BPTT.
  • Replacing emulator frames. When predicted frames are fed to a pretrained DQN, the score stays well above random play in 3 of 5 games, even with 100-step predictions.
  • Informed exploration. An ε-greedy variant uses the model to choose the action whose predicted frame is least visited, measured with a Gaussian-kernel visit count over recent frames. This improved DQN in 3 of 5 games, with the largest gain in QBert. It used a feedforward model on 84×84 grayscale input.
  • Representation analysis (Seaquest).
    • Action factors show positive similarity for shared movement directions and negative similarity for opposing directions.
    • The high-variance factors (about 40%) move only the controlled object.
    • The low-variance factors move the background and other objects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis and Satinder Singh. 2015. Action-Conditional Video Prediction using Deep Networks in Atari Games. arXiv:1507.08750.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1507.08750 · arXiv 1507.08750

@misc{oh2015action,
  title = {{Action-Conditional Video Prediction using Deep Networks in Atari Games}},
  author = {Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis and Satinder Singh},
  year = {2015},
  doi = {10.48550/arxiv.1507.08750},
  eprint = {1507.08750},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1507.08750}
}

References

37 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  2. M. G. Bellemare, J. Veness, and M. Bowling. Investigating contingency awareness using Atari 2600 games. In AAAI, 2012. find
  3. M. G. Bellemare, J. Veness, and M. Bowling. Bayesian learning of recursively factored environments. In ICML, 2013. find
  4. M. G. Bellemare, J. Veness, and E. Talvitie. Skip context tree switching. In ICML, 2014. find
  5. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127, 2009. find
  6. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, 2009.
  7. D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, 2012.
  8. A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015. find
  9. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
  10. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  11. X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In NIPS, 2014. find
  12. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. find
  13. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM Multimedia, 2014.
  14. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014. find
  15. L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. In ECML. 2006. find
  16. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  17. I. Lenz, R. Knepper, and A. Saxena. DeepMPC: Learning deep latent features for model predictive control. In RSS, 2015. find
  18. R. Memisevic. Learning to relate images. IEEE TPAMI, 35(8):1829–1846, 2013. find
  19. V. Michalski, R. Memisevic, and K. Konda. Modeling deep temporal dependencies with recurrent grammar cells. In NIPS, 2014. find
  20. R. Mittelman, B. Kuipers, S. Savarese, and H. Lee. Structured recurrent temporal restricted Boltzmann machines. In ICML, 2014. find
  21. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  22. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. find
  23. V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010. find
  24. S. Reed, K. Sohn, Y. Zhang, and H. Lee. Learning to disentangle factors of variation with manifold interaction. In ICML, 2014. find
  25. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza. Disentangling factors of variation for facial expression recognition. In ECCV. 2012. find
  26. J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. find
  27. J. Schmidhuber and R. Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2:125–134, 1991. find
  28. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICML, 2015. find
  29. I. Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted Boltzmann machine. In NIPS, 2009. find
  30. I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. In ICML, 2011. find
  31. I. Sutskever, O. Vinyals, and Q. Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
  32. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. find
  33. G. W. Taylor and G. E. Hinton. Factored conditional restricted Boltzmann machines for modeling motion style. In ICML, 2009. find
  34. T. Tieleman and G. Hinton. Lecture 6.5 - RMSProp: Divde the gradient by a running average of its recent magnitude. Coursera, 2012. find
  35. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, 2015. find
  36. C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992. find
  37. J. Yang, S. Reed, M.-H. Yang, and H. Lee. Weakly-supervised disentangling with recurrent transformations for 3D view synthesis. In NIPS, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2015-action-conditional-video-prediction-using/