Elamur Research⌕
☰
home / papers / 2015 / Massively Parallel Methods for Deep Reinforcement Learning

Massively Parallel Methods for Deep Reinforcement Learning

Paper · 2015 · Arun Nair, Praveen Srinivasan, Sam Blackwell et al.

Summary

DQN, the deep Q-learning agent that learns Atari 2600 games from raw pixels, had only been run on a single machine. Training one game took about two weeks of GPU time. Researchers at Google DeepMind introduce Gorila (General Reinforcement Learning Architecture), which they present as the first massively distributed framework for deep RL. It splits the work into four parts. Many actors interact with their own copies of the environment. Separate learners compute updates from sampled experience. The Q-network's parameters sit on a DistBelief-style sharded parameter server updated through ASGD. The experience replay memory is spread across machines. The authors applied Gorila to DQN on 49 games from the Arcade Learning Environment, with one shared set of hyperparameters. Gorila DQN beat the single-GPU baseline on 41 games in roughly half the training time, and it reached the baseline's scores about ten times faster on most games. The results show that DQN keeps improving with more compute and more parallel data. Gorila also gives a general template for scaling RL beyond one GPU.

Key points

  • Architecture: Gorila separates acting, learning, parameter storage (sharded parameter server, as in DistBelief) and a distributed replay memory. Learners apply asynchronous gradient updates to a DQN variant.
  • Training budget: Gorila DQN used 100 parallel actors and trained for up to 6 days. The single-GPU DQN baseline trained for 12–14 days.
  • Human-starts evaluation: Gorila beat single-GPU DQN on 41 of 49 games. It doubled the baseline's score on 22 games and scored 5× higher on 11.
  • Null-op evaluation: Gorila won on 31 of 49 games. The stronger human-starts results suggest better generalization to unfamiliar states. On 25 games it reached at least 75% of a professional human's score.
  • Speed: Gorila overtook the best single-GPU DQN scores on 19 games within 6 hours, 23 within 12 hours, 30 within 24 hours and 38 within 36 hours. On many games its scores were still rising when training stopped.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksHardware and Architecture · 167 works

Cite

Cite as Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu and David Silver. 2015. Massively Parallel Methods for Deep Reinforcement Learning. arXiv:1507.04296.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1507.04296 · arXiv 1507.04296

@misc{nair2015massively,
  title = {{Massively Parallel Methods for Deep Reinforcement Learning}},
  author = {Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro {De Maria}, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu and David Silver},
  year = {2015},
  doi = {10.48550/arxiv.1507.04296},
  eprint = {1507.04296},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1507.04296}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bellemare, Marc G, Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. arXiv preprint arXiv:1207.4708, 2012.
  2. Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with cots hpc systems. In Proceedings of The 30th International Conference on Machine Learning, pp. 1337–1345, 2013. find
  3. Dahl, George E, Yu, Dong, Deng, Li, and Acero, Alex. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 20(1):30– 42, 2012. find
  4. Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, pp. 1223–1231, 2012. find
  5. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. find
  6. Graves, Alex, Mohamed, A-R, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. find
  7. Grounds, Matthew and Kudenko, Daniel. Parallel reinforcement learning with linear function approximation. In Proceedings of the 5th, 6th and 7th European Conference on Adaptive and Learning Agents and Multi-agent Systems: Adaptation and Multi-agent Learning, pp. 60– 74. Springer-Verlag, 2008. find
  8. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoff. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1106–1114, 2012.
  9. Lauer, Martin and Riedmiller, Martin. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 535– 542. Morgan Kaufmann, 2000. find
  10. Li, Yuxi and Schuurmans, Dale. Mapreduce for parallel reinforcement learning. In Recent Advances in Reinforcement Learning - 9th European Workshop, EWRL 2011, Athens, Greece, September 9-11, 2011, Revised Selected Papers, pp. 309–320, 2011. find
  11. Lin, Long-Ji. Reinforcement learning for robots using neural networks. Technical report, DTIC Document, 1993. find
  12. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013. find
  13. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg,… find
  14. Silver, David, Newnham, Leonard, Barker, David, Weller, Suzanne, and McFall, Jason. Concurrent reinforcement learning from customer interactions. In Proceedings of the 30th International Conference on Machine Learning, pp. 924–932, 2013. find
  15. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  16. Sutton, R. and Barto, A. Reinforcement Learning: an Introduction. MIT Press, 1998. find
  17. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. find
  18. Tsitsiklis, J. and Roy, B. Van. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997. find
  19. Weiss, Gerhard. Distributed reinforcement learning. 15: 135–142, 1995. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nair-2015-massively-parallel-methods-deep-reinforcement/