Elamur Research⌕
☰
home / papers / 2016 / Reinforcement Learning with Unsupervised Auxiliary Tasks

Reinforcement Learning with Unsupervised Auxiliary Tasks

Paper · 2016 · Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki et al.

Summary

Deep RL agents usually learn only from extrinsic reward, which is often sparse. That leaves most of the information in an agent's observation stream unused. Jaderberg et al. (DeepMind) propose UNREAL (UNsupervised REinforcement and Auxiliary Learning), which extends the A3C actor-critic with extra objectives that need no additional environment signals. These objectives share the agent's CNN-LSTM representation.

Two kinds of auxiliary objectives are added:

  • Control tasks, learned off-policy with Q-learning. Pixel control rewards the agent for changing pixel intensities in image regions, and feature control does the same for hidden-layer activations.
  • Reward-focused tasks, drawn from a small replay buffer. Reward prediction is trained on sequences sampled with a bias toward rewarding events, and value function replay gives the critic extra updates.

Because the representation keeps developing when extrinsic reward is absent, the agent learns its main task much faster. The method is evaluated on raw-pixel Atari and on first-person 3D Labyrinth levels. It beats vanilla A3C, including a variant with a pixel-reconstruction loss, on learning speed, hyperparameter robustness and final score. This suggests that predicting and controlling features of experience is a stronger route to representation learning than plain reconstruction.

Key points

  • Labyrinth: UNREAL reaches on average 87% of expert human-normalised score versus 54% for A3C, and learns about 10× faster.
  • Atari: the mean is 880% of expert human performance, which surpassed the earlier state of the art.
  • Pixel control: the task acts on the central 80×80 crop, split into a 20×20 grid of 4×4 cells. Each cell's reward is the mean absolute change from the previous frame. Q-values come from a deconvolution head with a dueling parametrization.
  • Reward prediction: the network looks at three past frames and classifies the next reward as positive, negative or zero.
  • Setup:
    • Input is 84×84 RGB images, processed by the Mnih et al. (2013) CNN and an LSTM with 256 cells.
    • Training uses 20-step unrolls, a replay buffer holding the most recent 2k transitions, and 32 asynchronous threads with shared RMSprop.
    • Labyrinth agents have 17 actions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓Jagiellonian UniversityKrakow · Poland5 works · 575 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver and Koray Kavukcuoglu. 2016. Reinforcement Learning with Unsupervised Auxiliary Tasks. arXiv:1611.05397.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.05397 · arXiv 1611.05397

@misc{jaderberg2016reinforcement,
  title = {{Reinforcement Learning with Unsupervised Auxiliary Tasks}},
  author = {Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver and Koray Kavukcuoglu},
  year = {2016},
  doi = {10.48550/arxiv.1611.05397},
  eprint = {1611.05397},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.05397}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. André Barreto, Rémi Munos, Tom Schaul, and David Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv:1606.05312, 2016. — [arXiv:1606.05312](https://arxiv.org/abs/1606.05312) find
  2. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2012.
  3. OpenArena contributors. The openarena manual. 2005. URL http://openarena.wikia.com/wiki/Manual. ↗
  4. Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613–624, 1993. find
  5. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000. id software. Quake3. Quake-III-Arena. find
  6. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv preprint arXiv:1605.02097, 2016. — [arXiv:1605.02097](https://arxiv.org/abs/1605.02097) find
  7. George Konidaris and Andre S Barreto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems, pp. 1015– 1023, 2009. find
  8. Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016. — [arXiv:1606.02396](https://arxiv.org/abs/1606.02396) find
  9. Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. CoRR, abs/1609.05521, 2016. find
  10. Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent reinforcement learning: A hybrid approach. arXiv preprint arXiv:1509.03044, 2015. — [arXiv:1509.03044](https://arxiv.org/abs/1509.03044) find
  11. Long-Ji Lin and Tom M Mitchell. Memory approaches to reinforcement learning in non-markovian domains. Technical report, Carnegie Mellon University, School of Computer Science, 1992. find
  12. Piotr Mirowski, Razvan Pascanu, Fabio Viola, Andrea Banino, Hubert Soyer, Andy Ballard, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. 2016. find
  13. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013. find
  14. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… find
  15. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pp. 1928–1937, 2016. find
  16. Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pp. 2863–2871, 2015.
  17. Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605.09128, 2016. — [arXiv:1605.09128](https://arxiv.org/abs/1605.09128) find
  18. Jing Peng and Ronald J Williams. Incremental multi-step q-learning. Machine Learning, 22(1-3): 283–290, 1996. find
  19. Daniel L Schacter, Donna Rose Addis, Demis Hassabis, Victoria C Martin, R Nathan Spreng, and Karl K Szpunar. The future of memory: remembering, imagining, and the brain. Neuron, 76(4): 677–694, 2012. find
  20. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1312–1320, 2015a. find
  21. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015b. find
  22. Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010. find
  23. David Silver and Kamil Ciosek. Compositional planning using optimal option models. arXiv preprint arXiv:1206.6473, 2012. — [arXiv:1206.6473](https://arxiv.org/abs/1206.6473) find
  24. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. find
  25. Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPS, volume 99, pp. 1057– 1063, 1999a. find
  26. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999b. find
  27. Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent System… find
  28. Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. arXiv preprint arXiv:1604.07255, 2016. find
  29. Z. Wang, N. de Freitas, and M. Lanctot. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016. find
  30. Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989. find
  31. Christopher Xie, Sachin Patil, Teodor Mihai Moldovan, Sergey Levine, and Pieter Abbeel. Model-based reinforcement learning with parametrized physical models and optimism-driven exploration. CoRR, abs/1509.06824, 2015. — [arXiv:1509.06824](https://arxiv.org/abs/1509.06824) find
  32. Tom Zahavy, Nir Ben Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In Proceedings of the 33rd International Conference on Machine Learning, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jaderberg-2016-reinforcement-learning-unsupervised-auxiliary-tasks/