Elamur Research⌕
☰
home / papers / 2018 / Diversity is All You Need: Learning Skills without a Reward Function

Diversity is All You Need: Learning Skills without a Reward Function

Paper · 2018 · Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz et al.

Summary

The paper asks whether a reinforcement learning agent can acquire a repertoire of useful behaviours before it is ever given a task reward. The authors introduce DIAYN ("Diversity is All You Need"). The agent conditions its policy on a latent skill code drawn from a fixed uniform prior. A learned discriminator tries to guess which skill is active from each visited state. The policy receives a pseudo-reward of log q_φ(z|s) − log p(z) and is trained with SAC, while the discriminator is trained with SGD. Maximum-entropy training forces skills to stay as random as possible while remaining recognizable, which pushes them toward distinct regions of the state space.

On classic control tasks and simulated robots such as half-cheetah, hopper and ant, varied behaviours appear without supervision, including running, flipping and rotating. Some skills even solve the benchmark tasks without ever seeing the true reward. The learned skills can then:

  • initialise policies for new tasks,
  • serve as primitives for hierarchical RL on sparse-reward problems, and
  • support imitation.

This makes unsupervised skill discovery a promising way to pretrain agents and ease exploration.

Key points

  • Objective: maximise I(S; Z) + H[A|S] − I(A; Z|S). This rearranges to H[Z] − H[Z|S] + H[A|S,Z]: skills should be identifiable from states, not actions, and should act with maximum entropy.
  • Differences from Variational Intrinsic Control (Gregor et al., 2016):
    • maximum-entropy policies,
    • a fixed rather than learned p(z), which prevents collapse onto a few skills,
    • a discriminator applied at every state rather than only the final one.
  • Inverted Pendulum and Mountain Car: several distinct skills solve each task, differing in balancing position, oscillation pattern, or the strategy used to climb the hill.
  • Imitation experiment: across 600 synthetic expert trajectories, the skill with the highest discriminator likelihood is selected. DIAYN with 50 skills achieves the smallest L2 state distance to the expert, outperforming the "low entropy", "learned p(z)" and "few skills" (5 skills) baselines.
  • Predictive score: the discriminator score anticipates imitation quality before execution, which could help in safety-critical settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksControl and Systems Engineering · 62 works

Cite

Cite as Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz and Sergey Levine. 2018. Diversity is All You Need: Learning Skills without a Reward Function. arXiv:1802.06070.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.06070 · arXiv 1802.06070

@misc{eysenbach2018diversity,
  title = {{Diversity is All You Need: Learning Skills without a Reward Function}},
  author = {Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz and Sergey Levine},
  year = {2018},
  doi = {10.48550/arxiv.1802.06070},
  eprint = {1802.06070},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.06070}
}

References

50 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational autoencoding learning of options by reinforcement. NIPS Deep Reinforcement Learning Symposium, 2017. find
  2. David Barber Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in Neural Information Processing Systems, 16:201, 2004. find
  3. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1726–1734, 2017. find
  4. Adrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49–73, 2013. find
  5. Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471–1479, 2016. find
  6. Christopher M Bishop. Pattern Recognition and Machine Learning. Springer-Verlag New York, 2016. find
  7. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. find
  8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4302–4310, 2017. find
  9. Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pp. 271–278, 1993. find
  10. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329–1338, 2016. find
  11. Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv:1704.03012, 2017. — [arXiv:1704.03012](https://arxiv.org/abs/1704.03012) find
  12. Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710.09767, 2017. find
  13. Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2574–2584, 2017. find
  14. Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016. — [arXiv:1611.07507](https://arxiv.org/abs/1611.07507) find
  15. Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 3389–3396. find
  16. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165, 2017. — [arXiv:1702.08165](https://arxiv.org/abs/1702.08165) find
  17. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018. find
  18. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, pp. 6768–6777, 2017. find
  19. Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rk07ZXZRb. ↗
  20. Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.05182, 2016. — [arXiv:1610.05182](https://arxiv.org/abs/1610.05182) find
  21. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017. find
  22. Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109–1117, 2016. find
  23. Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous agent—environment systems. Adaptive Behavior, 19(1):16–39, 2011. find
  24. Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations. In Conference on Robot Learning, pp. 418–437, 2017. find
  25. Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189–223, 2011a. find
  26. Joel Lehman and Kenneth O Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 211–218. ACM, 2011b. find
  27. Robert K Merton. The matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810):56–63, 1968. find
  28. Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016. find
  29. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  30. Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125–2133, 2015. find
  31. Jean-Baptiste Mouret and Stéphane Doncieux. Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity. In Evolutionary Computation, 2009. CEC’09. IEEE Congress on, pp. 1161–1168. find
  32. Kevin P Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012. find
  33. Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2772–2782, 2017. find
  34. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265–286, 2007. find
  35. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv:1705.05363, 2017. find
  36. Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep rl for model-based control. arXiv preprint arXiv:1802.09081, 2018. — [arXiv:1802.09081](https://arxiv.org/abs/1802.09081) find
  37. Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016. find
  38. Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54–67, 2000. find
  39. Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation. IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010. find
  40. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015a. find
  41. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b. find
  42. John Schulman, Pieter Abbeel, and Xi Chen. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017. find
  43. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. find
  44. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  45. Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99–127, 2002. find
  46. Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017. — [arXiv:1712.06567](https://arxiv.org/abs/1712.06567) find
  47. Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017. find
  48. Brian G Woolley and Kenneth O Stanley. On the deleterious effects of a priori objectives on evolution and representation. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 957–964. ACM, 2011. find
  49. Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 3357–3364. find
  50. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eysenbach-2018-diversity-all-need-learning-skills/