Elamur Research⌕
☰
home / papers / 2017 / Automatic Goal Generation for Reinforcement Learning Agents

Automatic Goal Generation for Reinforcement Learning Agents

Paper · 2017 · Carlos Florensa, David Held, Xinyang Geng et al.

Summary

Standard reinforcement learning trains an agent to optimize a single reward, which works poorly when a robot must handle many related tasks, such as reaching arbitrary positions. Florensa, Held, Geng and Abbeel (ICML 2018) aim to maximize the average success rate across all feasible goals. Each goal is a region of the state space, and the agent gets a sparse binary reward only when it enters that region.

Their key idea is the Goal GAN, an adversarially trained generative model. The discriminator learns which goals are of intermediate difficulty for the current policy, meaning neither always nor never reached. The generator then proposes new goals with that property. Training cycles through three steps: label goals, refit the generator, and update the goal-conditioned policy. This produces a curriculum that keeps pace with the agent and needs no prior knowledge of the environment or reward shaping.

Experiments with point-mass and Ant agents show that the method reaches near-full goal coverage more efficiently than uniform goal sampling and SAGG-RIAC. Its advantage grows when feasible goals occupy only a small part of the space.

Key points

  • Goal labeling: goals are labeled by their empirical success rate. Those falling between thresholds R_min and R_max count as appropriately difficult. A regularized replay buffer of earlier goals is mixed in, for example 200 goals from the Goal GAN plus 100 from replay per iteration.
  • Multi-path point-mass task: coverage rises from 0.014 at iteration 1 to 0.53 at iteration 10, 0.78 at iteration 30 and 0.98 at iteration 100. The generator tracks several difficulty "fronts" at once without knowing how many modes exist.
  • N-dimensional Point Mass:
    • The state space is the hypercube [−5, 5]^N.
    • Feasible states make up 20% of the space in 2D and only about 0.00023:1 at N = 6.
    • Uniform sampling degrades sharply as N grows, while Goal GAN stays robust.
    • Results are measured after 200 outer iterations, averaged over 5 seeds.
  • Proposed future work: combining the method with HER (hindsight experience replay) and building hierarchical policies on top of the multi-goal policy. Code and videos are released on the project site.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Carlos Florensa, David Held, Xinyang Geng and Pieter Abbeel. 2017. Automatic Goal Generation for Reinforcement Learning Agents. arXiv:1705.06366.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.06366 · arXiv 1705.06366

@misc{florensa2017automatic,
  title = {{Automatic Goal Generation for Reinforcement Learning Agents}},
  author = {Carlos Florensa, David Held, Xinyang Geng and Pieter Abbeel},
  year = {2017},
  doi = {10.48550/arxiv.1705.06366},
  eprint = {1705.06366},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.06366}
}

References

45 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J. and Sastry, S. Surprise-based intrinsic motivation for deep reinforcement learning. In 2016 Deep Reinforcement Learning Workshop at NIPS, 2016. find
  2. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, 2017.
  3. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. In Stat, 2017. find
  4. Baldassarre, G. and Mirolli, M. Temporal-difference competence-based intrinsic motivation (td-cb-im): A mechanism that uses the td-error as an intrinsic reinforcement for deciding which skill to learn when. Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 255–276, 2012. find
  5. Baranes, A. and Oudeyer, P.-Y. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pp. 1766–1773. find
  6. Baranes, A. and Oudeyer, P.-Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49–73, 2013a. find
  7. Baranes, A. and Oudeyer, P.-Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1), 2013b. find
  8. Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 2016. find
  9. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, 2015. find
  10. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In International Conference on Machine Learning, pp. 41–48. ACM, 2009.
  11. Deisenroth, M. P., Neumann, G., Peters, J., et al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1–2):1–142, 2013. find
  12. Deisenroth, M. P., Englert, P., Peters, J., and Fox, D. Multi-task policy search for robotics. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pp. 3876–3881. find
  13. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning, 2016. find
  14. Fabisch, A. and Metzen, J. H. Active contextual policy search. Journal of Machine Learning Research, 15(1): 3371–3399, 2014. find
  15. Florensa, C., Duan, Y., and Abbeel, P. Stochastic neural networks for hierarchical reinforcement learning. In International Conference in Learning Representations, 2017a. find
  16. Florensa, C., Held, D., Wulfmeier, M., and Abbeel, P. Reverse curriculum generation for reinforcement learning. Conference on Robot Learning (CoRL), 2017b. find
  17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. In Advances in neural information processing systems, pp. 2672–2680, 2014. find
  18. Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. Automated Curriculum Learning for Neural Networks. arXiv preprint, arXiv:1704.03003, 2017. find
  19. Guo, X., Singh, S., Lewis, R., and Lee, H. Deep learning for reward design to improve monte carlo tree search in atari games. Conference on Uncertainty in Artificial Intelligence (UAI), 2016. find
  20. Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv: 1610.05182, 2016. — [arXiv:1610.05182](https://arxiv.org/abs/1610.05182) find
  21. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. Variational information maximizing exploration. Advances in Neural Information Processing Systems, 2016. find
  22. Karpathy, A. and Van De Panne, M. Curriculum learning for motor skills. In Canadian Conference on Artificial Intelligence, pp. 325–330. Springer, 2012. find
  23. Konidaris, G., Kuindersma, S., Grupen, R. A., and Barto, A. G. Autonomous skill acquisition on a mobile manipulator. In AAAI, 2011. find
  24. Kupcsik, A. G., Deisenroth, M. P., Peters, J., Neumann, G., and Others. Data-efficient generalization of robot skills with contextual policy search. In AAAI Conference on Artificial Intelligence, pp. 1401–1407, 2013. find
  25. Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. find
  26. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint, arXiv: 1509.02971, 2015. find
  27. Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smolley, S. P. On the effectiveness of least squares generative adversarial networks. arXiv preprint arXiv:1712.06391, 2017. — [arXiv:1712.06391](https://arxiv.org/abs/1712.06391) find
  28. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015. find
  29. Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., Kavukcuoglu, K., et al. Strategic attentive writer for learning macro-actions. ArXiv preprint, ArXiv:1606.04695, 2016. find
  30. Ranchod, P., Rosman, B., and Konidaris, G. Nonparametric bayesian reward segmentation for skill discovery using inverse reinforcement learning. In International Conference on Intelligent Robots and Systems, pp. 471–477. IEEE, 2015. find
  31. Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International Conference on Machine Learning, pp. 1312–1320, 2015. find
  32. Schmidhuber, J. Curious model-building control systems. In International Joint Conference on Neural Networks, pp. 1458–1463. IEEE, 1991. find
  33. Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010. find
  34. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015a. find
  35. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-Dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representation, 2015b. find
  36. Sharma, S. and Ravindran, B. Online Multi-Task Learning Using Biased Sampling. arXiv preprint arXiv: 1702.06053, 2017. — [arXiv:1702.06053](https://arxiv.org/abs/1702.06053) find
  37. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. find
  38. Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. arXiv preprint, arXiv: 1703.05407, 2017. find
  39. Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. In Advances in Neural Information Processing Systems, pp. 2154–2162, 2016. find
  40. Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., and Abbeel, P. \#exploration: A study of count-based exploration for deep reinforcement learning. CoRR, abs/1611.04717, 2016. find
  41. Tang, Y. and Salakhutdinov, R. Learning Stochastic Feedforward Neural Networks. Advances in Neural Information Processing Systems, 2:530–538, 2013. find
  42. Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. find
  43. Vigorito, C. M. and Barto, A. G. Intrinsically motivated hierarchical skill learning in structured environments. IEEE Transactions on Autonomous Mental Development, 2(2): 132–143, 2010. find
  44. Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find
  45. Zheng, S., Yue, Y., and Hobbs, J. Generating long-term trajectories using deep hierarchical networks. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29. 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/florensa-2017-automatic-goal-generation-reinforcement-learning/