Elamur Research⌕
☰
home / papers / 2017 / Hindsight Experience Replay

Hindsight Experience Replay

Paper · 2017 · Marcin Andrychowicz, Filip Wolski, Alex Ray et al.

Summary

Reinforcement learning (RL) struggles with tasks in which the reward is sparse and binary. The agent almost never observes success, so developers have to hand-craft complex reward functions. The authors from OpenAI propose Hindsight Experience Replay (HER). This technique makes it possible to learn from failed attempts as well. An episode in which the agent did not reach the intended goal is replayed from the buffer as if the goal had been the state the agent actually reached. To this end, the policy and the value function take as input both the current state and the goal, as in UVFA. The method can be combined with any off-policy algorithm, and the authors tested it with DQN and DDPG. The experiments were carried out in a simulated robotic arm on three tasks: pushing, sliding an object, and pick-and-place. The agent received only a binary signal indicating whether the task had been completed. Plain DDPG does not solve these tasks, but with HER they are solved. The policy trained in the simulator was transferred to a real Fetch robot. The value of the work is that it removes the need for manual reward shaping and acts as an implicit curriculum.

Key points

  • In a toy bit-flipping environment, DQN without HER copes only with n ≤ 13 bits, whereas with HER it handles up to n = 50.
  • The authors compared four strategies for choosing substitute goals: final, episode, random and future. The best is future with k = 4 or 8, and it is the only one that almost fully solves the sliding task. With k greater than 8, results deteriorate because too little ordinary data remains in the buffer.
  • Training the agent to reach many goals turned out to be easier than training it to reach one. HER is therefore useful even when only a single task is needed.
  • The pick-and-place policy was transferred to the physical Fetch robot without fine-tuning. The box position was estimated by a separately trained CNN from the robot's camera images. At first, 2 out of 5 attempts succeeded, and after retraining with Gaussian noise in the observations (std 1 cm), 5 out of 5.
  • The paper appeared at NIPS 2017. HER does not depend on prioritized experience replay, so the two methods can easily be combined.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksControl and Systems Engineering · 62 works

Cite

Cite as Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel and Wojciech Zaremba. 2017. Hindsight Experience Replay. arXiv:1707.01495.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.01495 · arXiv 1707.01495

@misc{andrychowicz2017hindsight,
  title = {{Hindsight Experience Replay}},
  author = {Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel and Wojciech Zaremba},
  year = {2017},
  doi = {10.48550/arxiv.1707.01495},
  eprint = {1707.01495},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.01495}
}

References

46 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.
  2. Bakker, B. and Schmidhuber, J. (2004). Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization. In Proc. of the 8-th Conf. on Intelligent Autonomous Systems, pages 438–445. find
  3. Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471–1479. find
  4. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM.
  5. Caruana, R. (1998). Multitask learning. In Learning to learn, pages 95–133. Springer. find
  6. Chebotar, Y., Kalakrishnan, M., Yahya, A., Li, A., Schaal, S., and Levine, S. (2016). Path integral guided policy search. arXiv preprint arXiv:1610.00529. — [arXiv:1610.00529](https://arxiv.org/abs/1610.00529) find
  7. Da Silva, B., Konidaris, G., and Barto, A. (2012). Learning parameterized skills. arXiv preprint arXiv:1206.6398. — [arXiv:1206.6398](https://arxiv.org/abs/1206.6398) find
  8. Devin, C., Gupta, A., Darrell, T., Abbeel, P., and Levine, S. (2016). Learning modular neural network policies for multi-task and multi-robot transfer. arXiv preprint arXiv:1609.07088. find
  9. Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71–99. find
  10. Foster, D. and Dayan, P. (2002). Structure in the space of value functions. Machine Learning, 49(2):325–346. find
  11. Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. (2017). Automated curriculum learning for neural networks. arXiv preprint arXiv:1704.03003. find
  12. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476. find
  13. Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. (2016). Continuous deep q-learning with model-based acceleration. arXiv preprint arXiv:1603.00748. — [arXiv:1603.00748](https://arxiv.org/abs/1603.00748) find
  14. Held, D., Geng, X., Florensa, C., and Abbeel, P. (2017). Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv:1705.06366.
  15. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109– 1117. find
  16. Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  17. Kober, J., Wilhelm, A., Oztop, E., and Peters, J. (2012). Reinforcement learning to adjust parametrized motor primitives to new situations. Autonomous Robots, 33(4):361–379. find
  18. Kolter, J. Z. and Ng, A. Y. (2009). Near-bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 513–520. ACM. find
  19. Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2015). End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702. find
  20. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. find
  21. Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321. find
  22. Metz, L., Ibarz, J., Jaitly, N., and Davidson, J. (2017). Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035. — [arXiv:1705.05035](https://arxiv.org/abs/1705.05035) find
  23. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. find
  24. Ng, A., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., and Liang, E. (2006). Autonomous inverted helicopter flight via reinforcement learning. Experimental Robotics IX, pages 363–372. find
  25. Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278–287. find
  26. Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. (2016). Deep exploration via bootstrapped dqn. In Advances In Neural Information Processing Systems, pages 4026–4034.
  27. Ostrovski, G., Bellemare, M. G., Oord, A. v. d., and Munos, R. (2017). Count-based exploration with neural density models. arXiv preprint arXiv:1703.01310. — [arXiv:1703.01310](https://arxiv.org/abs/1703.01310) find
  28. Peters, J. and Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682–697. find
  29. Pinto, L. and Gupta, A. (2016). Learning to push by grasping: Using multiple tasks for effective learning. arXiv preprint arXiv:1609.09025. — [arXiv:1609.09025](https://arxiv.org/abs/1609.09025) find
  30. Polyak, B. T. and Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855. find
  31. Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerik, M., Lampe, T., Tassa, Y., Erez, T., and Riedmiller, M. (2017). Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073. — [arXiv:1704.03073](https://arxiv.org/abs/1704.03073) find
  32. Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015a). Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1312–1320. find
  33. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015b). Prioritized experience replay. arXiv preprint arXiv:1511.05952. find
  34. Schmidhuber, J. (2004). Optimal ordered problem solver. Machine Learning, 54(3):211–254. find
  35. Schmidhuber, J. (2013). Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4. find
  36. Schmidhuber, J. and Huber, R. (1990). Learning to generate focus trajectories for attentive vision. Institut für Informatik. find
  37. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489. find
  38. Srivastava, R. K., Steunebrink, B. R., and Schmidhuber, J. (2013). First experiments with powerplay. Neural Networks, 41:130–136. find
  39. Strehl, A. L. and Littman, M. L. (2005). A theoretical analysis of model-based interval estimation. In Proceedings of the 22nd international conference on Machine learning, pages 856–863. ACM. find
  40. Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R. (2017). Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407. find
  41. Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., and Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2,… find
  42. Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). \# exploration: A study of count-based exploration for deep reinforcement learning. arXiv preprint arXiv:1611.04717. find
  43. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. arXiv preprint arXiv:1703.06907. find
  44. Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026–5033. IEEE. find
  45. Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. (2017). Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161. find
  46. Zaremba, W. and Sutskever, I. (2014). Learning to execute. arXiv preprint arXiv:1410.4615. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/