Elamur Research⌕
☰
home / papers / 2021 / Deep Reinforcement Learning at the Edge of the Statistical Precipice

Deep Reinforcement Learning at the Edge of the Statistical Precipice

Paper · 2021 · Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro et al.

Summary

The authors examine how reliably algorithms are compared in deep RL when only a few runs per task are available. Because computation is expensive, researchers typically perform 3–10 runs and report only point estimates: the mean or median of the normalized score. The randomness across runs is not taken into account. Using the Atari 100k benchmark as an example, the authors show that such numbers can deviate substantially from the expected values. As a result, conclusions about which method is better are sometimes wrong. As an alternative, they propose three tools: interval estimates via stratified bootstrap, performance profiles (score distributions across all runs), and more robust aggregate metrics, primarily the interquartile mean (IQM). With this toolkit, the authors reanalyze results on ALE, Procgen, and the DeepMind Control Suite. There, too, discrepancies with earlier comparisons emerge. All recommendations are implemented in the open-source library rliable. The work is important because it provides a practical methodology for reliable evaluation without requiring hundreds of runs.

Key points

  • The paper received the Outstanding Paper Award at NeurIPS 2021. The rliable library is available on GitHub together with a Colab notebook, and the raw data of individual runs is published at gs://rl-benchmark-data.
  • Evaluating 5 runs on more than 50 Atari 2600 games under the standard protocol requires more than 1000 GPU-days. Therefore, 3–10 runs predominate in practice.
  • The median remains highly uncertain even with 50 runs and does not change if almost half of the tasks have a score of zero. The mean is easily distorted by outlier tasks. IQM is computed over the middle 50 % of all runs, so it is robust to outliers and yields narrower confidence intervals.
  • The authors recommend additionally reporting the average probability of improvement and the optimality gap.
  • On Procgen, UCB-DrAC outperforms PLR with a probability of only 40–50 %. Many improvements claimed in the literature hold with a probability of only 50–70 %.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓Université de MontréalMontreal · Canada87 works · 91,597 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville and Marc G. Bellemare. 2021. Deep Reinforcement Learning at the Edge of the Statistical Precipice. arXiv:2108.13264.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.13264 · arXiv 2108.13264

@misc{agarwal2021deep,
  title = {{Deep Reinforcement Learning at the Edge of the Statistical Precipice}},
  author = {Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville and Marc G. Bellemare},
  year = {2021},
  doi = {10.48550/arxiv.2108.13264},
  eprint = {2108.13264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.13264}
}

References

116 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020. find
  2. Valentin Amrhein, Sander Greenland, and Blake McShane. Scientists rise up against statistical significance. Nature, 2019. find
  3. Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507–517. PMLR, 2020. find
  4. Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature News, 2016. find
  5. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  6. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. find
  7. Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 2020. find
  8. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. find
  9. Peter J Bickel, Friedrich Götze, and Willem R van Zwet. Resampling fewer than n observations: gains, losses, and remedies for losses. In Selected works of Willem van Zwet, pages 267–297. Springer, 2012. find
  10. Mauro Birattari and Marco Dorigo. How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs? Optimization letters, 2007. find
  11. Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference on Machine Learning, pages 725–734, 2019. find
  12. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3, 2021. find
  13. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/ jax. ↗
  14. Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018. — [arXiv:1812.06110](https://arxiv.org/abs/1812.06110) find
  15. Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, 2021. find
  16. Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. In International Conference on Learning Representations, 2020. find
  17. Kaleigh Clary, Emma Tosch, John Foley, and David Jensen. Let’s play again: Variability of deep reinforcement learning agents in atari environments. arXiv preprint arXiv:1904.06312, 2019. — [arXiv:1904.06312](https://arxiv.org/abs/1904.06312) find
  18. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020. find
  19. Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint arXiv:2009.04416, 2020. find
  20. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:1806.08295, 2018. find
  21. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker’s guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.06979, 2019. — [arXiv:1904.06979](https://arxiv.org/abs/1904.06979) find
  22. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018. find
  23. Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. find
  24. Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07002, 2021. find
  25. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. find
  26. Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201–213, 2002. find
  27. Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. find
  28. Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019. find
  29. Bradley Efron. Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7:1–26, 1979. find
  30. Bradley Efron. Better bootstrap confidence intervals. Journal of the American statistical Association, 1987. find
  31. Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503–556, 2005. find
  32. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning. PMLR, 2018. find
  33. Gerd Gigerenzer. Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 1(2):198–218, 2018. find
  34. Steven N Goodman, Daniele Fanelli, and John PA Ioannidis. What does research reproducibility mean? Science translational medicine, 8(341):341ps12–341ps12, 2016. find
  35. Sander Greenland, Stephen J Senn, Kenneth J Rothman, John B Carlin, Charles Poole, Steven N Goodman, and Douglas G Altman. Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology, 2016. find
  36. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870. PMLR, 2018. find
  37. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019. find
  38. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. find
  39. Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations, 2020. find
  40. Nathaniel E Helwig. Bootstrap Confidence Intervals, 01 2021. URL http://users.stat.umn.edu/ ~helwig/notes/npboot-notes.html\#bootstrap-confidence-intervals. ↗
  41. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. find
  42. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. find
  43. Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado van Hasselt. Muesli: Combining improvements in policy optimization. Proceedings of the 38th International Conference on Machine Learning, 2021. find
  44. John PA Ioannidis. Why most published research findings are false. PLoS medicine, 2(8):e124, 2005. find
  45. Alex Irpan. Deep reinforcement learning doesn’t work yet. https://www.alexirpan.com/2018/02/ 14/rl-hard.html, 2018. ↗
  46. Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133, 2017. — [arXiv:1708.04133](https://arxiv.org/abs/1708.04133) find
  47. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.
  48. Minqi Jiang, Ed Grefenstette, and Tim Rocktäschel. Prioritized level replay. International Conference on Machine Learning, 2021. find
  49. Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning, pages 4962–4973. PMLR, 2020. find
  50. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019. find
  51. Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.10181, 2020. — [arXiv:2003.10181](https://arxiv.org/abs/2003.10181) find
  52. Sergey Kolesnikov and Oleksii Hrinchuk. Catalyst. rl: a distributed framework for reproducible rl research. arXiv preprint arXiv:1903.00027, 2019. — [arXiv:1903.00027](https://arxiv.org/abs/1903.00027) find
  53. Ilya Kostrikov*, Denis Yarats*, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. find
  54. Piotr Kozakowski, Lukasz Kaiser, Henryk Michalewski, Afroz Mohiuddin, and Katarzyna Kańska. Qvalue weighted regression: Reinforcement learning with limited data. arXiv preprint arXiv:2102.06782, 2021. find
  55. Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. NeurIPS, 32:10724–10734, 2019. find
  56. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020. find
  57. Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 2020. find
  58. Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33, 2020. find
  59. Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. International Conference on Machine Learning, 2021. find
  60. Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 2020. find
  61. Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management science, 38(4): 555–593, 1992. find
  62. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. find
  63. Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Significant improvements over the state of the art? a case study of the ms marco document ranking leaderboard. arXiv preprint arXiv:2102.12887, 2021. — [arXiv:2102.12887](https://arxiv.org/abs/2102.12887) find
  64. Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Li Jian, Nenghai Yu, and Tie-Yan Liu. Return-based contrastive representation learning for reinforcement learning. In International Conference on Learning Representations, 2021. find
  65. Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551, 2021. — [arXiv:2103.04551](https://arxiv.org/abs/2103.04551) find
  66. Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning, 2021. find
  67. Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. arXiv preprint arXiv:1711.10337, 2017. find
  68. Nicolai A Lynnerup, Laura Nolling, Rasmus Hasle, and John Hallam. A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. In Conference on Robot Learning, 2020. find
  69. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
  70. Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018. — [arXiv:1803.07055](https://arxiv.org/abs/1803.07055) find
  71. Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60, 1947. find
  72. Muhammad Rizki Maulana and Wee Sun Lee. Ensemble and auxiliary tasks for data-efficient deep reinforcement learning. arXiv preprint arXiv:2107.01904, 2021. — [arXiv:2107.01904](https://arxiv.org/abs/2107.01904) find
  73. Blakeley B McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L Tackett. Abandon statistical significance. The American Statistician, 2019. find
  74. Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018. find
  75. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015. find
  76. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016. find
  77. Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic implementations for reproducibility in deep reinforcement learning. arXiv preprint arXiv:1809.05676, 2018. — [arXiv:1809.05676](https://arxiv.org/abs/1809.05676) find
  78. Harold Pashler and Eric-Jan Wagenmakers. Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on psychological science, 7(6):528–530, 2012. find
  79. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). arXiv preprint arXiv:2003.12206, 2020. find
  80. Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. International Conference on Machine Learning, 2021. find
  81. Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862, 2020. — [arXiv:2006.12862](https://arxiv.org/abs/2006.12862) find
  82. David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021. — [arXiv:2102.12425](https://arxiv.org/abs/2102.12425) find
  83. Ben Recht. Benchmarking Machine Learning with Performance Profiles, 03 2018. URL http://www.argmin.net/2018/03/26/performance-profiles/. ↗
  84. Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, 2017.
  85. Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, 2017. find
  86. Jan Robine, Tobias Uelwer, and Stefan Harmeling. Smaller world models for reinforcement learning. arXiv preprint arXiv:2010.05767, 2020. — [arXiv:2010.05767](https://arxiv.org/abs/2010.05767) find
  87. David Romer. In praise of confidence intervals. In AEA Papers and Proceedings, volume 110, pages 55–60, 2020. find
  88. Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. find
  89. Rohan Saphal, Balaraman Ravindran, Dheevatsa Mudigere, Sasikant Avancha, and Bharat Kaul. Seerl: Sample efficient ensemble reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1100–1108, 2021. find
  90. Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347, 2021. — [arXiv:2105.05347](https://arxiv.org/abs/2105.05347) find
  91. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020. find
  92. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  93. Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021. find
  94. Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al. The multiberts: Bert reproductions for robustness analysis. arXiv preprint arXiv:2106.16163, 2021. find
  95. Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, 2021. find
  96. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  97. Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020. — [arXiv:2010.09163](https://arxiv.org/abs/2010.09163) find
  98. Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136v2, 2020. — [arXiv:2004.04136](https://arxiv.org/abs/2004.04136) find
  99. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1): 9–44, 1988. find
  100. Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in neural information processing systems, 1996. find
  101. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2nd edition, 2018. find
  102. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999. find
  103. István Szita and András Lörincz. Learning tetris using the noisy cross-entropy method. Neural computation, 2006. find
  104. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. find
  105. Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhuman on atari? leveling the playing field. arXiv preprint arXiv:1908.04683, 2019. — [arXiv:1908.04683](https://arxiv.org/abs/1908.04683) find
  106. John W Tukey. A survey of sampling from contaminated distributions. Contributions to probability and statistics, pages 448–485, 1960. find
  107. Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? NeurIPS, 2019. find
  108. Gaël Varoquaux and Veronika Cheplygina. How i failed machine learning in medical imaging– shortcomings and recommendations. arXiv preprint arXiv:2103.10292, 2021. — [arXiv:2103.10292](https://arxiv.org/abs/2103.10292) find
  109. Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020. find
  110. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019. find
  111. Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. arXiv preprint arXiv:2010.10814, 2020. — [arXiv:2010.10814](https://arxiv.org/abs/2010.10814) find
  112. Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar. Moving to a world beyond “p \< 0.05”. The American Statistician, 2019. find
  113. Bernard L Welch. The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika, 34(1/2):28–35, 1947. find
  114. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019. — [arXiv:1910.01741](https://arxiv.org/abs/1910.01741) find
  115. Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. arXiv preprint arXiv:2010.07470, 2020. — [arXiv:2010.07470](https://arxiv.org/abs/2010.07470) find
  116. Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. arXiv preprint arXiv:2106.11872, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agarwal-2021-deep-reinforcement-learning-edge-statistical/