Elamur Research⌕
☰
home / papers / 2018 / Supervising strong learners by amplifying weak experts

Supervising strong learners by amplifying weak experts

Paper · 2018 · Paul Christiano, Buck Shlegeris and Dario Amodei

Summary

Many valuable tasks, such as shaping economic policy or securing large computer networks, are too complex for a person to demonstrate or reliably judge. Optimizing an easy-to-measure proxy instead can produce harmful behavior. The authors propose Iterated Amplification to address this. A human expert breaks a question into subquestions and hands them to copies of the current model. The expert then merges the returned subanswers into a final answer, and the model is trained with supervised learning to reproduce what this human-plus-copies team outputs. As the model improves, the team becomes stronger, so the training target keeps rising. No external reward function is needed, because the objective is carried implicitly by how the expert decomposes questions. The method resembles Expert Iteration but works without a reward. In algorithmic environments, a hand-coded decomposer replaced the human, and a learned predictor stood in for that expert. The system learned complex behaviors as well as direct supervised learning, but needed far fewer expert-provided examples. This suggests a possible path toward training signals for tasks beyond human scale.

Key points

  • Architecture: The model X is an encoder-decoder network with self-attention. It is split into a context-encoding phase and a question-answering phase.
  • Context reuse: Encoded context is shared across many questions and subquestions, which sped up training by roughly an order of magnitude.
  • Sample efficiency: Training with direct supervision depended on tens of millions of examples. Iterated Amplification needed only tens of thousands of decomposition examples (per-task figures are in Table 2, Appendix A).
  • Compute cost: Producing one (Q, A) pair calls X and the human predictor H′ between 3 and 10 times. Each pair is trained on about 10 times, so total computation per question roughly doubles.
  • Main simplifications: Decompositions came from an algorithm rather than a real human, and the tasks were combinatorial. Training used supervised learning instead of reward learning plus RL. Application to messy real-world tasks is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

Cite

Cite as Paul Christiano, Buck Shlegeris and Dario Amodei. 2018. Supervising strong learners by amplifying weak experts. arXiv:1810.08575.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.08575 · arXiv 1810.08575

@misc{christiano2018supervising,
  title = {{Supervising strong learners by amplifying weak experts}},
  author = {Paul Christiano, Buck Shlegeris and Dario Amodei},
  year = {2018},
  doi = {10.48550/arxiv.1810.08575},
  eprint = {1810.08575},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.08575}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004. find
  2. Dario Amodei and Jack Clark. Faulty reward functions in the wild, 2016. find
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.
  4. Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, pages 5366–5376, 2017.
  5. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. find
  6. Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. CoRR, abs/1704.06611, 2017. URL http://arxiv.org/abs/1704.06611. ↗
  7. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4302–4310, 2017. find
  8. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, volume 48, 2016. find
  9. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016. find
  10. Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in neural information processing systems, pages 3909–3917, 2016. find
  11. Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018.
  12. Łukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. arXiv preprint arXiv:1511.08228, 2015. find
  13. W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In International Conference on Knowledge Capture, pages 9–16, 2009. find
  14. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Belson, David M Bryson, Nick Cheney, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research com… — [arXiv:1803.03453](https://arxiv.org/abs/1803.03453) find
  15. James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. arXiv preprint arXiv:1701.06049, 2017. — [arXiv:1701.06049](https://arxiv.org/abs/1701.06049) find
  16. Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.04834, 2015. find
  17. Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine learning, pages 663–670, 2000. find
  18. Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pages 663–670, 2000. find
  19. Alex Nowak and Joan Bruna. Divide and conquer with neural networks. CoRR, abs/1611.02401, 2016. URL http://arxiv.org/abs/1611.02401. — [arXiv:1611.02401](https://arxiv.org/abs/1611.02401) ↗
  20. Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In International Conference on Rehabilitation Robotics, pages 1–7, 2011. find
  21. D. Silver, et al ..., and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv, December 2017a.
  22. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017b. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  24. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/christiano-2018-supervising-strong-learners-amplifying-weak/