Elamur Research⌕
☰
home / papers / 2024 / Transcendence: Generative Models Can Outperform The Experts That Train Them

Transcendence: Generative Models Can Outperform The Experts That Train Them

Paper · 2024 · Edwin Zhang, Vincent Zhu, Naomi Saphra et al.

Summary

Models trained with cross-entropy learn to mimic the distribution of their training data. This suggests they cannot outperform the humans who produced that data. The authors examine this assumption and define "transcendence" as a generative model earning higher reward than every expert behind its training set.

They first prove that, in an idealized setting, simply reproducing the averaged mixture of experts never beats the strongest expert. Transcendence becomes possible with low-temperature sampling, which acts like a majority vote across experts and filters out their individual, diverse mistakes ("wisdom of the crowd"). This happens exactly when the arg-max predictor is better than the best expert.

To test the theory, they trained ChessFormer, an autoregressive decoder-only transformer, on human chess transcripts with an upper limit on player rating. At low temperature, some models played above the strongest rating they saw in training. The paper offers a formal explanation for how imitation learning can exceed its teachers, while stressing that the effect comes from removing errors, not from new reasoning.

Key points

  • The paper formalizes transcendence and gives three main results:
    • Plain imitation without temperature changes cannot achieve it (Proposition 1).
    • Low-temperature sampling achieves it if and only if the arg-max predictor beats every expert (Proposition 2).
    • A single expert who errs uniformly at random can be "denoised" this way (Proposition 3).
  • ChessFormer 1000 and ChessFormer 1300 surpassed the top Glicko-2 rating in their training data. These ratings are reported with 95% confidence intervals.
  • Most of the gain at low temperature comes from large improvements on a small share of game states, likely the decisive ones.
  • Transcendence requires diverse training data; models trained on less varied datasets did not exceed their experts.
  • Supporting experiments include natural-language question answering and a toy linear model. The toy model was trained on 10K examples labeled by Gaussian-noisy experts. The work appeared at NeurIPS 2024, and code, data and models are at transcendence.eddie.win.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade and Eran Malach. 2024. Transcendence: Generative Models Can Outperform The Experts That Train Them. arXiv:2406.11741.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.11741 · arXiv 2406.11741

@misc{zhang2024transcendence,
  title = {{Transcendence: Generative Models Can Outperform The Experts That Train Them}},
  author = {Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade and Eran Malach},
  year = {2024},
  doi = {10.48550/arxiv.2406.11741},
  eprint = {2406.11741},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.11741}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Breiman. https://api.semanticscholar.org/CorpusID:47328136. URL ↗
  2. C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
  3. M. Campbell, A. J. Hoane, and F.-h. Hsu. Deep Blue. Artificial Intelligence, 134(1):57–83, Jan. 2002. ISSN 0004-3702. doi: 10.1016/S0004-3702(01)00129-1. — [doi:10.1016/s0004-3702(01)00129-1](https://doi.org/10.1016/s0004-3702%2801%2900129-1) find
  4. L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021. find
  5. X. Feng, Y. Luo, Z. Wang, H. Tang, M. Yang, K. Shao, D. Mguni, Y. Du, and J. Wang. ChessGPT: Bridging Policy Learning and Language Modeling. Advances in Neural Information Processing Systems, 36:7216–7262, Dec. 2023.
  6. Y. Freund and R. E. Schapire. A short introduction to boosting, 1999. https://api.semanticscholar.org/CorpusID:9621074. URL ↗
  7. M. E. Glickman. Example of the glicko-2 system. Boston University, 28, 2012. find
  8. M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem, 2021. find
  9. A. Jiang, L. Soriano Marcolino, A. D. Procaccia, T. Sandholm, N. Shah, and M. Tambe. Diverse randomized agents vote to win. Advances in Neural Information Processing Systems, 27, 2014. find
  10. S. M. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. URL https://api.semanticscholar.org/CorpusID:31442909. ↗
  11. A. Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT, 2022. ↗
  12. A. Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint arXiv:2403.15498, 2024. — [arXiv:2403.15498](https://arxiv.org/abs/2403.15498) find
  13. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  14. S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. find
  15. J. Li, E. Zhang, M. Yin, Q. Bai, Y.-X. Wang, and W. Y. Wang. Offline reinforcement learning with closed-form policy improvement operators. In International Conference on Machine Learning, pages 20485–20528. PMLR, 2023. find
  16. T. Lin, L. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020. find
  17. L. S. Marcolino, A. X. Jiang, and M. Tambe. Multi-agent team formation: Diversity beats strength? In IJCAI, volume 13, 2013. find
  18. R. McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson. Aligning Superhuman AI with Human Behavior: Chess as a Model System. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1677–1687, Aug. 2020. doi: 10.1145/3394486.3403219. — [doi:10.1145/3394486.3403219](https://doi.org/10.1145/3394486.3403219) find
  19. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023. find
  20. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. find
  21. R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016. find
  22. Y. Nasu. Efficiently Updatable Neural-Network-based Evaluation Functions for Computer Shogi, 2018. find
  23. D. Noever, M. Ciolino, and J. Kalin. The Chess Transformer: Mastering Play using Generative Language Models, Sept. 2020. find
  24. A. Odonnat, V. Feofanov, and I. Redko. Leveraging ensemble diversity for robust self-training in the presence of sample selection bias, 2024. find
  25. Pete. AlphaZero Crushes Stockfish In New 1,000-Game Match. https://www.chess.com/news/view/updated-alphazero-crushes-stockfish-in-new-1-000-game-match, Dec. 2018. ↗
  26. C. E. Shannon. XXII. Programming a computer for playing chess. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 41(314):256–275, Mar. 1950. ISSN 1941-5982, 1941-5990. doi: 10.1080/14786445008521796. — [doi:10.1080/14786445008521796](https://doi.org/10.1080/14786445008521796) find
  27. D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Dec. 2017.
  28. L. Soriano Marcolino, H. Xu, A. Xin Jiang, M. Tambe, and E. Bowring. Give a Hard Problem to a Diverse Team: Exploring Large Action Spaces. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), June 2014. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v28i1.8880. — [doi:10.1609/aaai.v28i1.8880](https://doi.org/10.1609/aaai.v28i1.8880) find
  29. The Stockfish developers (see AUTHORS file). Stockfish, 2024. URL https://stockfishchess.org/. ↗
  30. S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel. Chess as a Testbed for Language Model State Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385–11393, June 2022. doi: 10.1609/aaai.v36i10.21390. — [doi:10.1609/aaai.v36i10.21390](https://doi.org/10.1609/aaai.v36i10.21390) find
  31. A. Touati, A. Zhang, J. Pineau, and P. Vincent. Stable policy optimization via off-policy divergence regularization. In Conference on Uncertainty in Artificial Intelligence, pages 1328–1337. PMLR, 2020. find
  32. A. Turing. Chess (1953). In B. J. Copeland, editor, The Essential Turing, page 0. Oxford University Press, Sept. 2004. ISBN 978-0-19-825079-1. doi: 10.1093/oso/9780198250791.003.0023. — [doi:10.1093/oso/9780198250791.003.0023](https://doi.org/10.1093/oso/9780198250791.003.0023) find
  33. A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470, 2021. find
  34. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. find
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  36. Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698, 2020. find
  37. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2024-transcendence-generative-models-outperform-experts/