A General Theoretical Paradigm to Understand Learning from Human Preferences

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2310.12036

Summary

This paper by researchers at Google DeepMind provides a theoretical foundation for methods that train language models from human preferences. The authors show that RLHF and DPO rest on a shared assumption: a pairwise comparison of two responses can be replaced by a pointwise reward under the Bradley-Terry model. To get rid of this assumption, they propose a general objective, ΨPO. It depends only on pairwise preference probabilities and includes KL regularization that keeps the policy close to the reference policy. RLHF and DPO turn out to be special cases of ΨPO. The analysis shows that both methods can overfit: when the preferences in the data are deterministic or nearly deterministic, the regularization effectively stops working. As a solution, the authors take Ψ to be the identity mapping and obtain the IPO method. For it, they derive a simple empirical loss function that is optimized on samples of comparisons, and they prove guarantees. On small synthetic examples, IPO behaves stably where DPO produces degenerate policies. The paper explains why DPO can ignore the reference model and offers a simple alternative. However, IPO has not yet been tested on real language models.

Key points

ΨPO is a general objective with a non-decreasing function Ψ. If Ψ(q) = log(q/(1−q)) and the Bradley-Terry model holds, the optimal policies of ΨPO, RLHF and DPO coincide (Proposition 1).
IPO (Identity-PO): Ψ is the identity mapping. The problem is reduced to finding the root of an equation, and an empirical loss function is derived from this. It requires neither a reward model nor the Bradley-Terry assumption.
Experiment 1 has 3 actions and a dataset D₁ with a total order. DPO converges to a deterministic policy for any τ. IPO stays close to π_ref under strong regularization.
Experiment 2 uses the dataset D₃ = {(y_a, y_b), (y_b, y_a)}, where action y_c is never compared. DPO drives its probability to 0 regardless of τ. IPO lowers it gradually, depending on the regularization strength.
Experimental setup: softmax parameterization, Adam (lr 0.01), 18000 steps, mini-batch 9, 10 seeds, 95% confidence intervals, implementation in flax and optax. The authors leave evaluation on language models for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

236 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 230 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.12036 · arXiv 2310.12036

@misc{azar2023general,
  title = {{A General Theoretical Paradigm to Understand Learning from Human Preferences}},
  author = {Azar, Mohammad Gheshlaghi and Rowland, Mark and Piot, Bilal and Guo, Daniel and Calandriello, Daniele and Valko, Michal and Munos, Rémi},
  year = {2023},
  doi = {10.48550/arxiv.2310.12036},
  eprint = {2310.12036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.12036}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, et al. The DeepMind JAX ecosystem, 2020, 2020. URL http://github.com/deepmind. ↗
Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the Elo: Real-world games are transitive, not additive. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2023. find
Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. find
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference-based racing algorithm. Machine Learning, (3):327–351, 2014. find
Róbert Busa-Fekete, Balázs Szörenyi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based evolutionary direct policy search. In Autonomous Learning Workshop @ ICRA, 2013. find
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In Proceedings of the International Conference on Machine Learning, 2022. find
Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017. find
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the International Conference on Machine Learning, 2019. find
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http: //github.com/google/flax. find
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2014. find
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the Annual Meeting of Association for Computational Linguistics, 2020. find
Tyler Lu, Dávid Pál, and Martin Pál. Contextual multi-armed bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2010. find
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In Proceedings of the International Conference on Machine Learning, 2020. find
Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2020. find
OpenAI. Gpt-4 technical report, 2023. find our card
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller amd Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training… find
Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling RL: Reinforcement learning with trajectory preferences. arXiv, 2023. find
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. find
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv, 2023. find
Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised pretraining for sequence to sequence learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processings, 2016. find
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017. find
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020. find
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL? arXiv, 2023. find
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations, 2022. find
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv, 2023. find our card