Elamur Research
☰
home / papers / 2023 / Preference Ranking Optimization for Human Alignment

Preference Ranking Optimization for Human Alignment

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2306.17492

Summary

Large language models are usually aligned with human values through RLHF, which is complex, unstable and sensitive to hyperparameters. Most supervised alternatives contrast responses only in pairs, even when longer human rankings are available. The authors propose Preference Ranking Optimization (PRO), a purely supervised fine-tuning objective in which the LLM serves as its own reward model, scoring each candidate by its length-normalised log-likelihood. Given n ranked responses to a prompt, PRO marks the top candidate as positive and every lower-ranked one as negative. It then drops that candidate and repeats down the list. This recursive one-to-N contrast resembles the Plackett-Luce ranking model, and an extra NLL term on the best response keeps generations fluent. Rankings can be lengthened cheaply by adding responses from other LLMs (Alpaca, ChatGPT) ordered by a reward model, or through self-bootstrapping. On HH-RLHF, PRO outperforms baselines, and longer, more varied rankings bring the largest gains, so a 7B model approaches ChatGPT's reward scores. The method offers a simpler and more stable route to alignment from ranked preference data.

Key points

  • Objective: L_PRO combines the ranking loss with β·L_SFT, where β balances preference fit against text quality. With n = 2, PRO reduces to a pairwise Bradley-Terry contrast.
  • Data: Experiments use HH-RLHF. Some ranking variants are extended to length 3 with candidates from Alpaca or ChatGPT, which are ranked by a reward model.
  • Evaluation: The authors evaluate with automatic metrics, reward-model scoring, GPT-4 judgments and human evaluation. They report results comparable to ChatGPT and to human-written responses using only 7B parameters.
  • Findings: Gains grow as rankings get longer. Heterogeneous candidates help more than homogeneous ones, and candidate quality matters.
  • Ablation: The study covers the ranking loss, L_SFT and a dynamic temperature T, which penalises near-positive negatives less. Removing both L_SFT and T causes the sharpest performance collapse. The paper appeared at AAAI 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.17492 · arXiv 2306.17492

@misc{song2023preference,
  title = {{Preference Ranking Optimization for Human Alignment}},
  author = {Song, Feifan and Yu, Bowen and Li, Minghao and Yu, Haiyang and Huang, Fei and Li, Yongbin and Wang, Houfeng},
  year = {2023},
  doi = {10.48550/arxiv.2306.17492},
  eprint = {2306.17492},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.17492}
}

References

44 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. find
  2. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022b. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. find our card
  3. Bradley, R. A.; and Terry, M. E. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4): 324–345. find
  4. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877– 1901. find
  5. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712. find our card
  6. Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022. Palm: Scaling language modeling with pathways. arXiv:2204.02311. find
  7. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep Reinforcement Learning from Human Preferences. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volum… find
  8. Curran Associates, Inc. Dong, H.; Xiong, W.; Goyal, D.; Pan, R.; Diao, S.; Zhang, J.; Shum, K.; and Zhang, T. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767. find our card
  9. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 320–335. Dublin, Ireland: Associ… find our card
  10. Gao, L.; Schulman, J.; and Hilton, J. 2022. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760. find
  11. Gugger, S.; Debut, L.; Wolf, T.; Schmid, P.; Mueller, Z.; and Mangrulkar, S. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. find
  12. He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  13. Lee, K.; Smith, L.; and Abbeel, P. 2021. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv:2106.05091. — [arXiv:2106.05091](https://arxiv.org/abs/2106.05091) find
  14. Lei, W.; Zhang, Y.; Song, F.; Liang, H.; Mao, J.; Lv, J.; Yang, Z.; and Chua, T.-S. 2022. Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22… find
  15. Li, M.; Song, F.; Yu, B.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023. Api-bank: A benchmark for tool-augmented llms. arXiv:2304.08244. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) find
  16. Liu, H.; Sferrazza, C.; and Abbeel, P. 2023. Chain of Hindsight aligns Language Models with Feedback. arXiv:2302.02676. find
  17. Luce, R. D. 2012. Individual choice behavior: A theoretical analysis. Courier Corporation. find
  18. MacGlashan, J.; Ho, M. K.; Loftin, R.; Peng, B.; Wang, G.; Roberts, D. L.; Taylor, M. E.; and Littman, M. L. 2017. Interactive Learning from Policy-Dependent Human Feedback. In Precup, D.; and Teh, Y. W., eds., Proceedings of the 34th International Conference on Machine Learning, volume 70 of Procee… find
  19. Nakano, R.; Hilton, J.; Balaji, S.; Wu, J.; Ouyang, L.; Kim, C.; Hesse, C.; Jain, S.; Kosaraju, V.; Saunders, W.; et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv:2112.09332. find our card
  20. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. find our card
  21. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744. find
  22. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics. find our card
  23. Peng, B.; Li, C.; He, P.; Galley, M.; and Gao, J. 2023. Instruction tuning with gpt-4. arXiv:2304.03277. find our card
  24. Plackett, R. L. 1975. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2): 193–202. find
  25. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. find
  26. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. find
  27. Snell, C.; Kostrikov, I.; Su, Y.; Yang, M.; and Levine, S. 2022. Offline rl for natural language generation with implicit language q learning. arXiv:2206.11871. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) find
  28. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. F. 2020. Learning to summarize with human feedback. find
  29. Advances in Neural Information Processing Systems, 33: 3008–3021. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. find
  30. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. find
  31. Wang, P.; Li, L.; Chen, L.; Zhu, D.; Lin, B.; Cao, Y.; Liu, Q.; Liu, T.; and Sui, Z. 2023. Large language models are not fair evaluators. arXiv:2305.17926. find
  32. Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv:2212.10560. find
  33. Warnell, G.; Waytowich, N.; Lawhern, V.; and Stone, P. 2018. Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  34. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Le Scao, T.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. 2020. Transformers: State-of-the-Art Natur… find our card
  35. Wu, Z.; Hu, Y.; Shi, W.; Dziri, N.; Suhr, A.; Ammanabrolu, P.; Smith, N. A.; Ostendorf, M.; and Hajishirzi, H. 2023. find
  36. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. arXiv:2306.01693. find our card
  37. Xue, W.; An, B.; Yan, S.; and Xu, Z. 2023. Reinforcement Learning from Diverse Human Preferences. arXiv:2301.11774. — [arXiv:2301.11774](https://arxiv.org/abs/2301.11774) find
  38. Yuan, Z.; Yuan, H.; Tan, C.; Wang, W.; Huang, S.; and Huang, F. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv:2304.05302. find our card
  39. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. find
  40. Zhou, C.; Liu, P.; Xu, P.; Iyer, S.; Sun, J.; Mao, Y.; Ma, X.; Efrat, A.; Yu, P.; Yu, L.; et al. 2023. Lima: Less is more for alignment. arXiv:2305.11206. find
  41. Zhu, B.; Jiao, J.; and Jordan, M. I. 2023. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. arXiv:2301.11270. — [arXiv:2301.11270](https://arxiv.org/abs/2301.11270) find
  42. Zhu, B.; Sharma, H.; Frujeri, F. V.; Dong, S.; Zhu, C.; Jordan, M. I.; and Jiao, J. 2023. Fine-Tuning Language Models with Advantage-Induced Policy Alignment. arXiv:2306.02231. — [arXiv:2306.02231](https://arxiv.org/abs/2306.02231) find
  43. Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; and Irving, G. 2019. find
  44. Fine-tuning language models from human preferences. arXiv:1909.08593. Qualitative Analysis of PRO find