Elamur Research
☰
home / papers / 2023 / Secrets of RLHF in Large Language Models Part I: PPO

Secrets of RLHF in Large Language Models Part I: PPO

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2307.04964

Summary

RLHF is the main way to make LLMs helpful, honest and harmless. Its PPO stage, though, is unstable, sensitive to hyperparameters and costly to debug, and few open implementations existed. This report from the Fudan NLP Group and ByteDance breaks the RLHF pipeline into its stages: SFT, reward model training, and PPO, which runs four coordinated networks (policy, value, reward and reference). The authors test how each code-level and theory-level choice in PPO changes training behaviour. They find that reward and loss curves say little about whether a run is healthy. Perplexity, response length and KL divergence from the SFT model are better warning signs. Their experiments show that constraining the policy matters most for stability. They package the useful tricks, tuned so they do not interfere with each other, into PPO-max. Applied to 7B and 13B SFT models, PPO-max trains stably for longer. In human and GPT-4 judgments, its models beat their SFT baselines on helpfulness and harmlessness. The Chinese and English reward models and the code are released, which helps alignment work in languages other than English.

Key points

  • PPO-max combines policy constraints with other calibrated PPO tricks. The authors find that the reward model's quality sets the upper limit on how good the aligned policy can get.
  • On harmless prompts judged by GPT-4 against gpt-3.5-turbo-0613, the share of losses fell from 45% to 24% for the English model and from 37% to 29% for the Chinese model, compared with SFT. ChatGPT still comes out ahead overall.
  • On C-Eval (about 13k multiple-choice questions across 52 disciplines), plain PPO reduced Chinese language-understanding scores. The PPO-ptx variant, which mixes pre-training data into PPO, largely made up for this drop.
  • The resources released are Chinese and English reward models (the authors report they generalise well to other models) and the full PPO-max code at github.com/OpenLMLab/MOSS-RLHF.
  • Limitations the authors list: most experiments use a 7B model, the Chinese preference data is small and self-built, evaluation relies on human and GPT-4 judgments, and the reward score is a poor predictor of final RLHF quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.04964 · arXiv 2307.04964

@misc{zheng2023secrets,
  title = {{Secrets of RLHF in Large Language Models Part I: PPO}},
  author = {Zheng, Rui and Dou, Shihan and Gao, Songyang and Hua, Yuan and Shen, Wei and Wang, Binghai and Liu, Yan and Jin, Senjie and Liu, Qin and Zhou, Yuhao and Xiong, Limao and Chen, Lu and Xi, Zhiheng and Xu, Nuo and Lai, Wenbin and Zhu, Minghao and Chang, Cheng and Yin, Zhangyue and Weng, Rongxiang and Cheng, Wensen and Huang, Haoran and Sun, Tianxiang and Yan, Hang and Gui, Tao and Zhang, Qi and Qiu, Xipeng and Huang, Xuanjing},
  year = {2023},
  doi = {10.48550/arxiv.2307.04964},
  eprint = {2307.04964},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.04964}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Touvron, H., T. Lavril, G. Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. find
  2. Chiang, W.-L., Z. Li, Z. Lin, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023. find
  3. OpenAI. Gpt-4 technical report, 2023. find our card
  4. Zhao, W. X., K. Zhou, J. Li, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023. find our card
  5. Brown, T., B. Mann, N. Ryder, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  6. Peng, B., C. Li, P. He, et al. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. find our card
  7. Taori, R., I. Gulrajani, T. Zhang, et al. Stanford alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford\_alpaca, 2023. ↗
  8. Wei, J., X. Wang, D. Schuurmans, et al. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card
  9. Driess, D., F. Xia, M. S. Sajjadi, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. find
  10. Park, J. S., J. C. O’Brien, C. J. Cai, et al. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  11. Lucy, L., D. Bamman. Gender and representation bias in gpt-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55. 2021. find
  12. Thoppilan, R., D. De Freitas, J. Hall, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. find our card
  13. Bender, E. M., T. Gebru, A. McMillan-Major, et al. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. 2021. find
  14. Bommasani, R., D. A. Hudson, E. Adeli, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. find our card
  15. Altman, S. Planning for agi and beyond. https://openai.com/blog/planning-for-agi-and-beyond, 2022. ↗
  16. Ouyang, L., J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. find
  17. Bai, Y., A. Jones, K. Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  18. OpenLMLab. Open-Chinese-LLaMA: Chinese large language model base generated through incremental pre-training on chinese datasets. https://github.com/OpenLMLab/ OpenChineseLLaMA, 2023. ↗
  19. Chiang, W.-L., Z. Li, Z. Lin, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. find
  20. Ji, Y., Y. Deng, Y. Gong, et al. Belle: Be everyone’s large language model engine. https: //github.com/LianjiaTech/BELLE, 2023. find
  21. Beeching, E., Y. Belkada, K. Rasul, et al. StackLLaMA: An RL fine-tuned LLaMA model for stack exchange question and answering, 2023. find
  22. Christiano, P. F., J. Leike, T. Brown, et al. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett, eds., Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017. find
  23. MacGlashan, J., M. K. Ho, R. Loftin, et al. Interactive learning from policy-dependent human feedback. In D. Precup, Y. W. Teh, eds., Proceedings of the 34th International Conference on Machine Learning, vol. 70 of Proceedings of Machine Learning Research, pages 2285–2294. PMLR, 2017. find
  24. Ziegler, D. M., N. Stiennon, J. Wu, et al. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  25. Stiennon, N., L. Ouyang, J. Wu, et al. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  26. Bai, Y., S. Kadavath, S. Kundu, et al. Constitutional AI: Harmlessness from AI feedback, 2022. find our card
  27. Askell, A., Y. Bai, A. Chen, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find our card
  28. Andrychowicz, M., A. Raichuk, P. Stańczyk, et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations. 2021. find
  29. Engstrom, L., A. Ilyas, S. Santurkar, et al. Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020. find
  30. Holtzman, A., J. Buys, L. Du, et al. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. find
  31. Mnih, V., A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning. In M. Balcan, K. Q. Weinberger, eds., Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, vol. 48 of JMLR Workshop and Conference Proce… find
  32. Jaques, N., A. Ghandeharioun, J. H. Shen, et al. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456, 2019. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) find
  33. Schulman, J., S. Levine, P. Abbeel, et al. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. find
  34. Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms, 2017. find
  35. Huang, S., R. F. J. Dossa, A. Raffin, et al. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022. find
  36. Mnih, V., K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. find
  37. Qi Wang, J. J., Yiyuan Yang. Easy RL: Reinforcement Learning Tutorial. Posts and Telecom Press, Beijing, 2022. find
  38. Keskar, N., B. McCann, L. Varshney, et al. Ctrl: A conditional transformer language model for controllable generation. arXiv: Computation and Language, 2019. find our card
  39. Dubois, Y., X. Li, R. Taori, et al. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023. find our card
  40. Zheng, L., W.-L. Chiang, Y. Sheng, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. 2023. Human Prompt → How did life originate? (Helpful) SFT Response → find