Elamur Research⌕
☰
home / papers / 2024 / Secrets of RLHF in Large Language Models Part II: Reward Modeling

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Paper · 2024 · Binghai Wang, Rui Zheng, Lu Chen et al.

Summary

In RLHF, reward models stand in for human judgment, but they have two weaknesses. Preference labels are often wrong or unclear because annotators frequently disagree. Reward models also transfer poorly to out-of-distribution outputs, which makes iterative RLHF unreliable. This report from Fudan NLP Lab and Hikvision addresses both problems.

On the data side, the authors train an ensemble of reward models on the same comparisons. The average and spread of the models' score gaps serve as a measure of preference strength. This lets them separate likely mislabeled, ambiguous and normal pairs. They then flip incorrect labels, soften ambiguous ones and add an adaptive margin, scaled by strength, to the ranking loss.

On the algorithm side, a contrastive objective helps the model tell chosen and rejected answers apart. MetaRM, a meta-learning procedure, keeps the reward model sensitive to the policy's shifting outputs.

The authors report more stable PPO and steady gains over 3–4 iterative rounds on two datasets. This matters because a sturdier reward model underpins safer and more helpful alignment.

Key points

  • Prior annotation studies show weak human agreement: about 63% for Anthropic's crowd workers and 72.6 ± 1.5% among OpenAI labelers.
  • An ensemble of M = 10 reward models scored the Anthropic HH-RLHF training set:
    • About 25% of pairs had a negative mean preference difference, suggesting flipped labels.
    • Score variance traced a U-shape across strength groups.
    • Agreement with GPT-4 labels rose as preference strength grew.
  • Data-side fixes: label flipping, label smoothing and an adaptive margin in the Bradley-Terry loss.
  • Algorithm-side methods: contrastive learning and MetaRM. They enabled iterative RLHF with consistent gains over 3–4 rounds on HH-RLHF and OpenAI's summarization data.
  • Released resources (github.com/OpenLMLab/MOSS-RLHF): training code, HH-RLHF annotated with preference strength, and a GPT-4-cleaned validation set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🏢HikvisionHangzhou · China2 works · 165 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu and Yu-Gang Jiang. 2024. Secrets of RLHF in Large Language Models Part II: Reward Modeling. arXiv:2401.06080.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.06080 · arXiv 2401.06080

@misc{wang2024secrets,
  title = {{Secrets of RLHF in Large Language Models Part II: Reward Modeling}},
  author = {Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu and Yu-Gang Jiang},
  year = {2024},
  doi = {10.48550/arxiv.2401.06080},
  eprint = {2401.06080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.06080}
}

References

43 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leike, J., D. Krueger, T. Everitt, et al. Scalable agent alignment via reward modeling: a research direction, 2018.
  2. Kenton, Z., T. Everitt, L. Weidinger, et al. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021.
  3. Xi, Z., W. Chen, X. Guo, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023.
  4. Ouyang, L., J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. Challenge 1: Noise Incorrect Low training acc Ambiguous ... Poor PPO stability ... Reward Model Intention modeling , ) Enh…
  5. Bai, Y., A. Jones, K. Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  6. Bai, Y., S. Kadavath, S. Kundu, et al. Constitutional AI: Harmlessness from AI feedback, 2022.
  7. Kundu, S., Y. Bai, S. Kadavath, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023. find
  8. Lambert, N., T. Krendl Gilbert, T. Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv–2310, 2023. find
  9. Pitis, S. Failure modes of learning reward models for llms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning. 2023. find
  10. McKinney, L., Y. Duan, D. Krueger, et al. On the fragility of learned reward functions. arXiv preprint arXiv:2301.03652, 2023. — [arXiv:2301.03652](https://arxiv.org/abs/2301.03652) find
  11. Zheng, R., Z. Xi, Q. Liu, et al. Characterizing the impacts of instances on robustness. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2314–2332. 2023. find
  12. Stiennon, N., L. Ouyang, J. Wu, et al. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. find
  13. Ziegler, D. M., N. Stiennon, J. Wu, et al. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. find
  14. Glaese, A., N. McAleese, M. Trebacz, et al. Improving alignment of dialogue agents via targeted human judgements. CoRR, abs/2209.14375, 2022.
  15. Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. find
  16. Bradley, R. A., M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
  17. Jaques, N., A. Ghandeharioun, J. H. Shen, et al. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456, 2019. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) find
  18. Laidlaw, C., S. Singhal, A. Dragan. Preventing reward hacking with occupancy measure regularization. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems. 2023. find
  19. Reed, S., H. Lee, D. Anguelov, et al. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014. find
  20. Müller, R., S. Kornblith, G. E. Hinton. When does label smoothing help? In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, R. Garnett, eds., Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, D… find
  21. Touvron, H., L. Martin, K. Stone, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023.
  22. Zheng, R., S. Dou, S. Gao, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023.
  23. Caron, M., I. Misra, J. Mairal, et al. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020. find
  24. Gao, T., X. Yao, D. Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021.
  25. Bai, Y., A. Jones, K. Ndousse, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022. find
  26. Christiano, P. F., J. Leike, T. Brown, et al. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  27. Kaufmann, T., P. Weng, V. Bengs, et al. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023.
  28. Hong, J., K. Bhatia, A. Dragan. On the sensitivity of reward inference to misspecified human models. arXiv preprint arXiv:2212.04717, 2022. — [arXiv:2212.04717](https://arxiv.org/abs/2212.04717) find
  29. Knox, W. B., S. Hatgis-Kessell, S. Booth, et al. Models of human preference for learning reward functions. arXiv preprint arXiv:2206.02231, 2022. — [arXiv:2206.02231](https://arxiv.org/abs/2206.02231) find
  30. Casper, S., X. Davies, C. Shi, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. find
  31. Sharma, M., M. Tong, T. Korbak, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.
  32. Tamkin, A., A. Askell, L. Lovitt, et al. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689, 2023. find
  33. Ziegler, D. M., N. Stiennon, J. Wu, et al. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  34. Zheng, R., W. Shen, Y. Hua, et al. Improving generalization of alignment with human preferences through group invariant learning. arXiv preprint arXiv:2310.11971, 2023. — [arXiv:2310.11971](https://arxiv.org/abs/2310.11971) find
  35. Touvron, H., L. Martin, K. Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  36. Chiang, W.-L., Z. Li, Z. Lin, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. find
  37. Völske, M., M. Potthast, S. Syed, et al. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63. 2017. — TL;DR: Mining Reddit to Learn Automatic Summarization
  38. Köpf, A., Y. Kilcher, D. von Rütte, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
  39. Holtzman, A., J. Buys, L. Du, et al. The curious case of neural text degeneration, 2020.
  40. Schulman, J., P. Moritz, S. Levine, et al. High-dimensional continuous control using generalized advantage estimation, 2018. find
  41. Chang, Y., X. Wang, J. Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109, 2023.
  42. Zheng, L., W.-L. Chiang, Y. Sheng, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  43. Xi, Z., S. Jin, Y. Zhou, et al. Self-polish: Enhance reasoning in large language models via problem refinement. arXiv preprint arXiv:2305.14497, 2023. train eval 0.9 0.8 Accuracy 0.7 0.6 0.5 4000 6000 Steps Figure 19: Left: training trajectory of the reward model. Right: reward scores of chosen and… — [arXiv:2305.14497](https://arxiv.org/abs/2305.14497) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/