Elamur Research⌕
☰
home / papers / 2024 / West-of-N: Synthetic Preferences for Self-Improving Reward Models

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Paper · 2024 · Alizee Pace, Jonathan Mallinson, Eric Malmi et al.

Summary

Reward models drive RLHF, but improving them usually requires more human preference labels, which are slow, costly and noisy to collect. Pace et al. propose West-of-N, a self-training scheme that produces synthetic preference pairs without new annotation. First, a base preference model is trained on an existing preference dataset, which may be human-labeled or synthetic. Next, for each unlabeled prompt, the policy language model generates N candidate responses. The base model ranks them, and the top- and bottom-ranked candidates become a pseudo-labeled pair. These pairs are combined with the original data to train a student reward model. Because the responses are sampled from the policy itself, the new data matches the distribution the reward model will see during RL. Choosing the extremes of the pool also makes the labels more reliable. In experiments, including on Reddit TL;DR, the method improved every reward model tested, with gains similar to adding a comparable volume of human-labeled data. This offers a cheap, scalable way to strengthen reward models and, through them, downstream alignment.

Key points

  • The name combines Best-of-N and Worst-of-N. The authors present it as the first application of Best-of-N sampling and semi-supervised learning to reward model training, as opposed to policy training.
  • Pseudo-label accuracy grows with N and exceeds that of high-confidence human feedback. Up to N = 64, no reward hacking of the base model was observed. West-of-N responses remain orders of magnitude more likely under the policy than the off-policy human-dataset responses.
  • Keeping only pairs to which the base model assigns high preference probability brings further gains in accuracy and Best-of-N performance. Filtering out low-likelihood pairs mainly improves Best-of-N quality. This contrasts with Stiennon et al. (2020), who found that confidence filtering hurt reward models trained on human data.
  • In iterative West-of-N on Reddit TL;DR, each round uses the previous reward model as the base, with human and synthetic pairs mixed 1:1. Accuracy gains on held-out human data shrink, but Best-of-N win rates, judged by an Autorater and by GPT-4, keep improving. Per-metric gains in this experiment range from about +0.5% to +11.0%.
  • Compared with human feedback, RLAIF and RLCD, West-of-N is the only approach in the authors' comparison that is simultaneously on-policy, low-noise and scalable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Alizee Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause and Aliaksei Severyn. 2024. West-of-N: Synthetic Preferences for Self-Improving Reward Models. arXiv:2401.12086.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.12086 · arXiv 2401.12086

@misc{pace2024west,
  title = {{West-of-N: Synthetic Preferences for Self-Improving Reward Models}},
  author = {Alizee Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause and Aliaksei Severyn},
  year = {2024},
  doi = {10.48550/arxiv.2401.12086},
  eprint = {2401.12086},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.12086}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amini, M.-R., Feofanov, V., Pauletto, L., Devijver, E., and Maximov, Y. Self-training: A survey. arXiv preprint arXiv:2202.12040, 2022. — [arXiv:2202.12040](https://arxiv.org/abs/2202.12040) find
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. find
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.
  4. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
  5. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023. find
  6. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  7. Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
  8. Google. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  9. Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023.
  10. He, J., Gu, J., Shen, J., and Ranzato, M. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations, 2019. find
  11. Huang, T., Chu, J., and Wei, F. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. — [arXiv:2204.03649](https://arxiv.org/abs/2204.03649) find
  12. Kim, S., Bae, S., Shin, J., Kang, S., Kwak, D., Yoo, K. M., and Seo, M. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023. find
  13. Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. find
  14. Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023.
  15. Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. find
  16. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  17. OpenAI. Gpt-4 technical report, 2023.
  18. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  19. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  20. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  21. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  22. Scudder, H. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371, 1965. doi: 10.1109/TIT.1965.1053799. — [doi:10.1109/tit.1965.1053799](https://doi.org/10.1109/tit.1965.1053799) find
  23. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235, 2018. URL http://arxiv.org/abs/1804.04235. ↗
  24. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  25. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. find
  26. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  27. Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of lm alignment, 2023. find
  28. Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. find
  29. Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference optimization with the pairwise cringe loss, 2023. find
  30. Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. Rlcd: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.12950, 2023. find
  31. Yarowsky, D. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189–196, 1995.
  32. Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models, 2024. find
  33. Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
  34. Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
  35. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  36. Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. Rethinking pre-training and self-training. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3833–3845. Curran Associates, Inc., 2… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pace-2024-west-n-synthetic-preferences-self/