Elamur Research
☰
home / papers / 2024 / Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2410.18451

Summary

Reward models turn human preferences into a training signal for LLM alignment. Their quality depends heavily on the preference data, and open-source preference datasets tend to be noisy or inconsistently labelled. The authors take a data-first approach. Using only public sources (HelpSteer2, OffsetBias, WildGuardMix and several Magpie subsets), they reduce a pool of about 378K preference pairs to a compact set of about 80K through selection and filtering. They then train discriminative reward models on Gemma-2-27B and Llama-3.1-8B and compare several loss functions. The plain Bradley-Terry objective works best. As of October 2024, Skywork-Reward-Gemma-2-27B ranked first on the RewardBench leaderboard. The team also found that some training prompts overlapped with RewardBench prompts. They released a cleaned v0.2 of the dataset, and retraining on it raised scores slightly. The work shows that a small, carefully chosen dataset can beat much larger collections. The released data has already been used to build other leading reward models.

Key points

  • Data: Skywork Reward Preference 80K is built from public datasets only: HelpSteer2, OffsetBias, WildGuardMix, Magpie Ultra, Magpie Pro and Magpie Air. It is far smaller than earlier mixtures such as Preference 700K or the roughly 1.5M public pairs used for Llama 2.
  • Models: Skywork-Reward-Gemma-2-27B and Skywork-Reward-Llama-3.1-8B held 1st and 7th place on RewardBench as of October 2024. Both models and the datasets are on Hugging Face.
  • Loss ablation: In the authors' tests, the vanilla Bradley-Terry loss beat alternatives that optimise the margin between chosen and rejected responses.
  • Contamination: The RewardBench n-gram decontamination script (n from 7 to 13) found 5,402 contaminated prompts in v0.1, mainly from Magpie Ultra, and 445 in v0.2. By comparison, it found 15,349 in Preference 700K and 2,394 in Nectar.
  • Decontaminated retraining: Training on about 77K pairs raised the average RewardBench score from 92.5 to 93.1 for the 8B model and from 93.8 to 94.3 for the 27B model. Safety and Reasoning improved for both models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

103 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.18451 · arXiv 2410.18451

@misc{liu2024skywork,
  title = {{Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs}},
  author = {Liu, Chris Yuhao and Zeng, Liang and Liu, Jiacai and Yan, Rui and He, Jujie and Wang, Chaojie and Yan, Shuicheng and Liu, Yang and Zhou, Yahui},
  year = {2024},
  doi = {10.48550/arxiv.2410.18451},
  eprint = {2410.18451},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.18451}
}

References

54 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. find our card
  2. B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024. find
  3. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  4. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834, 2024. find
  5. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
  6. Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. find our card
  7. C. R. Carvalho, A. D. Polson, and J. G. Scott. The dangers of inference using the bradley-terry model. The Annals of Statistics, 38(3):1491–1514, 2010. find
  8. S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. find
  9. G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. find
  10. L. Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(coming soon), 2023. URL https://huggingface.co/datasets/LDJnr/Capybara . ↗
  11. H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024. find
  12. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. find
  13. K. Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pages 5988–6008. PMLR, 2022. find
  14. J. H. Friedman, T. Hastie, and R. Tibshirani. The Elements of Statistical Learning. Springer, 2001. find
  15. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. find
  16. I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT press, 2016. find
  17. S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495, 2024. find our card
  18. H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023. find our card
  19. J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024. find our card
  20. D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023. find our card
  21. L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghallah, X. Lu, M. Sap, Y. Choi, et al. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. arXiv preprint arXiv:2406.18510, 2024. find
  22. P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661–18673, 2020. find
  23. N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. find
  24. W. Lian, B. Goodson, E. Pentland, A. Cook, C. Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orc a/OpenOrca , 2023. find
  25. T. Lin. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. find our card
  26. X. Lou, D. Yan, W. Shen, Y. Yan, J. Xie, and J. Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024. — [arXiv:2410.00847](https://arxiv.org/abs/2410.00847) find
  27. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. find
  28. J. Park, S. Jwa, M. Ren, D. Kim, and S. Choi. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.06551, 2024. find
  29. R. Rafailov, J. Hejna, R. Park, and C. Finn. From 𝑟 to 𝑞 ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024a. find
  30. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b. find
  31. RyokoAI. ShareGPT52K Dataset. https://huggingface.co/datasets/RyokoAI/ShareG PT52K , 2023. ↗
  32. M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 555–562, 2010. find
  33. B. Schölkopf, A. J. Smola, K. R. Müller, P. J. Bartlett, W. S. Davidson, D. C. P. J. M. Williamson, and R. C. R. Schölkopf. Kernel methods for pattern analysis. In Proceedings of the IEEE, volume 12, pages 406–417. IEEE, 2001. find
  34. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  35. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab /stanford\_alpaca , 2023. ↗
  36. G. Team. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301. URL https://www.kaggle.com /m/3301 . — [doi:10.34740/kaggle/m/3301](https://doi.org/10.34740/kaggle/m/3301) ↗
  37. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. find our card
  38. G. Team, M. Reid, N. Savinov, D. Teplyashin, L. Dmitry, T. Lillicrap, J. Alayrac, R. Soricut, A. Lazaridou, O. Firat, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. in arxiv [cs. cl]. arxiv, 2024a. find our card
  39. G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024b. find
  40. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  41. H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024a. find
  42. H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024b. find our card
  43. P. Wang, A. Xu, Y. Zhou, C. Xiong, and S. Joty. Direct judgement preference optimization. arXiv preprint arXiv:2409.14664, 2024c. find
  44. T. Wang, I. Kulikov, O. Golovneva, P. Yu, W. Yuan, J. Dwivedi-Yu, R. Y. Pang, M. Fazel-Zarandi, J. Weston, and X. Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 2024d. find
  45. Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023. find
  46. Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024e. find
  47. G. I. Winata, D. Anugraha, L. Susanto, G. Kuwanto, and D. T. Wijaya. Metametrics: Calibrating metrics for generation tasks using human preferences. arXiv preprint arXiv:2410.02381, 2024. find
  48. Z. Xu, F. Jiang, L. Niu, Y. Deng, R. Poovendran, Y. Choi, and B. Y. Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024. find our card
  49. R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216, 2024. — [arXiv:2406.10216](https://arxiv.org/abs/2406.10216) find
  50. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024. find
  51. Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023. find
  52. Y. Zhang, G. Zhang, Y. Wu, K. Xu, and Q. Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197, 2024. — [arXiv:2410.02197](https://arxiv.org/abs/2410.02197) find
  53. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. find
  54. B. Zhu, E. Frick, T. Wu, H. Zhu, and J. Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023. find