Elamur Research⌕
☰
home / papers / 2024 / HelpSteer2-Preference: Complementing Ratings with Preferences

HelpSteer2-Preference: Complementing Ratings with Preferences

Paper · 2024 · Zhilin Wang, Alexander Bukharin, Olivier Delalleau et al.

Summary

Reward models for RLHF are usually trained in one of two ways: Bradley-Terry models learn from pairwise preferences, and regression models predict per-response ratings. No public dataset allowed a fair comparison, because each style needs labels collected in its own format. The authors fill this gap by releasing preference annotations gathered alongside the existing Likert-5 ratings of HelpSteer2, so both label types cover identical prompts and responses. Several Scale AI annotators judged each pair, recording which response wins, by how much, and a written justification. With this matched data, the team compares Bradley-Terry and Regression reward models directly. They then propose a combined, two-stage recipe in which a regression-trained model is further tuned with a Bradley-Terry objective. A Llama-3.1-70B-Instruct reward model built this way reached 94.1 on RewardBench. A policy aligned with it via REINFORCE scored 85.0 on Arena Hard. Both were leading results as of 1 Oct 2024. The openly licensed data and models give practitioners evidence for choosing and mixing reward-modeling styles.

Key points

  • HelpSteer2-Preference (CC-BY-4.0) contains 7,118 pairs: 6,766 for training and 352 for validation. Annotators used a forced-choice scale from −3 to +3 with no tie option, plus a "neither valid" choice. The authors call it the first open general-domain preference set with human-written justifications.
  • Cleaning raised annotator agreement. For each task, the three closest annotations were averaged. Tasks with a spread above 2 (about 10%) and pairs with a zero overall preference (22%) were dropped. These steps lifted quadratic-weighted Cohen's κ from 0.489 to 0.843 and then to 0.878, above HelpSteer2 helpfulness (0.791).
  • Position bias is mild. Response 2 won 41.6% of tasks versus 36.5% for Response 1. Preferences correlate strongly with HelpSteer2 helpfulness differences (Pearson R = 0.9024).
  • Released models: Llama-3.1-Nemotron-70B-Reward (94.1 RewardBench) and Llama-3.1-Nemotron-70B-Instruct (85.0 Arena Hard). MMLU, GSM8K and HumanEval stay roughly unchanged after REINFORCE training.
  • Ablations favour purpose-collected preferences. HelpSteer2 converted to pairs gave 92.9 on RewardBench, versus 93.7 with HelpSteer2-Preference. With DPO, Arena Hard reached 52.8 versus 47.7. The authors found 42 prompts overlapping with RewardBench; retraining without them still gave 93.3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev and Yi Dong. 2024. HelpSteer2-Preference: Complementing Ratings with Preferences. arXiv:2410.01257.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.01257 · arXiv 2410.01257

@misc{wang2024helpsteer2,
  title = {{HelpSteer2-Preference: Complementing Ratings with Preferences}},
  author = {Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev and Yi Dong},
  year = {2024},
  doi = {10.48550/arxiv.2410.01257},
  eprint = {2410.01257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.01257}
}

References

51 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. find
  2. AllenAI. Reward bench leaderboard. https://huggingface.co/spaces/allenai/reward-bench, 2024. ↗
  3. Duane F. Alwin and Jon A. Krosnick. The Measurement of Values in Surveys: A Comparison of Ratings and Rankings. Public Opinion Quarterly, 49(4):535–552, 01 1985. ISSN 0033-362X. doi: 10.1086/268949. URL https://doi.org/10.1086/268949. — [doi:10.1086/268949](https://doi.org/10.1086/268949) ↗
  4. Ron Artstein and Massimo Poesio. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596, 2008. doi: 10.1162/coli.07-034-R2. URL https://aclanthology.org/J08-4004. ↗
  5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
  7. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. find
  8. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF, 2024.
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — The Llama 3 Herd of Models
  10. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. find
  11. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of… find
  12. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019. URL https://openreview.net/forum?id=r1lgTGL5DE. ↗
  13. Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33: 18560–18572, 2020. find
  14. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Opena… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  15. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. find
  16. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. https://lmsys.org/blog/2024-04-19-arena-hard/, April 2024. ↗
  17. Chris Yuhao Liu and Liang Zeng. Skywork reward model. https://huggingface.co/Skywork/Skywork-Reward-Gemma-2-27B, September 2024. URL https://huggingface.co/Skywork/Skywork-Reward-Gemma-2-27B. ↗
  18. LMSys. Arena-hard-auto arena-hard-auto, 2024. leaderboard. find
  19. John A. McCarty and L. J. Shrum. The Measurement of Personal Values in Survey Research: A Test of Alternative Rating Procedures*. Public Opinion Quarterly, 64(3):271–298, 11 2000. ISSN 0033-362X. doi: 10.1086/317989. URL https://doi.org/10.1086/317989. — [doi:10.1086/317989](https://doi.org/10.1086/317989) ↗
  20. Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward, 2024. find
  21. Meta AI. Llama 3 model card. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md, 2024. ↗
  22. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted qu… — WebGPT: Browser-assisted question-answering with human feedback
  23. NLTK. nltk.tokenize.sent-tokenize. sent_tokenize.html, 2024. https://www.nltk.org/api/nltk.tokenize. ↗
  24. Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Sh… — Nemotron-4 340B Technical Report
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  26. Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024.
  27. Gloria Phillips-Wren, Daniel J. Power, and Manuel Mora. Cognitive bias, decision styles, and risk attitudes in decision making and dss. Journal of Decision Systems, 28(2):63–66, April 2019. ISSN 2116-7052. doi: 10.1080/12460125.2019.1646509. URL http://dx.doi.org/10.1080/12460125.2019.1646509. — [doi:10.1080/12460125.2019.1646509](https://doi.org/10.1080/12460125.2019.1646509) ↗
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  30. Scikit-Learn. Cohen kappa score. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html, 2024. https://github.com/lm-sys/ ↗
  31. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards… — Towards Understanding Sycophancy in Language Models
  32. Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison, 2024. URL https://arxiv.org/abs/2409.09603. — [arXiv:2409.09603](https://arxiv.org/abs/2409.09603) ↗
  33. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. find
  34. Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018. find
  35. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  36. Tatsu-Lab. Alpacaeval leaderboard. https://tatsu-lab.github.io/alpaca_eval/, 2023. ↗
  37. Tenyx. Tenyxchat: Language model alignment using tenyx fine-tuning, 2024. find
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  39. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023. find
  40. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards, 2024a. URL https://arxiv.org/abs/2402.18571. ↗
  41. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts, 2024b. URL https://arxiv.org/abs/2406.12845. ↗
  42. Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement preference optimization, 2024c. URL https://arxiv.org/abs/2409.14664. ↗
  43. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zha… find
  44. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-attribute helpfulness dataset for SteerLM. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings o…
  45. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. find
  46. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees, 2024.
  47. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following, 2024. URL https://arxiv.org/abs/2310.07641. ↗
  48. Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment, 2024. URL https://arxiv.org/abs/2404.16792. — [arXiv:2404.16792](https://arxiv.org/abs/2404.16792) ↗
  49. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  50. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving LLM helpfulness & harmlessness with RLAIF, November 2023. find
  51. Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf, 2024. URL https://arxiv.org/abs/2401.16335. — [arXiv:2401.16335](https://arxiv.org/abs/2401.16335) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer2-preference-complementing-ratings-preferences/