Elamur Research⌕
☰
home / papers / 2024 / Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation

Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation

Paper · 2024 · Jiaming Shen, Ran Xu, Yennie Jun et al.

Summary

Reward models used to align LLMs need preference data: a prompt, two candidate answers and a label saying which is better. Human annotation is costly, so labels are often produced by an LLM. Those automatic judgments can be wrong, and the errors degrade RM training. The authors (Google and Emory University) propose RMBoost, which reverses the usual order of synthetic data creation. First, a model writes one response and a preference label is fixed in advance. Then a second response is generated from the first one, either improved or deliberately degraded along chosen quality aspects such as helpfulness, relevance, completeness or faithfulness. Because each pair is built to match its label, label noise drops. Varying the aspects across prompts also makes the generated responses more diverse. In experiments on three datasets, RMBoost-generated data outperformed alternative synthetic pipelines, including RLAIF-style and RLCD-style generation, and clearly improved four different reward models. This offers a cheaper way to build reliable preference data for alignment.

Key points

  • Comparison with existing paradigms:
    • RLAIF generates two responses, then a preference model predicts the label.
    • RLCD uses two contrasting prompts and infers the label from which prompt was used. Its prompts usually target a single aspect, which limits diversity.
    • RMBoost conditions the second response on both the first response and the pre-selected label.
  • Two claimed benefits: pairs are constructed intentionally, which reduces labeling noise, and multi-aspect prompting gives fine-grained control and more varied data.
  • Evaluation scope: three diverse datasets and four distinct reward models. RMBoost beat the other synthetic preference generation methods tested.
  • TLDR Summarization setup: the prompts use a guideline with three aspects: Coherence, Accuracy and Coverage. To produce the less preferred summary, the model first picks a few of these aspects and then writes a response that is worse than the "good" first summary on them.
  • Paper details: Jiaming Shen, Ran Xu, Carl Yang, Zhen Qin, Tianqi Liu, Michael Bendersky and others. arXiv:2407.16008, v2 dated March 2025.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Emory UniversityAtlanta · United States36 works · 39,779 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner and Michael Bendersky. 2024. Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation. arXiv:2407.16008.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.16008 · arXiv 2407.16008

@misc{shen2024boosting,
  title = {{Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation}},
  author = {Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner and Michael Bendersky},
  year = {2024},
  doi = {10.48550/arxiv.2407.16008},
  eprint = {2407.16008},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.16008}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Muelle… — Constitutional AI: Harmlessness from AI Feedback
  3. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs the method of paired comparisons. Biometrika, 39:324–345, 1952. URL https://api.semanticscholar.org/CorpusID:121987403. ↗
  4. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. ArXiv, abs/2310.01377, 2023. URL https://api.semanticscholar.org/CorpusID: 263605623. ↗
  5. Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.06961, 2024. — [arXiv:2410.06961](https://arxiv.org/abs/2410.06961) find
  6. Joey Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models. In ICLR, 2023. URL https://openreview.net/forum?id= hJqGbUpDGV. ↗
  7. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In ACL, 2023.
  8. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. EMNLP, abs/2305.13735, 2023. URL https://api.semanticscholar.org/CorpusID: 258841835. ↗
  9. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In ICLR, 2024. URL https://openreview.net/forum?id=PXD3FAVHJT. ↗
  10. W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro G Allievi. Models of human preference for learning reward functions. TMLR, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=hpKJkVoThY. ↗
  11. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. ArXiv, abs/2309.00267, 2023. URL https://api.semanticscholar.org/ CorpusID:261493811. ↗
  12. Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064, 2024. find
  13. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data for language models. 2024a. URL https://api.semanticscholar.org/ CorpusID:269042851. ↗
  14. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. ICLR, 2024b.
  15. Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. Advances in Neural Information Processing Systems, 31, 2018. find
  16. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. find
  17. Decoupled weight decay regularization. arXiv preprint Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. NeurIPS, 35:462–477, 2022.
  18. Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L’eonard Hussenot, et al. Gemma: Open models based on gemini research and technology. ArXiv, abs/2403.08295, 2024. URL https:/… find
  19. OpenAI. Gpt-4 technical report, 2023.
  20. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — Training language models to follow instructions with human feedback
  21. Aliz’ee Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. ArXiv, abs/2401.12086, 2024. URL https://api.semanticscholar.org/CorpusID:267069315. ↗
  22. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
  23. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.
  24. Giuseppe Russo, Nora Hollenstein, Claudiu Musat, and Ce Zhang. Control, generate, augment: A scalable framework for multi-attribute text generation. EMNLP Findings, 2020. find
  25. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of-thought demonstrations for large language models. ICML, abs/2302.00618, 2023. URL https://api.semanticscholar.org/CorpusID: 256459681. — [arXiv:2302.00618](https://arxiv.org/abs/2302.00618) ↗
  26. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  27. Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. ICLR, 2015. find
  28. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. NeurIPS, 36, 2023.
  29. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  30. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023. URL https://api.semanticscholar.org… ↗
  31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL, 2023. URL https://api.semanticscholar.org/CorpusID:254877310. ↗
  32. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. NeurIPS, abs/2306.01693, 2023. URL https://api. semanticscholar.org/CorpusID:25906409…
  33. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
  34. Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rlcd: Reinforcement learning from contrastive distillation for language model alignment. 2023. URL https://api.semanticscholar.org/CorpusID:260357852. ↗
  35. Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation. In EMNLP, pp. 11653–11669, 2022.
  36. Dian Yu, Zhou Yu, and Kenji Sagae. Attribute alignment: Controlling text generation from pre-trained language models. EMNLP Findings, 2021. find
  37. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J. Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. NeurIPS (Datasets and Benchmark Track), abs/2306.15895, 2023a. URL https://api.semanticscholar.org… ↗
  38. Yue Yu, Yuchen Zhuang, Rongzhi Zhang, Yu Meng, Jiaming Shen, and Chao Zhang. Regen: Zero-shot text classification via training data generation with progressive dense retrieval. ACL Findings, 2023b. find
  39. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. NeurIPS, 36, 2024a. find
  40. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024b. URL https://openreview.net/forum?id=0NphYCmgua. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2024-boosting-reward-model-preference-conditional/