Elamur Research⌕
☰
home / papers / 2025 / Checklists Are Better Than Reward Models For Aligning Language Models

Checklists Are Better Than Reward Models For Aligning Language Models

Paper · 2025 · Vijay Viswanathan, Yanchao Sun, Shuang Ma et al.

Summary

Reinforcement learning works well for tasks with checkable answers. It is much harder to apply to open-ended instruction following, where reward models can be gamed and generic criteria such as "helpfulness" miss request-specific needs. The authors propose Reinforcement Learning from Checklist Feedback (RLCF), which turns each instruction into its own weighted list of yes/no requirements.

To build the checklists, several small models first draft responses of mixed quality. A large model then lists the ways those drafts could fail and assigns each item an importance weight. Each response is graded item by item, using an LLM judge or a verification program where one applies. The weighted total is used to rank sampled response pairs for DPO.

Applied to Qwen2.5-7B-Instruct, RLCF was the only tested feedback source that improved all five benchmarks. It also transferred off-policy to other 7–8B models.

This matters because RLCF needs only a teacher model, with no human labels. It suggests that instruction-specific rubrics can provide a stronger alignment signal than learned reward models.

Key points

  • WildChecklists: 130,000 WildChat instructions paired with synthetic checklists.
    • Candidate responses came from Qwen2.5-0.5B, 1.5B, 3B and 7B, and Qwen2.5-72B-Instruct wrote the checklists.
    • Two universal requirements (staying concise and on-topic, matching the appropriate tone) were added to curb reward hacking.
  • Checklist quality: candidate-based checklists beat direct prompting on overall preference, at 56% vs 38% in manual evaluation and 51.2% vs 40.6% when judged by gpt-4o on InFoBench.
  • Scoring and training:
    • Qwen2.5-72B-Instruct rates each checklist item from 0 to 100, averaged over 25 samples, with code verifiers handling hard discrete constraints.
    • Response pairs are sampled at temperature 1.3 and top-p 0.9, then used for DPO.
    • Baselines included DPO on UltraFeedback and on Skywork reward-model rankings.
  • Results on Qwen2.5-7B-Instruct:
    • FollowBench hard satisfaction rate: about +4 points (5.4% relative).
    • InFoBench: about +6 points (6.9% relative).
    • Arena-Hard win rate: about +3 points (6.4% relative).
    • Off-policy, Llama 3.1 8B Instruct rose from 68.0 to 72.3 FollowBench HSR, and OLMo 2 7B Instruct rose from 74.6 to 76.6 IFEval average.
  • Cost: the scoring step requires about 4 days of compute across a node of eight H100 GPUs. Using 5 judge samples instead of 25 roughly halves the time with a small accuracy loss.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓University of Maryland, College ParkCollege Park · United States141 works · 29,174 citations🏢AppleCupertino · United States23 works · 2,318 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig and Tongshuang Wu. 2025. Checklists Are Better Than Reward Models For Aligning Language Models. arXiv:2507.18624.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.18624 · arXiv 2507.18624

@misc{viswanathan2025checklists,
  title = {{Checklists Are Better Than Reward Models For Aligning Language Models}},
  author = {Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig and Tongshuang Wu},
  year = {2025},
  doi = {10.48550/arxiv.2507.18624},
  eprint = {2507.18624},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.18624}
}

References

54 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. "we need structured output": Towards user-centered constraints on large language model output. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2024a. U… find
  2. Wenting Zhao, Xiang Ren, John Frederick Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. ArXiv, abs/2405.01470, 2024. URL https://api.semanticscholar.org/CorpusID:269390491. ↗
  3. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations. Colin Raffel, Noam M. Shazeer… — Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  4. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://api. semanticscholar.org/Co… find
  5. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya… find
  6. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. ArXiv, abs/2406.08464, 2024. URL https://api.semanticscholar.org/ CorpusID:270391432. ↗
  7. Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini,… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  8. Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019. URL https://api.semanticscholar.org/CorpusID:202660943. ↗
  9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Chris Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jef… — Constitutional AI: Harmlessness from AI Feedback
  10. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Che… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  11. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025. URL https://arxiv.org/abs/2507.02833. ↗
  12. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models. ArXiv, abs/2406.13542, 2024. URL https://api.semanticscholar.org/ CorpusID:270620157. ↗
  13. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preference: Complementing ratings with preferences. ArXiv, abs/2410.01257, 2024a. URL https://api.semanticscholar.org/CorpusID:273025954. ↗
  14. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, Dj Dvijotham, Adam Fisch, Katherine Heller, Stephen R. Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. ArXiv, abs/2312.09… — [arXiv:2312.09244](https://arxiv.org/abs/2312.09244) find
  15. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment. ArXiv, abs/2310.16944… find
  16. Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. ArXiv, abs/2503.01067, 2025. URL https://api.semanticscholar.org/CorpusID:276742134. — [arXiv:2503.01067](https://arxiv.org/abs/2503.01067) ↗
  17. Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, A. See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz,… — Improving alignment of dialogue agents via targeted human judgements
  18. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Annual Meeting of the Association for Computational Linguistics, 2023. URL…
  19. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. 2024.
  20. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. ArXiv, abs/2406.11939, 2024. URL https://api.semanticscholar.org/ CorpusID:270562889. ↗
  21. Sukai Huang, Shu-Wei Liu, Nir Lipovetzky, and Trevor Cohn. The dark side of rich rewards: Understanding and mitigating noise in vlm rewards. 2024. URL https://api.semanticscholar.org/CorpusID:272832041. ↗
  22. Jonathan Cook, Tim Rocktäschel, Jakob N. Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve llm evaluation and generation. ArXiv, abs/2410.03608, 2024. URL https://api.semanticscholar.org/CorpusID:273162357. ↗
  23. Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with instructable reward models. In International Conference on Learning Representations, 2023. URL https://api.semanticscholar.org/ CorpusID:263831633. ↗
  24. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei… find
  25. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ArXiv, abs/1904.09751, 2019. URL https://api.semanticscholar.org/ CorpusID:127986954. ↗
  26. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instruction-following ability of large language models. ArXiv, abs/2404.02823, 2024. URL https://api.semanticscholar.org/CorpusID:268876020. ↗
  27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings of the 29th Symposium on Operating Systems Principles, 2023. URL https://ap…
  28. Victor Wang, Michael J.Q. Zhang, and Eunsol Choi. Improving llm-as-a-judge inference with the judgment distribution. ArXiv, abs/2503.03064, 2025a. URL https://api.semanticscholar.org/CorpusID:276781945. ↗
  29. Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. Why do large language models (llms) struggle to count letters? ArXiv, abs/2412.18626, 2024. URL https://api.semanticscholar.org/CorpusID:275118941. — [arXiv:2412.18626](https://arxiv.org/abs/2412.18626) ↗
  30. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. ArXiv, abs/2311.07911, 2023. URL https://api.semanticscholar.org/CorpusID:265157752. ↗
  31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290, 2023. URL https://api.semanticscholar.org/CorpusID:258959321. ↗
  32. Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, OpenLLMAI Team, Netease Fuxi, AI Lab, and Alibaba Group. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. ArXiv, abs/2405.11143, 2024. URL https://api.semanticscholar.org/CorpusID: 269921667. ↗
  33. Yann Dubois, Bal’azs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. ArXiv, abs/2404.04475, 2024. URL https://api.semanticscholar.org/CorpusID:269004605. ↗
  34. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL https://api.semanticscholar.org/CorpusID: 7200347. ↗
  35. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. ArXiv, abs/2403.13372, 2024. URL https://api.semanticscholar.org/CorpusID:268536974. ↗
  36. Chris Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. ArXiv, abs/2410.18451, 2024b. URL https://api.semanticscholar.org/CorpusID:273549327. ↗
  37. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Conference on Empirical Methods in Natural Language Processing, 2024b. URL https://api.semanticscholar.org/CorpusID: 270562658. ↗
  38. Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Rewardbench: Evaluating reward models for language modeling. ArXiv, abs/2403.13787, 2… find
  39. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. ArXiv, abs/2310.01377, 2023. URL https://api.semanticscholar.org/CorpusID:263605623. ↗
  40. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Daniel Morrison, Noah A. Smith, Hanna Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. 2025. URL https://api.semanticscholar.org/CorpusID:279119102. ↗
  41. Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. ArXiv, abs/2503.15477, 2025. URL https://api.semanticscholar.org/CorpusID:277112967. — [arXiv:2503.15477](https://arxiv.org/abs/2503.15477) ↗
  42. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. ArXiv, abs/2308.01263, 2023. URL https://api.semanticscholar.org/CorpusID:260378842. ↗
  43. Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021. URL https://api.semanticscholar.org/Cor… ↗
  44. Stephanie C. Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar.org/CorpusID:237532606. ↗
  45. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur’elien Rodrig… — The Llama 3 Herd of Models
  46. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish I… find
  47. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024c.
  48. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv, abs/2304.12244, 2023. URL https://api.semanticscholar.org/CorpusID:258298159. ↗
  49. Qi He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https://api.semanticscholar.org/CorpusID:26936… ↗
  50. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024. URL https://api.semanticscholar.org/ CorpusID:267412607. ↗
  51. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. ArXiv, abs/2310.15123, 2023. URL https://api.semanticscholar.org/CorpusID:264591429. ↗
  52. Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. ArXiv, abs/2412.13091, 2024. URL https://api.semanticscholar.org/CorpusID:274788535. ↗
  53. Zongqi Wang, Tianle Gu, Chen Gong, Xin Tian, Siqi Bao, and Yujiu Yang. From rankings to insights: Evaluation should shift focus from leaderboard to feedback. ArXiv, abs/2505.06698, 2025b. URL https://api.semanticscholar.org/CorpusID:278501945. — [arXiv:2505.06698](https://arxiv.org/abs/2505.06698) ↗
  54. Jacob Dineen, Aswin RRV, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, et al. Qa-lign: Aligning llms through constitutionally decomposed qa. arXiv preprint arXiv:2506.08123, 2025. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/viswanathan-2025-checklists-better-than-reward-models/