Elamur Research⌕
☰
home / papers / 2026 / P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

Paper · 2026 · Kwangwook Seo and Dongha Lee

Summary

Personalized reward models usually condition on a user's past interactions. They treat that context as a fixed persona or an implicit signal. This makes it hard to see what actually drives a specific judgment, and it misses how a user's priorities change from one query to the next. P-Check addresses this with a plug-and-play checklist generator that sits alongside an LLM-as-a-Judge reward model. For each user history and query, it writes explicit, query-specific evaluation criteria, and the judge scores candidate responses against them. A naive approach would distill checklists from a strong LLM, but those tend to reward generic quality and include trivial items. The authors instead propose Preference-Contrastive Criterion Weighting, which gives each criterion a saliency weight based on how much it helps separate the preferred response from contrasting ones. Across three personalized reward benchmarks, P-Check beats prior personalized reward models in accuracy and holds up out-of-distribution. Its signals also improve downstream personalized generation. The work points toward reward models that are more transparent and better suited to individual users.

Key points

  • Preliminary study (100 users from PersonalRewardBench):
    • LLM judges (GPT-4o, GPT-4o-mini, Qwen3-8b, Llama3-8b) were close to chance (about 50–53%) at choosing the oracle checklist over a counter-preference one.
    • Supplying an explicit oracle checklist raised preference accuracy markedly.
    • An oracle descriptive persona helped only slightly.
  • Training data: checklists are synthesized from logged preference pairs, grounded in a condensed user-level summary (GP_u) and the current query. Rejection sampling removes low-quality outputs.
  • Criterion weighting has two steps:
    • Inter-user contrastive sampling adds responses written for users with different preference axes to each pair.
    • Personalized saliency scoring measures how much the chosen-vs-contrast separation shrinks when a criterion is removed. These scores become extra training labels.
  • Evaluation: in-distribution and OOD settings on PRISM, ChatbotArena and BESPOKE. Checklist signals helped when used as rewards for DPO and Best-of-N selection. They also worked as verbal feedback to the generator, without any policy updates.
  • Limitations:
    • Subtle factors such as tone or "feel" are hard to express as discrete criteria.
    • Errors in a checklist carry over into the reward.
    • The multi-step training pipeline adds cost, although test-time latency stays modest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yonsei UniversitySeoul · South Korea17 works · 501 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Kwangwook Seo and Dongha Lee. 2026. P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2011

@inproceedings{seo2026p,
  title = {{P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist}},
  author = {Kwangwook Seo and Dongha Lee},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2011},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2011}
}

References

53 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. 2022. Fine-tuning language models to find agreement among humans with diverse preferences.… find
  2. Nishant Balepur, Vishakh Padmakumar, Fumeng Yang, Shi Feng, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. Whose boat does it float? improving personalization in preference tuning via inferred user personas. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti…
  3. Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, Dongwook Choi, Minseok Kang, Gwanhoon Im, ByeongUng Cho, Hyojun Kim, Jun Hee Han, Taeyoon Kwon, Minju Kim, Beong woo Kwak, and 2 others. 2025. Web-shepherd: Advanci… find
  4. Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak. 2025a. PAL: Sample-efficient personalized reward modeling for pluralistic alignment. In The Thirteenth International Conference on Learning Representations. find
  5. Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. 2025b. PAD: Personalized alignment at decoding-time. In The Thirteenth International Conference on Learning Representations. find
  6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. Preprint, arXiv:2403.04132.
  7. Robin Gregory, Sarah Lichtenstein, and Paul Slovic. 1993. Valuing environmental resources: a constructive approach. Journal of Risk and Uncertainty, 7(2):177–197. find
  8. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. Preprint, arXiv:2507.17746. find
  9. Chrisopher K Hsee, George F Loewenstein, Sally Blount, and Max H Bazerman. 1999. Preference reversals between joint and separate evaluations of options: A review and theoretical analysis. Psychological bulletin, 125(5):576. find
  10. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computati…
  11. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. Preprint, arXiv:2310.11564. find
  12. Hyunseo Kim, Sangam Lee, Kwangwook Seo, and Dongha Lee. 2025a. Bespoke: Benchmark for search-augmented large language model personalization via diagnostic feedback. Preprint, arXiv:2509.21106. — [arXiv:2509.21106](https://arxiv.org/abs/2509.21106) find
  13. Jaehyung Kim and Yiming Yang. 2025. Few-shot personalization of LLMs with mis-aligned responses. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11943–11974, Albu… find
  14. Minbeom Kim, Kang-il Lee, Seongho Joo, Hwaran Lee, Thibaut Thonet, and Kyomin Jung. 2025b. Drift: Decoding-time personalized alignments with implicit user preferences. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6107–6126, Suzhou, China. Association for Computatio… find
  15. Seoyeon Kim, Kwangwook Seo, Hyungjoo Chae, Jinyoung Yeo, and Dongha Lee. 2024a. Ver-ifiNER: Verification-augmented NER via knowledge-grounded reasoning with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page… find
  16. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024b. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
  17. Sunghwan Kim, Kwangwook Seo, Tongyoung Kim, Jinyoung Yeo, and Dongha Lee. 2025c. Stop playing the guessing game! target-free user simulation for evaluating conversational recommender systems. Preprint, arXiv:2411.16160. — [arXiv:2411.16160](https://arxiv.org/abs/2411.16160) find
  18. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The PRISM alignment dataset: What participatory, representative and individualised human feedb… find
  19. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  20. Shuyue Stella Li, Melanie Sclar, Hunter Lang, Ansong Ni, Jacqueline He, Puxin Xu, Andrew Cohen, Chan Young Park, Yulia Tsvetkov, and Asli Celikyilmaz. 2025. Prefpalette: Personalized preference modeling with latent attributes. In Second Conference on Language Modeling. find
  21. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2025a. LLMs + persona-plug = personalized LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9373–9385, Vie…
  22. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2025b. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. Preprint, arXiv:2510.07743.
  23. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  24. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025c. Inference-time scaling for generalist reward modeling. Preprint, arXiv:2504.02495.
  25. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  26. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  27. Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. 2024. Improving context-aware preference modeling for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  28. Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2024. Personalizing reinforcement learning from human feedback with variational preference learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  29. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Information Processing Sy… find
  30. Michael J. Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, and Diyi Yang. 2025. SynthesizeMe! inducing persona-guided prompts for personalized reward models in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… find
  31. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024. LaMP: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7370–7392, Bangkok, Thailand. Association for C…
  32. Kwangwook Seo, Donguk Kwon, and Dongha Lee. 2025. MT-RAIG: Novel benchmark and evaluation framework for retrieval-augmented insight generation over multiple tables. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23142– 23172,… find
  33. Kwangwook Seo, Jinyoung Yeo, and Dongha Lee. 2024. Unveiling implicit table knowledge with question-then-pinpoint reasoner for insightful table summarization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12337– 12362, Miami, Florida, USA. Association for Computatio… find
  34. Paul Slovic. 1995. The construction of preference. American psychologist, 50(5):364. find
  35. Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. Checklists are better than reward models for aligning language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  36. Shujin Wu, Yi R. Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. 2025. Aligning LLMs with individual preferences via interaction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7648–7662, Abu Dhabi, UAE. Association for Computational Linguist…
  37. Zhouhang Xie, Junda Wu, Yiran Shen, Raghav Jain, Yu Xia, Xintong Li, Aaron Chang, Ryan A. Rossi, Tong Yu, Sachin Kumar, Bodhisattwa Prasad Majumder, Jingbo Shang, Prithviraj Ammanabrolu, and Julian McAuley. 2025. A survey on personalized and pluralistic preference alignment in large language models.… find
  38. Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sainbayar Sukhbaatar. 2025. Stepwiser: Stepwise generative judges for wiser reasoning. Preprint, arXiv:2508.19229. — [arXiv:2508.19229](https://arxiv.org/abs/2508.19229) find
  39. Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. 2025a. GenARM: Reward guided generation with autoregressive reward model for test-time alignment. In The Thirteenth International Conference on Learning Representations. find
  40. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2025b. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. In International Conference on Learning Representations (ICLR).
  41. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024. FLASK: Fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations.
  42. Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun LIU. 2025. Learning LLM-as-a-judge for preference alignment. In The Thirteenth International Conference on Learning Representations. find
  43. LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  44. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593. find
  45. Ping Yu, Weizhe Yuan, Olga Golovneva, Tianhao Wu, Sainbayar Sukhbaatar, Jason E Weston, and Jing Xu. 2025. R.i.p.: Better models by survival of the fittest prompts. In International Conference on Machine Learning (ICML). find
  46. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. Preprint, arXiv:2308.01825.
  47. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24.
  48. Zhiwei Zhang, Hui Liu, Xiaomin Li, Zhenwei Dai, Jingying Zeng, Fali Wang, Minhua Lin, Ramraj Chandradevan, Zhen Li, Chen Luo, Xianfeng Tang, Qi He, and Suhang Wang. 2025. Bradley-terry and multi-objective reward modeling are complementary. Preprint, arXiv:2507.07375. — [arXiv:2507.07375](https://arxiv.org/abs/2507.07375) find
  49. Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. 2025a. Genprm: Scaling test-time compute of process reward models via generative reasoning. Preprint, arXiv:2504.00891. — [arXiv:2504.00891](https://arxiv.org/abs/2504.00891) find
  50. Siyan Zhao, John Dang, and Aditya Grover. 2024. Group preference optimization: Few-shot alignment of large language models. In The Twelfth International Conference on Learning Representations. find
  51. Weixiang Zhao, Xingyu Sui, Yulin Hu, Jiahe Guo, Haix-iao Liu, Biye Li, Yanyan Zhao, Bing Qin, and Ting Liu. 2025b. Teaching language models to evolve with users: Dynamic profile modeling for personalized alignment. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  52. Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. 2025c. Nextquill: Causal preference modeling for enhancing llm personalization. Preprint, arXiv:2506.02368. — [arXiv:2506.02368](https://arxiv.org/abs/2506.02368) find
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seo-2026-p-check-advancing-personalized-reward/