Elamur Research⌕
☰
home / papers / 2026 / PRISM: Probabilistic Reward Model with Inherent Structural Modeling

PRISM: Probabilistic Reward Model with Inherent Structural Modeling

Paper · 2026 · Yuhang Zhou, Yixin Cao, Yuchen Ni et al.

Summary

Reward models and LLM judges usually reduce many human judgments to one scalar. This blurs two distinct things: legitimate disagreement between annotators who value different qualities (Subjective Preference), and low-confidence judgments caused by ambiguity or missing expertise (Epistemic Uncertainty). The authors argue this collapse makes alignment fragile and invites reward hacking.

PRISM instead treats the reward as a conditional Mixture of Gaussians built on a frozen LLM backbone. Several lightweight expert heads each predict a mean score and a bounded standard deviation, and an input-aware router assigns mixture weights.

Training has two stages:

  • Stage 1: Experts specialize without a router by optimizing a mixture likelihood over large-scale pairwise comparisons. The variance lets an expert signal low confidence rather than drift toward an average.
  • Stage 2: Experts are frozen, and the router is learned from a small context-labeled set plus unlabeled preference pairs.

PRISM outperforms scalar Bradley-Terry baselines in accuracy and generalization. In rubric-based RL with GRPO, it curbs reward hacking and produces policies that hold up better under distribution shift.

Key points

  • Architecture: Each expert has a linear mean head and a sigmoid-bounded σ-head. The variance acts as a reliability gate that dampens gradients from uncertain experts, for example on ambiguous or out-of-domain inputs.

  • Policy results: On Arena-Hard v2.0, PRISM-MoG gave the best policies across all three backbones, compared with Single-BT and the untrained base:

    Policy modelPRISM-MoGSingle-BTBase
    LLaMA3.2-3B3.752.31.2
    LLaMA3.1-8B6.554.052.55
    Qwen3-4B20.219.112.7

  • Ablations: "Weight" (same router weights, but variance discarded) and "Uniform" (no router) both trail MoG. This indicates that both the uncertainty information and input-conditional routing contribute to the gains.

  • Limitations: The RL pipeline still collapses the distribution into a scalar surrogate. Public datasets with heterogeneous preferences are scarce, which restricts evaluation.

  • Code: Released at github.com/ALEX-nlp/PRISM. Published at ACL 2026 (Long Papers).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🏢ByteDanceBeijing · China47 works · 4,339 citations🎓Shanghai Innovation InstituteShanghai · China34 works · 325 citations🎓New York University ShanghaiShanghai · China11 works · 644 citations

Also on the byline: Shanghai Lixin University of Accounting and Finance.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Yuhang Zhou, Yixin Cao, Yuchen Ni, Shihan Dou, Xutian Chen, Ge Zhang, Xiang Liu and Guangnan Ye. 2026. PRISM: Probabilistic Reward Model with Inherent Structural Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.563

@inproceedings{zhou2026prism,
  title = {{PRISM: Probabilistic Reward Model with Inherent Structural Modeling}},
  author = {Yuhang Zhou, Yixin Cao, Yuchen Ni, Shihan Dou, Xutian Chen, Ge Zhang, Xiang Liu and Guangnan Ye},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.563},
  url = {https://doi.org/10.18653/v1/2026.acl-long.563}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  2. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  3. Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, et al. 2025. Toward generalizable evaluation in the llm era: A survey beyond benchmarks. arXiv preprint arXiv:2504.18838. — [arXiv:2504.18838](https://arxiv.org/abs/2504.18838) find
  4. Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Bedi, and Mengdi Wang. 2024. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment. find
  5. Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1–22. find
  6. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024a. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542. find
  7. Hanze Dong, Hanze Dong, Xiong Wei, Wei Xiong, Hanze Dong, Deepanshu Goyal, Deepanshu Goyal, Wei Xiong, Zhang Yihan, Rui Pan, Deepanshu Goyal, Shizhe Diao, Pan Rui, Jipeng Zhang, Shizhe Diao, Shizhe Diao, Kashun Shum, Jipeng Zhang, Zhang Jipeng, Tong Zhang, Kashun Shum, Kashun Shum, Zhang, and Tong Z… — RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
  8. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024b. Rlhf workflow: From reward modeling to online rlhf.
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Cornelia Gruber, Helen Alber, Bernd Bischl, Göran Kauermann, Barbara Plank, and Matthias Aßenmacher. 2025. Revisiting active learning under (human) label variation. arXiv preprint arXiv:2507.02593. find
  11. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746.
  12. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, et al. 2025. Reinforcement learning with rubric anchors. arXiv preprint arXiv:2508.12790. — [arXiv:2508.12790](https://arxiv.org/abs/2508.12790) find
  13. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2025. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: N… find
  14. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Rewardbench: Evaluating reward models for language modeling. find
  15. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. find
  16. Yoonho Lee, Jonathan Williams, Henrik Marklund, Archit Sharma, Eric Mitchell, Anikait Singh, and Chelsea Finn. 2024. Test-time alignment via hypothesis reweighting. arXiv preprint arXiv:2412.08812. — [arXiv:2412.08812](https://arxiv.org/abs/2412.08812) find
  17. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. find
  18. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Nat… find
  19. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.
  20. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184.
  21. Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2024. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847. — [arXiv:2410.00847](https://arxiv.org/abs/2410.00847) find
  22. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024a. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845.
  23. Ryan Lowe and Jan Leike. 2022. Aligning language models to follow instructions. OpenAI Blog, January, 27. find
  24. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.
  25. David JC MacKay. 1992. Bayesian interpolation. Neural computation, 4(3):415–447. find
  26. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  27. Charlie Masters, Stefano V Albrecht, et al. 2025. Arcane: A multi-agent framework for interpretable and configurable alignment. arXiv preprint arXiv:2512.06196. — [arXiv:2512.06196](https://arxiv.org/abs/2512.06196) find
  28. In Jae Myung. 2003. Tutorial on maximum likelihood estimation. Journal of mathematical Psychology, 47(1):90–100. find
  29. Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. 2024. Improving context-aware preference modeling for language models. find
  30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  31. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  32. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  33. Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, and Han Zhao. 2025. Micro: Mixture modeling and context-aware routing for personalized preference learning. arXiv preprint arXiv:2505.24846. find
  34. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024b. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Processing Systems, 37:1474–1501. find
  35. Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Ellie Evans, Daniel Egert, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. 2025. Rlbff: Binary flexible feedback to bridge between human feedback verifiable rewards. find
  36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  37. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for llms. In Advances in Neural Information Processing Systems. find
  38. Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. 2024. Online iterative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems, 37:81773–81807. find
  39. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.
  40. Yuhang Zhou, Xutian Chen, Yixin Cao, Yuchen Ni, Yu He, Siyu Tian, Xiang Liu, Jian Zhang, Chuanjun Ji, Guangnan Ye, et al. 2025. Teach2eval: An indirect evaluation method for llm by judging how it teaches. arXiv preprint arXiv:2505.12259. — [arXiv:2505.12259](https://arxiv.org/abs/2505.12259) find
  41. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. find
  42. Wangtao Sun, Xiang Cheng, Xing Yu, Haotian Xu, Zhao Yang, Shizhu He, Jun Zhao, and Kang Liu. 2025. Probabilistic uncertain reward model. arXiv preprint arXiv:2503.22480. — [arXiv:2503.22480](https://arxiv.org/abs/2503.22480) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-prism-probabilistic-reward-model-inherent/