Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems
Paper · 2025 · Hao Peng, Yunjia Qi, Xiaozhi Wang et al.
Summary
Reward models for LLMs are usually trained only on human preferences. They can favour superficial traits such as longer or more polished answers and miss factual mistakes or ignored constraints. The authors propose agentic reward modeling, in which the final reward is a weighted combination of a base preference score and the outputs of verification agents selected for each instruction. Their system, RewardAgent, has three modules:
- Router: decides which checkers an instruction needs.
- Verification agents: one checks factuality pairwise, focusing only on where two responses disagree and gathering evidence from a search engine or the LLM's own knowledge. The other checks instruction-following by extracting hard constraints, writing Python checker code and running it.
- Judger: merges these signals with ArmoRM preference scores.
RewardAgent was tested on reward benchmarks, best-of-n response selection and the construction of DPO training pairs. It consistently beat plain reward models. The work shows that verifiable correctness checks can make reward signals more reliable and easier to interpret.
Key points
- Two variants were built. RewardAgent MINI runs on GPT-4o mini. RewardAgent LLAMA runs on Llama3-8B Instruct, with Qwen2.5-Coder 7B handling the instruction-following agent. ArmoRM supplies the preference score in both.
- The authors created a new benchmark, IF-Bench, with 444 instances. Each pairs an instruction containing several hard constraints with one compliant and one violating response. RM-Bench and JudgeBench were also used.
- Best-of-n experiments sampled 32 responses at temperature 1.0 from Llama3-8B Instruct and GPT-4o. They were run on TriviaQA, IFEval and CELLO, where RewardAgent outperformed ArmoRM.
- For DPO, zephyr-7b-sft-full was trained on pairs selected from UltraFeedback and from on-policy data (20,000 instructions with 8 samples each). RewardAgent-labelled data beat ArmoRM-labelled data, with the largest gains in the on-policy setting. It also beat the original GPT-4-based UltraFeedback labels.
- Code is released at github.com/THU-KEG/Agentic-Reward-Modeling.
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works
Cite
Cite as Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou and Juanzi Li. 2025. Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
@inproceedings{peng2025agentic,
title = {{Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems}},
author = {Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou and Juanzi Li},
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2025},
doi = {10.18653/v1/2025.acl-long.775},
url = {https://doi.org/10.18653/v1/2025.acl-long.775}
}References
- Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisu-pati, Toby Drane, and Young Sun Park. 2025. Atla selene mini: A general purpose evaluation model. Preprint, arXiv:2501.17195. — [arXiv:2501.17195](https://arxiv.org/abs/2501.17195) find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2025-agentic-reward-modeling-integrating-human/