Elamur Research⌕
☰
home / papers / 2025 / Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Paper · 2025 · Anisha Gunjal, Anthony Wang, Elaine Lau et al.

Summary

Reinforcement learning with verifiable rewards (RLVR) works well for math and code because answers can be checked automatically. Most real-world questions, however, are judged against many soft criteria rather than one correct answer. Preference-based reward models are a common substitute, but they are costly to train and tend to favour superficial traits such as length. The authors propose Rubrics as Rewards (RaR). For every training prompt, a strong LLM (GPT-4o or o3-mini), conditioned on a reference answer, writes a checklist of 7–20 weighted criteria. An LLM judge then scores rollouts against this checklist in one of two ways: item by item with a weighted average (explicit aggregation), or holistically in a single call (implicit aggregation). The score drives GRPO training of Qwen2.5-7B. Compared with judges that give direct Likert ratings, the best RaR setup gains up to 31% relative on HealthBench and 7% on GPQA-Diamond. The two rubric-annotated datasets are publicly released. The work shows that interpretable rubrics can extend RL post-training to expert domains without verifiable answers, and that they make small judges more dependable.

Key points

  • Two released training sets on Hugging Face, each with about 20k prompts drawn from natural_reasoning, SCP-116K, GeneralThought-430K and medical-o1 sources:
    • RaR-Medicine, with rubrics written by GPT-4o.
    • RaR-Science, with rubrics written by o3-mini and aligned to GPQA-Diamond categories.
  • Rubric items carry categorical labels (Essential, Important, Optional, Pitfall). RLVR is formally a special case of the framework: a single criterion with weight 1 that checks for an exact match.
  • Training setup:
    • Algorithm and policy: GRPO on Qwen2.5-7B.
    • Sampling: 16 rollouts per prompt.
    • Judge: gpt-4o-mini.
    • Batch size 96 and learning rate 5e-6.
    • Hardware: one node with 8 H100 GPUs.
  • Ablations on HealthBench-1k:
    • Rubrics with a broad range of criteria beat essential-only checks.
    • Adding weights or synthetic pitfall items changed results very little.
    • Reference-guided rubrics outperformed reference-free ones; among the reference-free generators, GPT-4o produced the most useful rubrics.
  • Rubric-structured rewards align better with human preferences when the judge is small, and they reduce performance variance across judge sizes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu and Sean Hendryx. 2025. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.17746 · arXiv 2507.17746

@misc{gunjal2025rubrics,
  title = {{Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains}},
  author = {Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu and Sean Hendryx},
  year = {2025},
  doi = {10.48550/arxiv.2507.17746},
  eprint = {2507.17746},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.17746}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Anugraha, Z. Tang, L. J. V. Miranda, H. Zhao, M. R. Farhansyah, G. Kuwanto, D. Wijaya, and G. I. Winata. R3: Robust rubric-agnostic reward models. arXiv preprint arXiv:2505.13388, 2025. — [arXiv:2505.13388](https://arxiv.org/abs/2505.13388) find
  2. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025.
  3. J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 2024.
  4. L. Chen, C. Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319, 2024. find
  5. X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y. Wang, H. Wang, Y. Zhang, D. Zhang, T. Zhang, et al. Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387, 2025.
  6. G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
  7. J. Dineen, A. RRV, Q. Liu, Z. Xu, X. Ye, M. Shen, Z. Li, S. Lu, C. Baral, M. Chen, et al. Qa-lign: Aligning llms through constitutionally decomposed qa. arXiv preprint arXiv:2506.08123, 2025. find
  8. V. Gallego. Configurable preference tuning with rubric-guided synthetic data. arXiv preprint arXiv:2506.11702, 2025. — [arXiv:2506.11702](https://arxiv.org/abs/2506.11702) find
  9. General Reasoning. General reasoning, 2025. URL https://gr.inc/. ↗
  10. A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023.
  11. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  12. J. Guo, Z. Chi, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei. Reward reasoning model. arXiv preprint arXiv:2505.14674, 2025. — [arXiv:2505.14674](https://arxiv.org/abs/2505.14674) find
  13. H. Hashemi, J. Eisner, C. Rosset, B. Van Durme, and C. Kedzie. Llm-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. arXiv preprint arXiv:2501.00274, 2024.
  14. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
  15. A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  16. M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang. Process reward models that think. arXiv preprint arXiv:2504.16828, 2025. — [arXiv:2504.16828](https://arxiv.org/abs/2504.16828) find
  17. S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024.
  18. Z. M. Kim, C. Park, V. Raheja, S. Kim, and D. Kang. Toward evaluative thinking: Meta policy optimization with evolving reward models. arXiv preprint arXiv:2504.20157, 2025. — [arXiv:2504.20157](https://arxiv.org/abs/2504.20157) find
  19. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. T \\ " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.
  20. S. Li, S. Dong, K. Luan, X. Di, and C. Ding. Enhancing reasoning through process supervision with monte carlo tree search. arXiv preprint arXiv:2501.01478, 2025. — [arXiv:2501.01478](https://arxiv.org/abs/2501.01478) find
  21. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. — Let's Verify Step by Step
  22. D. Lu, X. Tan, R. Xu, T. Yao, C. Qu, W. Chu, Y. Xu, and Y. Qi. Scp-116k: A high-quality problem-solution dataset and a generalized pipeline for automated extraction in the higher education science domain. arXiv preprint arXiv:2501.15587, 2025. — [arXiv:2501.15587](https://arxiv.org/abs/2501.15587) find
  23. X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen. General-reasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652, 2025.
  24. OpenAI o3-mini. Openai o3-min, 2025. URL https://openai.com/index/openai-o3-mini/. ↗
  25. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
  26. A. Pathak, R. Gandhi, V. Uttam, Y. Nakka, A. R. Jindal, P. Ghosh, A. Ramamoorthy, S. Verma, A. Mittal, A. Ased, et al. Rubric is all you need: Enhancing llm-based code evaluation with question-specific rubrics. arXiv preprint arXiv:2503.23989, 2025. — [arXiv:2503.23989](https://arxiv.org/abs/2503.23989) find
  27. J. Ruan, I. Nair, S. Cao, A. Liu, S. Munir, M. Pollens-Dempsey, T. Chiang, L. Kates, N. David, S. Chen, et al. Expertlongbench: Benchmarking language models on expert-level long-form generation tasks with structured checklists. arXiv preprint arXiv:2506.01241, 2025. — [arXiv:2506.01241](https://arxiv.org/abs/2506.01241) find
  28. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  29. P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
  30. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. find
  31. Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) find
  32. Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) find
  33. V. Viswanathan, Y. Sun, S. Ma, X. Kong, M. Cao, G. Neubig, and T. Wu. Checklists are better than reward models for aligning language models. arXiv preprint arXiv:2507.18624, 2025.
  34. H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18571, 2024.
  35. C. Whitehouse, T. Wang, P. Yu, X. Li, J. Weston, I. Kulikov, and S. Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320, 2025. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) find
  36. Z. Ye, F. Greenlee-Scott, M. Bartolo, P. Blunsom, J. A. Campos, and M. Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850, 2024. find
  37. W. Yuan, J. Yu, S. Jiang, K. Padthe, Y. Li, I. Kulikov, K. Cho, D. Wang, Y. Tian, J. E. Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXiv:2502.13124, 2025. find
  38. S. Zhang, Q. Liu, G. Qin, T. Naumann, and H. Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXiv:2502.19655, 2025. find
  39. X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025. find
  40. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/