Elamur Research⌕
☰
home / papers / 2026 / Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling

Paper · 2026 · Pankayaraj Pathmanathan and Furong Huang

Summary

Reward models (RMs) trained on human preference pairs often fail under distribution shift or small targeted edits. Earlier ways of finding these failures depend on knowing the relevant preference attributes ahead of time, typically by prompting GPT-4 to write counterfactuals, so they generalise poorly to new models or datasets. The authors define an RM failure as a response that keeps its true preference class but gets the wrong reward ordering. To find such failures, they let the RM steer its own attack. A preference-aligned policy proposes top-k next tokens, and decoding chooses the token that pushes the reward the wrong way, with a likelihood term that keeps the text fluent. Their framework, REFORM, adds the discovered failures to only the 5% most influential training pairs and then fine-tunes the RM. On Anthropic Helpful Harmless and PKU Beavertails, robustness rises by 35–45% on average with no loss of in-distribution accuracy. Downstream alignment quality is also preserved. The approach offers a cheap way to stress-test and repair RMs without attribute priors or large relabelling efforts.

Key points

  • Failure definition and search: A failure is a class-consistent but reward-inconsistent variant, such as a preferred answer that receives a low score.
    • Discovery works by token-level controlled decoding over the top-k candidates of an aligned policy.
    • It scores partial sequences with the RM as a proxy, following Khanov et al.
    • A weight α on log-likelihood under the policy keeps the output coherent.
  • Black-box and model-agnostic: The method treats both the policy and the RM as black boxes. Its decoding cost is comparable to standard controlled decoding.
  • Small, targeted correction: Only the top 5% most influential pairs, ranked by influence scores, are augmented. The RM is then fine-tuned with the Bradley–Terry loss.
  • Results:
    • Tested RMs include Mistral-7B and Qwen-14B, with an average robustness gain of 35–45%.
    • Under Best of N (N=16), PPO (LoRA r=256, α=1024, lr 3×10⁻⁶) and DPO, utility, readability and diversity hold steady or improve slightly.
    • Spurious correlations linked to reward hacking are reduced.
  • Limitations:
    • The method surfaces failure examples but does not explain why they occur.
    • It cannot enumerate every possible failure, so the authors make no claim of universal robustness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Pankayaraj Pathmanathan and Furong Huang. 2026. Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.418

@inproceedings{pathmanathan2026teach,
  title = {{Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling}},
  author = {Pankayaraj Pathmanathan and Furong Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.418},
  url = {https://doi.org/10.18653/v1/2026.acl-long.418}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  2. Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. 2024. Zero-shot llm-guided counterfactual generation: A case study on nlp model evaluation. Preprint, arXiv:2405.04793. — [arXiv:2405.04793](https://arxiv.org/abs/2405.04793) find
  3. Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324. find
  4. Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. 2024. Transfer q star: Principled decoding for llm alignment. Preprint, arXiv:2405.20495. — [arXiv:2405.20495](https://arxiv.org/abs/2405.20495) find
  5. Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Odin: Disentangled reward mitigates hacking in rlhf. Preprint, arXiv:2402.07319. find
  6. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2024. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. Preprint, arXiv… — [arXiv:2312.09244](https://arxiv.org/abs/2312.09244) find
  7. Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimization. Preprint, arXiv:2210.10760.
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  9. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. 2024. Direct language model alignment from online ai feedback. Preprint, arXiv:2402.04792. — [arXiv:2402.04792](https://arxiv.org/abs/2402.04792) find
  10. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2022. Unsolved problems in ml safety. Preprint, arXiv:2109.13916. find
  11. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657.
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Junqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lecue, and Manuela Veloso. 2025. Interpreting language reward models via contrastive explanations. Preprint, arXiv:2411.16502. — [arXiv:2411.16502](https://arxiv.org/abs/2411.16502) find
  14. Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. 2024. Args: Alignment as reward-guided search. Preprint, arXiv:2402.01694.
  15. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of rlhf on llm generalisation and diversity. Preprint, arXiv:2310.06452.
  16. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. Preprint, arXiv:1510.03055.
  17. Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, and Minlie Huang. 2022. Rethinking and refining the distinct metric. Preprint, arXiv:2202.13587. find
  18. Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. 2025. Rrm: Robust reward model training mitigates reward hackin… — [arXiv:2409.13156](https://arxiv.org/abs/2409.13156) find
  19. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. 2024. Controlled decoding from language models. Preprint, arXiv:2310.17022. find
  20. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assis…
  21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  22. Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. 2024. Offsetbias: Leveraging debiased data for tuning evaluators. Preprint, arXiv:2407.06551.
  23. Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, and Furong Huang. 2024. Is poisoning a real threat to llm alignment? maybe more so than you think. arXiv preprint arXiv:2406.12091. — [arXiv:2406.12091](https://arxiv.org/abs/2406.12091) find
  24. Pankayaraj Pathmanathan, Udari Madhushani Sehwag, Michael-Andrei Panaitescu-Liess, and Furong Huang. 2025. Advbdgen: Adversarially fortified prompt-specific fuzzy backdoor generator against llm alignment. Preprint, arXiv:2410.11283. — [arXiv:2410.11283](https://arxiv.org/abs/2410.11283) find
  25. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. Preprint, arXiv:2312.11456. find
  26. Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. 2025. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. Preprint, arXiv:2410.08193. — [arXiv:2410.08193](https://arxiv.org/abs/2410.08193) find
  27. Linyi Yang, Eoin M. Kenny, Tin Lok James Ng, Yi Yang, Barry Smyth, and Ruihai Dong. 2020. Generating plausible counterfactual explanations for deep transformers in financial text classification. Preprint, arXiv:2010.12512. find
  28. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. Preprint, arXiv:2310.07641. find
  29. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and stella: distillation of sota embedding models. Preprint, arXiv:2412.19048. — [arXiv:2412.19048](https://arxiv.org/abs/2412.19048) find
  30. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! Preprint, arXiv:2310.03693.
  31. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  32. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  34. Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models. Preprint, arXiv:2401.12187. — [arXiv:2401.12187](https://arxiv.org/abs/2401.12187) find
  35. Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. Preprint, arXiv:2310.05199. find
  36. Gemini Team. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pathmanathan-2026-teach-reward-model-correct-itself/