Elamur Research⌕
☰
home / papers / 2026 / Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Debiasing Reward Models via Causally Motivated Inference-Time Intervention

Paper · 2026 · Kazutoshi Shinoda, Kosuke Nishida and Kyosuke Nishida

Summary

Reward models (RMs) used in RLHF often reward surface style instead of substance. They favor longer answers, more paragraphs, exclamation marks, bold Markdown and lexical overlap with the query, and these preferences can carry over into aligned LLMs. Earlier inference-time fixes handle only length and tend to trade accuracy in one evaluation setting for another.

The authors propose CIRM (Causal Intervention for Reward Models), which requires no retraining:

  • Neuron selection: on a small validation set, it ranks neurons by Spearman's ρ between their last-token activations and five measured spurious features, then keeps the most positively and negatively correlated ones.
  • Intervention: at inference, those activations are clamped to their validation medians.
  • Causal framing: the standard Bradley–Terry RM is reinterpreted as estimating a total effect. Fixing the mediator neurons turns it into an estimate of the controlled direct effect.

CIRM lowers bias sensitivity on RM benchmarks without performance trade-offs. Preference data labeled by small RMs using CIRM produced DPO-trained LLMs that rival those built with a 70B state-of-the-art RM. The analysis also shows that bias signals sit mostly in early layers.

Key points

  • Five bias types are targeted: length (character count), paragraphs (number of "\n\n"), query–response token overlap, "!" count, and "**" count.
  • The validation set is 500 RewardBench instances. For each bias, k is chosen from {50, …, 5000}, giving 16,807 joint configurations, searched with Optuna's Tree-structured Parzen Estimator over 100 trials.
  • The method edits under 2% of RM neurons. It is applied to the GRM and FsfairX models and compared against Length Penalty (α = 0.001) and LWR.
  • 2B and 7B RMs with CIRM, used for preference annotation, reach alignment comparable to a 70B RM on AlpacaEval 2.0 and MT-Bench. Unlike LP and LWR, CIRM did not amplify biases.
  • On TruthfulQA, CIRM improves truthfulness in 3 of 4 settings, for example Gemma-2-9B-it with GRM goes from 62.78 to 63.06.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NTTTokyo · Japan28 works · 5,199 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Kazutoshi Shinoda, Kosuke Nishida and Kyosuke Nishida. 2026. Debiasing Reward Models via Causally Motivated Inference-Time Intervention. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1029

@inproceedings{shinoda2026debiasing,
  title = {{Debiasing Reward Models via Causally Motivated Inference-Time Intervention}},
  author = {Kazutoshi Shinoda, Kosuke Nishida and Kyosuke Nishida},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1029},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1029}
}

References

40 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In KDD, page 2623–2631.
  2. Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  4. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In NIPS. find
  5. Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  6. Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. ODIN: Disentangled reward mitigates hacking in RLHF. In ICML. find
  7. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning. find
  8. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. RLHF workflow: From reward modeling to online RLHF. Transactions on Machine Learning Research.
  9. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling. find
  10. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D’Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2024. Helping or herding? reward model ensembles mitigate but do not elim… find
  11. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Zeyu Huang, Zihan Qiu, Zili Wang, Edoardo M Ponti, and Ivan Titov. 2024. Post-hoc reward calibration: A case study on length bias. arXiv preprint arXiv:2409.17407. — [arXiv:2409.17407](https://arxiv.org/abs/2409.17407) find
  14. Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. In NAACL, pages 6919–6971.
  15. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, pages 53728–53741.
  16. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. RewardBench: Evaluating reward models for language modeling. In Findings of NAACL, pages 1755–1797. find
  17. Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. WARM: On the benefits of weight averaged reward models. In ICML. find
  18. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, pages 3214–3252.
  19. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2025. RM-bench: Benchmarking reward models of language models with subtlety and style. In ICLR.
  20. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL, pages 3428–3448.
  21. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. find
  22. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS, pages 124198– 124235. find
  23. Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. 2024. InfoRM: Mitigating reward hacking in RLHF via information-theoretic reward modeling. In NeurIPS, pages 134387–134429. find
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  25. Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024a. OffsetBias: Leveraging debiased data for tuning evaluators. In Findings of EMNLP, pages 1043–1067.
  26. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024b. Disentangling length from quality in direct preference optimization. In Findings of ACL, pages 4998–5017.
  27. Judea Pearl. 2009. Causality: Models, Reasoning and Inference, 2nd edition. Cambridge University Press, USA. find
  28. Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2021. Can question generation debias question answering models? a case study on question–context lexical overlap. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 63–72. find
  29. Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2022. Look to the right: Mitigating relative position bias in extractive question answering. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 418–425. find
  30. Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2023. Which shortcut solution do question answering models prefer to learn? In AAAI, pages 13564– 13572. find
  31. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. A long way to go: Investigating length correlations in RLHF. In COLM.
  32. C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  33. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML, pages 3319–3328. find
  34. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance. In ACL, pages 8717–8729.
  35. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In NeurIPS, pages 12388–12401. find
  36. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2025. Helpsteer2-preference: Complementing ratings with preferences. In ICLR. find
  37. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer 2: Open-source dataset for training top-performing reward models. In NeurIPS, pages 1474–1501. find
  38. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for LLMs. In NeurIPS. find
  39. For RM-Bench, we followed the official score calculation process (Easy, Normal, Hard, and Overall) averaging over different domains as in this code. find
  40. Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. 2025. From lists to emojis: How format bias affects model alignment. In ACL, pages 26940–26961. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shinoda-2026-debiasing-reward-models-causally-motivated/