Elamur Research⌕
☰
home / papers / 2026 / BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Paper · 2026 · Saket Reddy, Ke Yang and ChengXiang Zhai

Summary

Reducing social bias in LLMs is hard to optimise because no single "correct" unbiased answer exists, so reward signals are noisy and subjective. Existing preference-based approaches each have weaknesses. DPO (tested here in its IPO variant) learns offline from fixed pairs and generalises poorly. PPO explores online but depends on a critic whose value estimates become unreliable in such noisy settings.

The authors propose BiasGRPO, which applies DeepSeek's Group Relative Policy Optimization to debiasing. The pipeline has three parts:

  • the base GRPO algorithm, which replaces the critic with rewards normalised within a group of sampled completions;
  • a curated, synthetically extended dataset;
  • a compact bias-specific reward model.

Because rewards are compared within a group, the policy still receives a learning signal when every completion is biased: the least biased one gets a positive advantage. On Microsoft's Phi-2, BiasGRPO beats DPO and PPO across bias benchmarks while keeping TruthfulQA performance, and it largely carries over to Llama 3.2 (3B). The released resources give RLHF pipelines a cheap, ready-made way to add bias mitigation.

Key points

  • Released resources: a dataset covering 11 domains and a 0.1B-parameter bias reward model, both on Hugging Face. The reward model is meant to plug into multi-objective RLHF without extra compute or loss of model knowledge.
  • Evaluation: benchmarks were BOLD, RealToxicityPrompts, BBQ and TruthfulQA, with a default group size of G = 4.
  • PPO instability: during PPO training, critic value loss fell only from 0.2507 to 0.2134. PPO reward standard deviation dropped only from 0.1703 to 0.1279.
  • Group-size ablation (Phi-2): larger groups helped consistently across benchmarks.
    • BOLD (overall): .0293 (base) → .0140 (G = 4) → .0124 (G = 8)
    • BBQ (overall): .2750 → .3123 → .3781
    • TruthfulQA: .3843 → .3941 → .4137
    • G = 2 clearly lagged behind G = 4, as did Online DPO. The authors take this as evidence that group-relative normalisation, not online training alone, drives the gains.
  • Robustness and generalisation:
    • Swapping in an alternative stereotype reward model still gave large improvements over the base model, though the custom reward model was better on most benchmarks.
    • On Llama 3.2 (3B), GRPO led on all benchmarks except BBQ, where every method hurt performance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Saket Reddy, Ke Yang and ChengXiang Zhai. 2026. BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2052

@inproceedings{reddy2026biasgrpo,
  title = {{BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization}},
  author = {Saket Reddy, Ke Yang and ChengXiang Zhai},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2052},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2052}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Allam. 2024. BiasDPO: Mitigating bias in language models through direct preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 42–50. find
  2. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistic…
  3. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. arXiv preprint arXiv:1903.04561. — [arXiv:1903.04561](https://arxiv.org/abs/1903.04561) find
  4. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872… find
  5. Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. 2022. Reward modeling for mitigating toxicity in transformer-based language models. Applied Intelligence, pages 1–16. find
  6. Dan Friedman and Adji Bousso Dieng. 2023. The Vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research. find
  7. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097– 1179.
  8. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369.
  9. Jan Hauke and Tomasz Kossowski. 2011. Comparison of values of Pearson’s and Spearman’s correlation coefficients on the same sets of data. Quaestiones Geographicae, 30(2):87–93. find
  10. Chenghua Huang, Lu Wang, Fangkai Yang, Pu Zhao, Zhixu Li, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2025. Lean and mean: Decoupled value policy optimization with global value guidance. arXiv preprint arXiv:2502.16944. — [arXiv:2502.16944](https://arxiv.org/abs/2502.16944) find
  11. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. 2025. Writing-Zero: Bridge the gap between non-verifiable tasks and verifiable rewards. arXiv preprint arXiv:2506.00103. — [arXiv:2506.00103](https://arxiv.org/abs/2506.00103) find
  12. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489.
  13. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need II: Phi-1.5 technical report. arXiv preprint arXiv:2309.05463.
  14. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252.
  15. Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, and Chen Huang. 2024. On the limited generalization capability of the implicit reward model induced by direct preference optimization. In Findings of the Association for Computational Linguistic… find
  16. Yang Liu. 2024. Quantifying stereotypes in language. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1223– 1240. find
  17. Yiran Liu, Ke Yang, Zehan Qi, Xiao Liu, Yang Yu, and ChengXiang Zhai. 2025. Bias and volatility: A statistical framework for evaluating large language model’s stereotypes and the associated generation inconsistency. Preprint, arXiv:2402.15481. — [arXiv:2402.15481](https://arxiv.org/abs/2402.15481) find
  18. Jordan J Louviere, Terry N Flynn, and Anthony AJ Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  19. John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations. find
  20. Lucas Maystre and Matthias Grossglauser. 2015. Fast and accurate inference of Plackett-Luce models. In Advances in Neural Information Processing Systems, volume 28. find
  21. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  22. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  23. Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  24. Meta. 2024. Llama 3.2 Model Card. find
  25. Yixuan Tang and Yi Yang. 2025. SR-GRPO: Stable rank Https://github.com/meta-llama/llama-as an intrinsic geometric reward for large language models/blob/main/models/llama3_2/MODEL_CARD.md. model alignment. arXiv preprint arXiv:2512.02807. — [arXiv:2512.02807](https://arxiv.org/abs/2512.02807) find
  26. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. find
  27. Valurank. 2022. distilroberta-bias. StereoSet: Measuring stereotypical bias in pretrained https://huggingface.co/valurank/ language models. In Proceedings of the 59th Annual distilroberta-bias. Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Berti… ↗
  28. Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: Origins, inventory and discussion. ACM Journal of Data and Information Quality. find
  29. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83. find
  30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pag…
  31. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105.
  32. Perspective API. 2017. Perspective API: Attributes and Languages. https://developers.perspectiveapi.com/s/ about-the-api-attributes-and-languages? language=en_US. ↗
  33. Yihong Wu, Liheng Ma, Lei Ding, Muzhi Li, Xinyu Wang, Kejia Chen, Zhan Su, Zhanguang Zhang, Chenyang Huang, Yingxue Zhang, Mark Coates, and Jian-Yun Nie. 2025. It takes two: Your GRPO is secretly DPO. In Advances in Neural Information Processing Systems. find
  34. Xuan Xie, Xuan Wang, and Wenjie Wang. 2025. Da-GRPO: Rectifying gradient conflict in reasoning via distinctiveness-aware group relative policy optimization. arXiv preprint arXiv:2512.06337. — [arXiv:2512.06337](https://arxiv.org/abs/2512.06337) find
  35. Shaokun Zhang. 2024. bias-detection. https://huggingface.co/D1V1DE/bias-detection. ↗
  36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volu…
  37. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/reddy-2026-biasgrpo-stabilizing-bias-mitigation-high/