Elamur Research⌕
☰
home / papers / 2026 / MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

Paper · 2026 · Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili et al.

Summary

The paper tackles a weakness of GRPO when training LLMs to follow prompts that bundle several verifiable constraints, such as formatting, keywords, casing or exact endings. Rule-based checkers produce discrete rewards with little spread, so z-score group normalization fails in three ways. Low-variance amplification blows small reward gaps up into large advantages. Mean-centering blindness hides how good a group is in absolute terms. Zero-variance collapse leaves homogeneous groups with no gradient at all. The proposed MDP-GRPO combines four fixes: sampling at multiple temperatures to diversify groups, dual-anchor advantages that blend group-relative and goal-aware signals, bounded loss-averse shaping inspired by Prospect Theory, and asymmetric KL regularization. The authors also build a 3,000-prompt multi-constraint training set. They test on Gemma-2-2B and Llama-3.2-3B using IFEval, FollowBench and a custom test set. MDP-GRPO variants beat standard GRPO, with the largest gains on harder prompts, while scores on MMLU and ARC are largely retained. The work matters because it makes critic-free RLVR more stable and cheaper, including with small group sizes.

Key points

  • Dataset: 3,000 English training prompts.
    • About a third of the seed prompts come from He et al. (2024).
    • Each prompt receives 1–6 injected constraints drawn from 26 types in 9 categories.
    • Every constraint is checked by deterministic validators (regexes and parsers).
    • The reward is the share of constraints satisfied.
  • Headline results:
    • Strict constraint satisfaction rises by up to 5.0% on Llama-3.2-3B over GRPO.
    • DA-GRPO lifts Gemma-2-2B strict success from 29.0% to 32.6%.
  • Small groups (G = 4): multi-temperature sampling helps most in this regime.
    • On Gemma-2-2B, MT-GRPO gains +1.4/+1.3 (SSR/HSR) on IFEval.
    • On the custom set, its HSR of 30.6% exceeds G = 8 GRPO's 29.0%.
  • Training diagnostics:
    • Dual anchoring reduces the share of groups with zero reward variance.
    • Prospect-theoretic shaping keeps KL drift in check.
    • Gains concentrate on format, length and punctuation constraints rather than knowledge-dependent ones such as language.
  • Limitations:
    • Only 2B and 3B models are tested.
    • The setup is English-only.
    • It relies on objective checkers.
    • Several hyperparameters may need retuning for new domains.
    • Code is released at github.com/m-salmani78/MDP-GRPO.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TehranTehran · Iran9 works · 182 citations🎓Allameh Tabataba'i UniversityTehran · Iran3 works · 21 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili and Mohammad Javad Dousti. 2026. MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1982

@inproceedings{salmanizarchi2026mdp,
  title = {{MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following}},
  author = {Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili and Mohammad Javad Dousti},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1982},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1982}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the Judge? A Study on Judgement Bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  2. Xin Chen, Yarden As, and Andreas Krause. 2025. Learning Safety Constraints for Large Language Models. arXiv preprint arXiv:2505.24445. — [arXiv:2505.24445](https://arxiv.org/abs/2505.24445) find
  3. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research. find
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457.
  5. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model Alignment as Prospect Theoretic Optimization. In Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR. find
  6. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in llms through rein… find
  8. Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024. From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA. Association for Com… find
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300.
  10. Wenke Huang, Quan Zhang, Yiyang Fang, Jian Liang, Xuankun Rong, Huanjin Yao, Guancheng Wan, Ke Liang, Wenwen He, Mingjun Li, Leszek Rutkowski, Mang Ye, Bo Du, and Dacheng Tao. 2025. MAPO: Mixed Advantage Policy Optimization. arXiv preprint arXiv:2509.18849. — [arXiv:2509.18849](https://arxiv.org/abs/2509.18849) find
  11. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computa…
  12. Daniel Kahneman and Amos Tversky. 1979. Prospect Theory: An Analysis of Decision under Risk. Econometrica. find
  13. Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. 2024. Understanding Catastrophic Forgetting in Language Model Fine-tuning. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models. find
  14. Wenyun Li, Wenjie Huang, and Chen Sun. 2025. Shaping Sparse Rewards in Reinforcement Learning. arXiv preprint arXiv:2501.19128. — [arXiv:2501.19128](https://arxiv.org/abs/2501.19128) find
  15. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. Mitigating the Alignment Tax of RLHF. In Proceedings of the 2024 Conference on Empirical Me…
  16. Wenhao Liu, Zhengkang Guo, Mingchen Xie, Jingwen Xu, Zisu Huang, Muzhao Tian, Jianhan Xu, Muling Wu, Xiaohua Wang, Changze Lv, He-Da Wang, Hu Yao, Xiaoqing Zheng, and Xuanjing Huang. 2025. RECAST: Expanding the boundaries of LLMs’ complex instruction following with multi-constraint data. In Submitte… find
  17. Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, and 1 others. 2025. NGRPO: Negative-enhanced group relative policy optimization. arXiv preprint arXiv:2509.18851. — [arXiv:2509.18851](https://arxiv.org/abs/2509.18851) find
  18. Savinay Narendra, Kaushal Shetty, and Adwait Ratnaparkhi. 2024. Enhancing Contract Negotiations with LLM-Based Legal Document Comparison. In Natural Legal Language Processing Workshop. Association for Computational Linguistics. find
  19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems.
  20. Hannes Westermann. 2024. Dallma: Semi-Structured Legal Reasoning and Drafting with Document Automation and LLM Assistance. In ICML Workshop (GenLaw). find
  21. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. In The eleventh international conference on learning representations.
  22. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2025. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. In The Thirteenth International Conference on Learning Representations.
  23. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-Following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  24. Kanghee Park, Timothy Zhou, and Loris D’Antoni. 2025. Flexible and Efficient Grammar-Constrained Decoding. arXiv preprint arXiv:2502.05111. — [arXiv:2502.05111](https://arxiv.org/abs/2502.05111) find
  25. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems.
  26. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  27. Zhengyuan Shen, Darren Yow-Bang Wang, Soumya Smruti Mishra, Zhichao Xu, Yifei Teng, and Haibo Ding. 2025. SLOT: Structuring the Output of Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Li… find
  28. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press. find
  29. Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, and 1 others. 2025. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs. arXiv preprint arXiv:2506.14245. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zarchi-2026-mdp-grpo-stabilized-group-relative/