Elamur Research⌕
☰
home / papers / 2026 / Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts

Paper · 2026 · Di Zhang, Xun Wu, Shaohan Huang et al.

Summary

The paper addresses why reinforcement learning with verifiable rewards (RLVR) often breaks down when the policy is a Mixture-of-Experts (MoE) model, frequently ending in sudden reward collapse. The authors trace this to router shift: between policy updates, the experts chosen for a token, and their routing probabilities, can change. This widens the off-policy gap. Training logs show routing stability degrading over time, alongside spiking importance ratios and bursts of clipping that come before collapse.

Their fix is Router-Shift Policy Optimization (RSPO), a drop-in change to GRPO-style objectives. For each token, it measures how much the current router's probability on the previously activated top-K experts has dropped, averaged across MoE layers. This becomes a weight in (0, 1], which is detached from the gradient, given a minimum floor, and multiplied into the importance ratio before clipping and aggregation. Tokens with strongly shifted routing therefore count less, and the router is never frozen or forced to replay old decisions.

Experiments on Countdown (Qwen2.5-MoE) and math/code reasoning (Qwen3-30B-A3B) show more stable training and better results than GRPO, GSPO and GMPO. This matters for anyone post-training sparse models with RL.

Key points

  • Router-shift ratio: γ = exp(−Δ), where Δ is the average log-probability drop on the old top-K experts across layers. The floor γ_min is set to 0.8, and the share of tokens falling below the floor is tracked as a diagnostic.
  • Plug-in stabilizer: On Qwen2.5-MoE with the Countdown task, adding router-shift weighting stabilizes GRPO, GSPO and GMPO alike. The main large-scale variant is RSPO = GMPO + RS on Qwen3-30B-A3B, which improves Pass@1 on both math and code benchmarks.
  • Rigid alternatives fall short: Freezing the router or replaying routing decisions performed worse in the authors' tests. Freezing limits how the router adapts to the RL objective, and replay restricts router exploration.
  • Overhead: Caching the old routing statistics costs about 20.8% of training throughput compared with GMPO on Qwen3-30B-A3B.
  • Limitations: Large-scale evaluation covers mainly one MoE architecture and only rule-based rewards. Interaction with MoE load-balancing losses is not studied.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui and Furu Wei. 2026. Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1165

@inproceedings{zhang2026stable,
  title = {{Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts}},
  author = {Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui and Furu Wei},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1165},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1165}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, and 1 others. 2025. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585.
  2. Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Stablemoe: Stable routing strategy for mixture of experts. arXiv preprint arXiv:2204.08396. find
  3. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  4. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  5. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  6. Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, and 1 others. 2024. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37:19209– 19253. find
  7. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857…
  8. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  9. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, and 1 others. 2025. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog. find
  10. OpenAI. 2024. Learning to reason with llms. find
  11. Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. 2023. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951. — [arXiv:2308.00951](https://arxiv.org/abs/2308.00951) find
  12. Tian Qin, David Alvarez-Melis, Samy Jelassi, and Eran Malach. 2025. To backtrack or not to backtrack: When sequential search limits model reasoning. arXiv preprint arXiv:2504.07052. — [arXiv:2504.07052](https://arxiv.org/abs/2504.07052) find
  13. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  14. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  15. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025a. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534.
  16. Ling Team, Bin Hu, Cai Chen, Deng Zhao, Ding Liu, Dingnan Jin, Feng Zhu, Hao Dai, Hongzhi Luan, Jia Guo, and 1 others. 2025b. Ring-lite: Scalable reasoning via c3po-stabilized reinforcement learning for llms. arXiv preprint arXiv:2506.14731.
  17. Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. 2024a. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664. — [arXiv:2408.15664](https://arxiv.org/abs/2408.15664) find
  18. Ziteng Wang, Jun Zhu, and Jianfei Chen. 2024b. Re-moe: Fully differentiable mixture-of-experts with relu routing. arXiv preprint arXiv:2412.14711. — [arXiv:2412.14711](https://arxiv.org/abs/2412.14711) find
  19. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  20. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  21. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, and 1 others. 2025. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. — [arXiv:2507.20673](https://arxiv.org/abs/2507.20673) find
  22. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 others. 2025. Group sequence policy optimization. arXiv preprint arXiv:2507.18071.
  23. Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, and 1 others. 2022. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-stable-effective-reinforcement-learning-mixture/