Elamur Research⌕
☰
home / papers / 2026 / Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting

Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting

Paper · 2026 · Cheng Wang, Qin Liu, Wenxuan Zhou et al.

Summary

GRPO is widely used to strengthen LLM reasoning, but it has trouble balancing exploration and exploitation, and its policy entropy becomes unstable during training. The authors build on prior theory (Cui et al., 2025) showing that entropy change follows the covariance between token log-probabilities and advantages. Fine-tuning DeepSeek-R1-Distill-Qwen-1.5B, they find that a tiny share of tokens with extreme covariance values dominates the updates. They propose CW-GRPO, which computes a per-token covariance score and scales each token's advantage with a Gaussian kernel. The kernel's bandwidth is set to the empirical standard deviation of the covariances, so the method needs no extra hyperparameters. Weights are normalized to preserve the expected loss. Training uses Open-RS (7k math problems), and evaluation covers five math benchmarks at the 1.5B and 7B scales. CW-GRPO achieves higher average pass@1 than vanilla GRPO and the Clip-Cov baseline, and it keeps entropy and checkpoint performance stable. This makes it a cheap, drop-in fix for RL fine-tuning.

Key points

  • Covariance concentration: At the top 0.01% of tokens, covariance magnitudes reach about 11.5 (positive) and −13.6 (negative). At the 100th-percentile cutoff, the values are only about 0.06 and −0.04.
  • 1.5B results (DeepSeek-R1-Distill-Qwen-1.5B): Average pass@1 is 55.3 for CW-GRPO, compared with 52.6 for GRPO, 53.6 for Clip-Cov and 48.9 for the base model. The largest gain is on AMC23 (77.5 vs 67.5). CW-GRPO trails GRPO on AIME24 (30.0 vs 33.3).
  • 7B results (Qwen2.5-Math-7B-Instruct): Average pass@1 is 46.7, compared with 44.1 for GRPO and 44.7 for Clip-Cov. The text states a +2.4 gain over GRPO, while the table implies about +2.6. The gains concentrate on AMC23 and OlympiadBench, and the method is slightly lower on Minerva.
  • Checkpoint stability: Across training checkpoints, vanilla GRPO falls on MATH-500 from 85.0 to 79.8 as entropy fluctuates. CW-GRPO stays between 86.2 and 87.0.
  • Setup and limitations: Training uses a sampling temperature of 0.7, and evaluation uses AIME24, MATH-500, AMC23, Minerva and OlympiadBench. Experiments are limited to math tasks and models of 7B parameters or fewer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓University of California, DavisDavis · United States28 works · 41,916 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Cheng Wang, Qin Liu, Wenxuan Zhou and Muhao Chen. 2026. Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.45

@inproceedings{wang2026taming,
  title = {{Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting}},
  author = {Cheng Wang, Qin Liu, Wenxuan Zhou and Muhao Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.45},
  url = {https://doi.org/10.18653/v1/2026.acl-short.45}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. 2025a. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. find
  2. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843– 3857.
  3. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. 2025b. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. find
  4. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, et al. 2025c. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. find
  5. Quy-Anh Dang and Chris Ngo. 2025. Reinforcement learning for reasoning in small llms: What works and what doesn’t. find
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
  7. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. Omni-math: A universal olympiad level ma…
  8. Daya Guo and DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9. find
  10. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. https://github.com/project-numina/ aimo-progress-prize/bl… ↗
  11. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  12. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.
  13. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  14. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. Rewarding progress: Scaling automated process verifiers for llm reasoning. find
  15. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024a. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-taming-extreme-tokens-covariance-aware/