Elamur Research⌕
☰
home / papers / 2026 / Calibration-Aware Policy Optimization for Reasoning LLMs

Calibration-Aware Policy Optimization for Reasoning LLMs

Paper · 2026 · Ziqi Wang, Xingzhou Lou, Meiqi Wu et al.

Summary

RL fine-tuning with verifiable rewards, especially GRPO, makes reasoning LLMs more accurate but also overconfident. Wrong answers can end up with lower perplexity than right ones, which hurts relative calibration as measured by AUC. The authors show this effect empirically and argue theoretically that it comes from GRPO's advantage estimation. Because that estimation looks only at rewards and ignores uncertainty, its gradients act as an inconsistent surrogate for AUC optimization.

They propose CAPO (Calibration-Aware Policy Optimization), which makes two changes:

  • It computes advantages from a logistic AUC surrogate that is provably consistent and has a regret bound, so the advantages account for model uncertainty.
  • It masks noisy samples using the reference model, which keeps training stable.

Trained on Qwen2.5-Math-1.5B and 7B and tested on six math benchmarks, CAPO substantially improves perplexity-based calibration while matching or beating GRPO accuracy. It also gives better precision–coverage trade-offs when the model is allowed to abstain, and higher accuracy under confidence-based inference-time scaling. This matters for reducing hallucinations and for any system that relies on model confidence.

Key points

  • Baselines: GRPO, GSPO, CoDaPO, CDE and SimKO. Calibration is measured by AUC-mean, the AUC averaged per question, with confidence taken from response perplexity.
  • Calibration gains: AUC-mean rises from 0.63 to 0.78 for the 1.5B model and from 0.54 to 0.79 for the 7B model. Accuracy (mean@8) stays on par with or above GRPO and is the best on AIME 2024, AIME 2025 and Minerva.
  • Other methods: competing calibration fixes yield small AUC gains or lose accuracy. SimKO drops about 12% on AMC and 7.7% on AIME 2024 compared with GRPO.
  • Downstream tasks: CAPO reaches about 5% higher accuracy than GRPO in inference-time scaling, using the Zhou et al. (2025b) algorithm. Its precision–coverage curves are Pareto-optimal against all baselines for both model sizes.
  • Ablations: removing noise masking causes entropy to grow and accuracy to stall. Adding masking alone to GRPO does not improve AUC. Results are robust to τ ∈ {0.4, 0.6, 1.0} and to the masking thresholds. Evaluation covers only mathematical reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China28 works · 5,973 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Ziqi Wang, Xingzhou Lou, Meiqi Wu, Zhengqi Wen and Junge Zhang. 2026. Calibration-Aware Policy Optimization for Reasoning LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.836

@inproceedings{wang2026calibration,
  title = {{Calibration-Aware Policy Optimization for Reasoning LLMs}},
  author = {Ziqi Wang, Xingzhou Lou, Meiqi Wu, Zhengqi Wen and Junge Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.836},
  url = {https://doi.org/10.18653/v1/2026.acl-long.836}
}

References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Bereket and Jure Leskovec. 2025. Uncalibrated reasoning: Grpo induces overconfidence for stochastic outcomes. arXiv preprint arXiv:2508.11800. find
  2. Toon Calders and Szymon Jaroszewicz. 2007. Efficient auc optimization for classification. In European Conference on Principles of Data Mining and Knowledge Discovery, pages 42–53. Springer. find
  3. Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. 2019. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning, pages 961–970. find
  4. Yu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, Xia Hu, and Helen Zhou. 2025. Learning to route llms with confidence tokens. Preprint, arXiv:2410.13284. — [arXiv:2410.13284](https://arxiv.org/abs/2410.13284) find
  5. Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, and 1 others. 2025. Cde: Curiosity-driven exploration for efficient reinforcement learning in large language models. arXiv preprint arXiv:2509.09675. — [arXiv:2509.09675](https://arxiv.org/abs/2509.09675) find
  6. Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. 2025. Beyond binary rewards: Training lms to reason about their uncertainty. arXiv preprint arXiv:2507.16806. find
  7. Qi Feng, Yihong Liu, and Hinrich Schütze. 2025. Your pretrained model tells the difficulty itself: A self-adaptive curriculum learning paradigm for natural language understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research W… find
  8. Wei Gao, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. 2013. One-pass auc optimization. In International Conference on Machine Learning, pages 906–914. PMLR. find
  9. Wei Gao and Zhi-Hua Zhou. 2012. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645. — [arXiv:1208.0645](https://arxiv.org/abs/1208.0645) find
  10. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  11. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting…
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  13. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  14. Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. 2025. Why language models hallucinate. arXiv preprint arXiv:2509.04664. — [arXiv:2509.04664](https://arxiv.org/abs/2509.04664) find
  15. Wojciech Kotlowski, Krzysztof J Dembczynski, and Eyke Huellermeier. 2011. Bipartite ranking through minimization of univariate loss. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1113–1120. find
  16. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857…
  17. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  18. Charles X Ling, Jin Huang, Harry Zhang, and 1 others. 2003. Auc: a statistically consistent and more discriminating measure than accuracy. In International Joint Conference on Artificial Intelligence, volume 3, pages 519–524. find
  19. Harder, Philip Torr, Jesper Ferkinghoff-Borg, Kaspar Martens, and Julien Fauqueur. 2025. Towards agents that know when they don’t know: Uncertainty as a control signal for structured reasoning. arXiv preprint arXiv:2509.02401. — [arXiv:2509.02401](https://arxiv.org/abs/2509.02401) find
  20. Haotian Liu, Shuo Wang, and Hongteng Xu. 2025a. C 2 GSPG: Confidence-calibrated group sequence policy gradient towards self-aware reasoning. arXiv preprint arXiv:2509.23129. — [arXiv:2509.23129](https://arxiv.org/abs/2509.23129) find
  21. Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. 2024. Quantifying uncertainty in natural language explanations of large language models. In International Conference on Artificial Intelligence and Statistics, pages 1072–1080. PMLR. find
  22. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025b. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  23. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, and 1 others. 2025a. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog. find
  24. Yichen Luo, Yebo Feng, Jiahua Xu, Paolo Tasca, and Yang Liu. 2025b. Llm-powered multi-agent system for automated crypto portfolio management. Preprint, arXiv:2501.00826. — [arXiv:2501.00826](https://arxiv.org/abs/2501.00826) find
  25. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9004– 9017. find
  26. Ruotian Peng, Yi Ren, Zhouliang Yu, Weiyang Liu, and Yandong Wen. 2025. Simko: Simple pass@ k policy optimization. arXiv preprint arXiv:2510.14807. — [arXiv:2510.14807](https://arxiv.org/abs/2510.14807) find
  27. Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. 2025. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics… find
  28. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning, pages 1889–1897. PMLR. find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Josefa Lia Stoisser, Marc Boubnovski Martell, Lawrence Phillips, Gianluca Mazzoni, Lea Mørch Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. When to trust llms: Aligning confidence with response quality. arXiv preprint arXiv:2404.… find
  32. Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shelmanov, and Maxim Panov. 2025. Uncertainty quantification for llms through minimum bayes risk: Bridging confidence and consistency. arXiv preprint arXiv:2502.04964. — [arXiv:2502.04964](https://arxiv.org/abs/2502.04964) find
  33. Xiaoxuan Wang, Yihe Deng, Mingyu Derek Ma, and Wei Wang. 2025. Entropy-based adaptive weighting for self-training. Preprint, arXiv:2503.23913. — [arXiv:2503.23913](https://arxiv.org/abs/2503.23913) find
  34. David Warren and Mark Dras. 2025. Bi-directional model cascading with proxy confidence. Preprint, arXiv:2504.19391. — [arXiv:2504.19391](https://arxiv.org/abs/2504.19391) find
  35. Bingbing Wen, Chenjun Xu, Robert Wolfe, Lucy Lu Wang, Bill Howe, and 1 others. 2024. Mitigating overconfidence in large language models: A behavioral lens on confidence estimation and calibration. In NeurIPS 2024 Workshop on Behavioral Machine Learning. find
  36. Jiancong Xiao, Bojian Hou, Zhanliang Wang, Ruochen Jin, Qi Long, Weijie J Su, and Li Shen. 2025. Restoring calibration for aligned large language models: A calibration-aware fine-tuning approach. arXiv preprint arXiv:2505.01997. — [arXiv:2505.01997](https://arxiv.org/abs/2505.01997) find
  37. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. Preprint, arXiv:2306.13063. find
  38. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024a. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.…
  39. Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024b. On verbalized confidence scores for llms. Preprint, arXiv:2412.14737. — [arXiv:2412.14737](https://arxiv.org/abs/2412.14737) find
  40. Tianbao Yang and Yiming Ying. 2022. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55(8):1–37. find
  41. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  42. Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. 2021. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3040–3049. find
  43. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
  44. Qingcheng Zeng, Weihao Xuan, Leyang Cui, and Rob Voigt. 2025. Thinking out loud: Do reasoning models know when they’re right? arXiv preprint arXiv:2504.06564.
  45. Peilin Zhao, Steven CH Hoi, Rong Jin, and Tianbo Yang. 2011. Online auc maximization. find
  46. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group sequence policy optimization. Preprint, arXiv:2507.18071.
  47. Zhanke Zhou, Xiangyu Lu, Chentao Cao, Brando Miranda, Tongliang Liu, Bo Han, and Sanmi Koyejo. 2025a. Codapo: Confidence and difficulty-adaptive policy optimization for post-training language models. In 2nd AI for Math Workshop@ ICML 2025. find
  48. Zhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao, Lan-Zhe Guo, Xiaoxing Ma, and Yu-Feng Li. 2025b. Bridging internal probability and self-consistency for effective and efficient llm reasoning. arXiv preprint arXiv:2502.00511. — [arXiv:2502.00511](https://arxiv.org/abs/2502.00511) find
  49. Dixian Zhu, Xiaodong Wu, and Tianbao Yang. 2022. Benchmarking deep auroc optimization: Loss functions and algorithmic choices. arXiv preprint arXiv:2203.14177. — [arXiv:2203.14177](https://arxiv.org/abs/2203.14177) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-calibration-aware-policy-optimization-reasoning/