Elamur Research⌕
☰
home / papers / 2026 / DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs

DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs

Paper · 2026 · Bo Zhang, Jiawei Zhang, Cong Gao et al.

Summary

Large reasoning models solve math word problems well on clean text. When the input contains typos or similar character-level corruption, their accuracy falls, but their confidence stays high. The authors document this mismatch and introduce DisCal, a calibration method built specifically for noisy inputs.

They create noisy versions of math benchmarks by corrupting 30% of non-numeric words with one random edit type (deletion, insertion, substitution or swap). Numbers and symbols are left intact. Each question is sampled five times. The spread of the final boxed answers yields an entropy signal and a stability signal (the share held by the most frequent answer). Token likelihoods of the primary answer give an intrinsic confidence. A small calibrator with only three learned weights turns the two distributional signals into a gate that multiplicatively scales the intrinsic confidence.

Under noise, DisCal beats prior calibration approaches across several math benchmarks and models. It reduces ECE by as much as 39.21% and raises AUROC by up to 31.44%. Reliable confidence matters for real, messy user queries, where it can trigger abstention or review.

Key points

  • Motivating finding: with 30% character noise, DeepSeek-7B on GSM8K drops from 85.7% to 59.1% accuracy, while ECE grows from 9.7% to 35.8%. This indicates strong overconfidence.
  • DisCal combines three uncertainty signals:
    • distributional entropy and distributional stability, both taken from K = 5 sampled answers;
    • intrinsic confidence, computed as the geometric mean of answer-token likelihoods.
  • The calibrator learns just three global parameters (w_H, w_S, b). Final confidence is the intrinsic confidence multiplied by a learned gate g.
  • On AIME with DeepSeek-7B, the gate separates outcomes clearly: mean g is 0.643 for correct predictions and 0.143 for wrong ones. Going beyond K = 5 samples brings little extra benefit on GSM8K and SVAMP.
  • In one real typo-laden query, the model reached the right answer in only 3 of 5 samples. DisCal lowered its confidence from 0.9995 to 0.3852. The main experiments, however, use simulated rather than authentic noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shandong UniversityJinan · China19 works · 952 citations🎓Nankai UniversityTianjin · China36 works · 762 citations🔬PLA Academy of Military ScienceBeijing · China5 works · 510 citations

Also on the byline: PLA Rocket Force University of Engineering.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Bo Zhang, Jiawei Zhang, Cong Gao, Bingxu Han, Minghao Hu, Jun Zhang, Yunbo Cao, Zhunchen Luo, Wen Yao, Guotong Geng and Zhong Wang. 2026. DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.660

@inproceedings{zhang2026discal,
  title = {{DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs}},
  author = {Bo Zhang, Jiawei Zhang, Cong Gao, Bingxu Han, Minghao Hu, Jun Zhang, Yunbo Cao, Zhunchen Luo, Wen Yao, Guotong Geng and Zhong Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.660},
  url = {https://doi.org/10.18653/v1/2026.acl-long.660}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Mathematical Olympiad. 2025. AIMO validation AIME dataset. https://huggingface.co/ datasets/AI-MO/aimo-validation-aime. Licensed under Apache 2.0. ↗
  2. Amirhossein Aliakbarzadeh, Lucie Flek, and Akbar Karimi. 2025. Exploring robustness of multilingual llms on real-world noisy data. arXiv preprint arXiv:2501.08322. — [arXiv:2501.08322](https://arxiv.org/abs/2501.08322) find
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncertainty estimation for language models. In Proceedings of the 202… find
  5. Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, and 1 others. 2023. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589. find
  6. Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Yuren Hao, Xiang Wan, and Chengxiang Zhai. 2025. An investigation of robustness of llms in mathematical reasoning: Benchmarking with mathematically-equivalent transformation of advanced mathematical problems. arXiv preprint arXiv:2508.08833. find
  9. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. 2025. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. find
  10. Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, and 1 others. 2025. Math-perturb: Benchmarking llms’ math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.06453. find
  11. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  12. Neeraja Kirtane, Yuvraj Khanna, and Peter Relan. 2025. Mathrobust-lv: Evaluation of large language models’ robustness to linguistic variations in mathematical reasoning. arXiv preprint arXiv:2510.06430. find
  13. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.
  14. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. GSM-plus: A comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  15. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187. find
  16. Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. 2025. Calibrating large language models with sample consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19260– 19268. find
  17. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
  18. Yuhu Shang, Xiang Cheng, Yimeng Ren, Huijia Wu, Xuexiong Luo, Kangkang Lu, Jian Zhao, and Zhaofeng He. 2026. From chaos to cure: A prefix heuristics guided model-agnostic adaptive detoxification framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 32902–32910… find
  19. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  20. Safal Shrestha, Minwu Kim, and Keith Ross. 2025. Mathematical reasoning in large language models: Assessing logical and arithmetic errors across wide numerical ranges. arXiv preprint arXiv:2502.08680. find
  21. Linwei Tao, Haolan Guo, Minjing Dong, and Chang Xu. 2024. Consistency calibration: Improving uncertainty calibration via consistency among perturbed neighbors. arXiv preprint arXiv:2410.12295. — [arXiv:2410.12295](https://arxiv.org/abs/2410.12295) find
  22. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
  24. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025a. Qwen2.5 technical report. Preprint, arXiv:24… find
  25. Yuli Yang, Hiroaki Yamada, and Takenobu Tokunaga. 2025b. Evaluating robustness of llms to numerical variations in mathematical reasoning. In The Sixth Workshop on Insights from Negative Results in NLP, pages 171–180. find
  26. Tong Yu, Yongcheng Jing, Xikun Zhang, Wentao Jiang, Wenjie Wu, Yingjie Wang, Wenbin Hu, Bo Du, and Dacheng Tao. 2025a. Benchmarking reasoning robustness in large language models. arXiv preprint arXiv:2503.04550. — [arXiv:2503.04550](https://arxiv.org/abs/2503.04550) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-discal-distribution-aware-calibration-mathematical/