Elamur Research⌕
☰
home / papers / 2026 / Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Paper · 2026 · Zhouxuwen, Fangxin Liu, Chao Wang et al.

Summary

Speculative decoding speeds up LLM inference. A small draft model proposes tokens, and a large target model checks them in parallel. Its exact token-level verification, however, discards draft tokens that are correct in meaning but differ in surface form (e.g., ‘x’ vs ‘*’, ‘a’ vs ‘the’). The authors call this a Semantics-Alignment Mismatch. To address it, they propose Calibrated Speculative Decoding (CSD), a training-free add-on to rejection-sampling verification.

They first analysed rejections between LLaMA-3.2-1B-Instruct and LLaMA-3-70B-Instruct. Rejected pairs recur in a heavily skewed way, yet the same pair can be harmless or wrong depending on context. CSD therefore combines two modules:

  • Online Correction Memory tracks past rejected draft–target pairs and offers frequent ones as rescue candidates.
  • Semantic Consistency Gating admits a candidate only if the target model's probability ratio clears a threshold τ.

Across several model pairs and benchmarks, CSD reaches up to 2.33× speedup. Accuracy is preserved and even improves on reasoning tasks, with negligible added cost. This makes CSD a practical, low-effort upgrade for LLM serving.

Key points

  • Rejection analysis: The authors studied 28,000 samples from CNN/DailyMail and MATH. The 20% most frequent rejection patterns cover about 69.0% of all rejections. For a single token pair, the target probability ratio varies from near 1 down to below 10⁻¹⁰.
  • Results: Peak speedup is 2.33×. HumanEval accuracy climbs 2.5 points, while MATH500 accuracy gains 2.0 points. On MATH500, the acceptance rate grows from 63.6% to 79.6%.
  • Ablation: Using only one module (OCM or SCG with τ = 0.01) speeds decoding up but lowers accuracy. Both modules together are needed for safe acceleration. Best settings are τ = 0.01 and λ = 6.
  • Calibration: A Universal Calibration Set of 6,000 RedPajama sequences keeps the average speedup at 2.02×, equal to task-specific calibration. Even without initial calibration data, the acceptance rate improves by about 7% thanks to online updates.
  • Comparison and overhead: CSD reaches 1.89× speedup versus 1.76× for Reflective Verification. Its runtime overhead is only 0.01–0.02% of total latency (Qwen-72B/7B, Llama-70B/1B).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Zhouxuwen, Fangxin Liu, Chao Wang, Xiao Zheng, Hao Zheng, Min He, Li Jiang and Haibing Guan. 2026. Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1369

@inproceedings{zhouxuwen2026calibrated,
  title = {{Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference}},
  author = {Zhouxuwen, Fangxin Liu, Chao Wang, Xiao Zheng, Hao Zheng, Min He, Li Jiang and Haibing Guan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1369},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1369}
}

References

37 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. 2025. Judge decoding: Faster speculative sampling requires going beyond model alignment. arXiv preprint arXiv:2501.19309. find
  4. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.
  5. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  6. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  7. Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. 2025. Towards high-goodput llm serving with prefill-decode multiplexing. arXiv preprint arXiv:2504.14489. find
  8. Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2024. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374. find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  10. Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and 1 others. 2024. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint arXiv:2402.02082. — [arXiv:2402.02082](https://arxiv.org/abs/2402.02082) find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  12. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, and 1 others. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Associa…
  13. Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. 2025. R2r: Efficiently navigating divergent reasoning paths with small-large model token routing. arXiv preprint arXiv:2505.21600. — [arXiv:2505.21600](https://arxiv.org/abs/2505.21600) find
  14. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.
  15. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  16. Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. 2024. Ai and memory wall. IEEE Micro, 44(3):33– 39. find
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  18. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
  20. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.
  21. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. find
  22. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857…
  23. Jinze Li, Yixing Xu, Guanchen Li, Shuo Yang, Jinfeng Xu, Xuanwu Yin, Dong Li, Edith CH Ngai, and Emad Barsoum. 2025. Training-free loosely speculative decoding: Accepting semantically correct drafts beyond exact match. arXiv preprint arXiv:2511.22972. — [arXiv:2511.22972](https://arxiv.org/abs/2511.22972) find
  24. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077.
  25. Fangxin Liu, Ning Yang, Haomin Li, Zongwu Wang, Zhuoran Song, Songwen Pei, and Li Jiang. 2024a. Spark: Scalable and precision-aware acceleration of neural networks via efficient encoding. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 1029–1042. IEEE. find
  26. Fangxin Liu, Wenbo Zhao, Zhezhi He, Yanzhi Wang, Zongwu Wang, Changzhi Dai, Xiaoyao Liang, and Li Jiang. 2021. Improving neural network efficiency via post-training quantization with adaptive floating-point. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5281– 5290… find
  27. Jiahao Liu, Qifan Wang, Jingang Wang, and Xunliang Cai. 2024b. Speculative decoding via early-exiting for faster llm inference with thompson sampling control mechanism. arXiv preprint arXiv:2406.03853. find
  28. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, and 1 others. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM… find
  29. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  30. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of machine learning and systems, 5:606–624. find
  31. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  32. Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. 2023. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427. find
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  34. Yixuan Wang, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che, and 1 others. 2025. Think before you accept: Semantic reflective verification for faster speculative decoding. arXiv preprint arXiv:2505.18629. — [arXiv:2505.18629](https://arxiv.org/abs/2505.18629) find
  35. Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. Redpajama: an open da…
  36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural lan…
  37. A Extended Evaluation on Instruction Following and Long-Context Tasks To further demonstrate the versatility of CSD across highly complex scenarios, we conducted additional evaluations on IFEval (Zhou et al., 2023a) (measuring strict instruction-following capabilities) and RULER (Hsieh et al., 2024)… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhouxuwen-2026-calibrated-speculative-decoding-frequency-guided/