Elamur Research⌕
☰
home / papers / 2026 / MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

Paper · 2026 · Jingwei Song, Xinyu Wang, Hanbin Wang et al.

Summary

Speculative decoding speeds up LLM inference: a small drafter proposes tokens and the large target model checks them in parallel. Drafting has improved a lot, but verification still demands exact agreement with the target. The authors argue this wastes effort when the target barely prefers its top token over the runner-up. In those cases, rejecting the draft forces a costly rollback for almost no quality gain.

MARS (Margin-Aware Speculative Verification) measures how decisive the target is using the ratio of its second-highest to highest raw logit. When that ratio exceeds a threshold (θ = 0.9 by default), a draft matching either of the top two tokens is accepted. Otherwise, standard verification applies. The method needs no training or auxiliary judge model and changes only the accept/reject rule. It works with EAGLE-3 and vanilla speculative decoding, for chain and tree drafts and for greedy or sampled decoding.

Across target models from 8B to 235B, MARS gives consistent speedups with near-lossless quality. This makes it a cheap, plug-in improvement for efficient inference. Note that it is a lossy scheme, so it no longer guarantees the target model's exact output distribution.

Key points

  • Why logits instead of probabilities: the confidence measure is the logit ratio r = z₂/z₁. Softmax exaggerates gaps in an exponential, scale-dependent way, so probability ratios can overstate how confident the model is.
  • Headline result: the maximum reported speedup is 4.76× on LLaMA-3.1-70B, evaluated on chat, coding and reasoning benchmarks.
  • Threshold sensitivity: on WMT19 Zh-En (BLEU/chrF), quality stays near baseline for θ ≥ 0.90 but drops noticeably below 0.88. For example, Qwen3-32B reaches 29.76 BLEU at θ = 0.90 versus 30.01 for the baseline.
  • Plugged into standard speculative decoding (SPD), Qwen3-0.6B→Qwen3-32B on GSM8K: speedup rises from 2.15× to 2.70× and average accepted length τ from 4.23 to 4.94, with accuracy kept at 0.901.
  • Same setup, LLaMA-3.1-8B→70B on HumanEval: speedup rises from 3.15× to 4.36× and accuracy goes from 0.712 to 0.732.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang and Lynn Ai. 2026. MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.421

@inproceedings{song2026mars,
  title = {{MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification}},
  author = {Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang and Lynn Ai},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.421},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.421}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109. — [arXiv:2402.05109](https://arxiv.org/abs/2402.05109) find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Gregor Bachmann and Dan Alistarh. 2025. Judge decoding: Faster speculative sampling requires going beyond model alignment. In The Thirteenth International Conference on Learning Representations (ICLR). Oral Presentation. find
  4. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.
  5. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  7. Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2025. Sequoia: Scalable, robust, and hardware-aware speculative decoding. Preprint, arXiv:2402.12374. find
  8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  10. Wikimedia Foundation. 2019. find
  11. Acl 2019 fourth conference on machine translation (wmt19), shared task: Machine translation of news. find
  12. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. Preprint, arXiv:2402.02057.
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  14. Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D. Lee, and Di He. 2024. Rest: Retrieval-based speculative decoding. Preprint, arXiv:2311.08252. find
  15. Karl Moritz Hermann, Tomáš Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701.
  16. Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. 2023. Speculative decoding with big little decoder. Preprint, arXiv:2302.07863. find
  17. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. Preprint, arXiv:2211.17192. find
  18. Jinze Li, Yixing Xu, Guanchen Li, and 1 others. 2025a. Training-free loosely speculative decoding: Accepting semantically correct drafts beyond exact match. arXiv preprint arXiv:2511.22972. — [arXiv:2511.22972](https://arxiv.org/abs/2511.22972) find
  19. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle-2: Faster inference of language models with dynamic draft trees. Preprint, arXiv:2406.16858.
  20. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025b. Eagle-3: Scaling up inference acceleration of large language models via training-time test. Preprint, arXiv:2503.01840. — [arXiv:2503.01840](https://arxiv.org/abs/2503.01840) find
  21. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025c. Eagle: Speculative sampling requires rethinking feature uncertainty. Preprint, arXiv:2401.15077.
  22. Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Online speculative decoding. Preprint, arXiv:2310.07177. find
  23. Xupeng Miao, Gabriele Oliaro, Zhihao He, Aaron Schulze, Cheng-Hao Parisot, Adam Paszke, and Zhihao Jia. 2024. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectur… find
  24. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. find
  25. Shwetha Somasundaram, Anirudh Phukan, and Apoorv Saxena. 2024. Pld+: Accelerating llm inference by leveraging language model artifacts. Preprint, arXiv:2412.01447. find
  26. Benjamin Spector and Chris Re. 2023. Accelerating llm inference with staged speculative decoding. Preprint, arXiv:2308.04623. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) find
  27. Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2024. Spectr: Fast speculative decoding via optimal transport. Preprint, arXiv:2310.15141. — [arXiv:2310.15141](https://arxiv.org/abs/2310.15141) find
  28. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  29. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  30. Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu. 2025. Draft model knows when to stop: Self-verification speculative decoding for long-form generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16696–16708,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2026-mars-unleashing-power-speculative-decoding/