Elamur Research⌕
☰
home / papers / 2026 / Training-Free Adaptive Speculative Decoding via Linguistic Priors

Training-Free Adaptive Speculative Decoding via Linguistic Priors

Paper · 2026 · Jingyi Wang, Jiaqi Huang, Zunnan Xu et al.

Summary

Speculative decoding speeds up LLM inference: a small drafter proposes tokens and a large target model checks them in parallel. Existing schemes apply one uncertainty criterion to every token, even though nouns are naturally less predictable than punctuation. LinguaSpec, from Tsinghua University, adds linguistic knowledge to this loop without any training. A precomputed lookup table, the Static Linguistic Probe (SLP), assigns each vocabulary token to one of four coarse POS-based classes at no runtime cost. Syntactic Normalized Surprisal (SNS) then divides a token's raw surprisal by the expected entropy of its class, so content words are not penalized for inherent variability. When SNS is low, the draft tree is grown deeper. The deferred verification window is also set per class. The priors are calibrated on WikiText-2. Against EAGLE-3, the method reaches up to 5.30× speedup on LLaMA-3.3-70B-Instruct while preserving output quality. Because it is plug-and-play, it is attractive for latency-sensitive deployment.

Key points

  • LLaMA vocabulary breakdown: Nouns 28.78%, Proper Nouns 25.72%, Punctuation 15.18%, and other function words (particles, conjunctions) 4.74%.
  • Elastic expansion ablation (GSM8K): dynamic expansion with K = 4 and stability threshold γ = −1.0 gives 4.39× speedup (τ = 6.99). A static +4 depth increase yields longer drafts (τ = 7.44) but only 3.93× speedup, because drafting takes 54.6 ms vs. 46.7 ms.
  • Verification window ablation (HumanEval):
    • Uniform windows trade quality for speed: W = 2 gives 4.65× and 65.1% Pass@1, while W = 4 gives 4.15× and 68.4%.
    • The POS-adaptive setting reaches 4.46× and 68.3%.
  • Inconsistency on which class gets strict checks: the introduction and ablation say content words get strict checks (W = 4) and structural tokens get relaxed checks (W = 2). The Figure 2 caption states the reverse.
  • Models and limitations: models include LLaMA-3.1-8B, LLaMA-3.3-70B and a DeepSeek-distilled 8B model. Stated limitations are that priors may not transfer to specialized domains or typologically different languages, coarse POS classes miss finer syntax, and there is a small risk of semantic drift.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jingyi Wang, Jiaqi Huang, Zunnan Xu, Jun Zhou, Kehong Yuan and Xiang Qian. 2026. Training-Free Adaptive Speculative Decoding via Linguistic Priors. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1065

@inproceedings{wang2026training,
  title = {{Training-Free Adaptive Speculative Decoding via Linguistic Priors}},
  author = {Jingyi Wang, Jiaqi Huang, Zunnan Xu, Jun Zhou, Kehong Yuan and Xiang Qian},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1065},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1065}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, and 1 others. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  2. Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109. — [arXiv:2402.05109](https://arxiv.org/abs/2402.05109) find
  3. Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. 2025. Judge decoding: Faster speculative sampling requires going beyond model alignment. arXiv preprint arXiv:2501.19309. find
  4. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning (ICML).
  5. Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. 2025. Acpbench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26559–26568. find
  6. Charlie Chen and 1 others. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  7. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. find
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  9. Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith CH Ngai, and Emad Barsoum. 2025a. Gumiho: A hybrid architecture to prioritize early tokens in speculative decoding. arXiv preprint arXiv:2503.10135. — [arXiv:2503.10135](https://arxiv.org/abs/2503.10135) find
  10. DeepSeek-AI. 2025. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Keshwam, Ahmad Al-dahle, Adithya Renduchintala, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Koumchatzky, and Jelena Yangel. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710.
  13. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  15. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4129–4138.
  16. Kaixuan Huang, Xudong Guo, and Mengdi Wang. 2024. Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715. — [arXiv:2405.19715](https://arxiv.org/abs/2405.19715) find
  17. Feiye Huo, Jianchao Tan, Kefeng Zhang, Xunliang Cai, and Shengli Sun. 2025. C2t: A classifier-based tree construction method in speculative decoding. arXiv preprint arXiv:2502.13652. — [arXiv:2502.13652](https://arxiv.org/abs/2502.13652) find
  18. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657.
  19. Jinze Li, Yixing Xu, Guanchen Li, Shuo Yang, Jinfeng Xu, Xuanwu Yin, Dong Li, Edith CH Ngai, and Emad Barsoum. 2025b. Training-free loosely speculative decoding: Accepting semantically correct drafts beyond exact match. arXiv preprint arXiv:2511.22972. — [arXiv:2511.22972](https://arxiv.org/abs/2511.22972) find
  20. Jinze Li, Yixing Xu, Guanchen Li, Shuo Yang, Jinfeng Xu, Xuanwu Yin, Dong Li, Edith CH Ngai, and Emad Barsoum. 2025c. Training-free loosely speculative decoding: Accepting semantically correct drafts beyond exact match. arXiv preprint arXiv:2511.22972. — [arXiv:2511.22972](https://arxiv.org/abs/2511.22972) find
  21. Yuhui Li, Fangyun Wei, Chamara Zhang, and Hongyang Zhang. 2024a. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858.
  22. Yuhui Li, Fangyun Wei, Chamara Zhang, and Hongyang Zhang. 2024b. Eagle: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning (ICML).
  23. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025d. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840. — [arXiv:2503.01840](https://arxiv.org/abs/2503.01840) find
  24. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542. — [arXiv:2408.15542](https://arxiv.org/abs/2408.15542) find
  25. Siran Liu, Yang Ye, Qianchao Zhu, Zane Cao, and Yongchao He. 2025. Heterospec: Leveraging contextual heterogeneity for efficient speculative decoding. arXiv preprint arXiv:2505.13254. find
  26. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, and 1 others. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM… find
  27. Shuo Roy and 1 others. 2023. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264. find
  28. Richard Shin, Christopher H Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural… find
  29. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912. find
  30. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  31. Wangchunshu Zhou and 1 others. 2023. Controlled text generation with natural language instructions. arXiv preprint arXiv:2304.13779. — [arXiv:2304.13779](https://arxiv.org/abs/2304.13779) find
  32. Matthieu Zimmer, Milan Gritta, Gerasimos Lampouras, Haitham Bou Ammar, and Jun Wang. 2024. Mixture of attentions for speculative decoding. arXiv preprint arXiv:2410.03804. — [arXiv:2410.03804](https://arxiv.org/abs/2410.03804) find
  33. Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. 2024. Specexec: Massively parallel speculative decoding for interactive llm inference on consumer devices. Advances in Neural Information Processing Systems, 37:16342–16368. find
  34. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601.
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
  36. Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–3925.
  37. Zilin Xiao, Hongming Zhang, Tao Ge, Siru Ouyang, Vicente Ordonez, and Dong Yu. 2024. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv preprint arXiv:2410.05589. — [arXiv:2410.05589](https://arxiv.org/abs/2410.05589) find
  38. Hanling Yi, Feng Lin, Hongbin Li, Ning Peiyang, Xiaotian Yu, and Rong Xiao. 2024. Generation meets verification: Accelerating large language model inference with smart parallel auto-correct decoding. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5285–5299. find
  39. Jun Zhang, Jue Wang, Huan Li, Ladan Ding, Lianli Wu, and Gholamreza Haffari. 2024a. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). Preprint version appeared in…
  40. Situo Zhang, Hankun Wang, Da Ma, Zichen Zhu, Lu Chen, Kunyao Lan, and Kai Yu. 2024b. Adaea-gle: Optimizing speculative decoding via explicit modeling of adaptive draft structures. arXiv preprint arXiv:2412.18910. — [arXiv:2412.18910](https://arxiv.org/abs/2412.18910) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-training-free-adaptive-speculative-decoding/