Elamur Research⌕
☰
home / papers / 2026 / An Empirical Study of Speculative Decoding for Small Language Models

An Empirical Study of Speculative Decoding for Small Language Models

Paper · 2026 · Luca Mainardi, Selcuk Sandikci and Joaquin Vanschoren

Summary

Speculative decoding speeds up LLM inference: a cheap drafter proposes tokens, and the target model checks them in a single forward pass. Prior research has focused mostly on 7B–70B models. This paper asks whether the technique also pays off for small models of 1–2B parameters, which matter for on-device and agentic applications. The authors benchmark representatives of five method categories on Qwen2.5-1.5B-Instruct, SmolLM2-1.7B-Instruct and Llama-3.2-1B-Instruct, with Vicuna-7B-v1.3 as a large-model reference:

  • SpS with family-matched drafters
  • EAGLE-2
  • Kangaroo
  • Lookahead Decoding
  • PLD and REST

They extend SpecBench to support these models, add Kangaroo, and add per-component timing. The central finding is that small targets verify drafts very quickly, so the cost of producing drafts dominates, not their accuracy. As a result:

  • Off-the-shelf small drafters actually slow generation down.
  • Self-drafting helps only when its draft module is lightweight and trained for the specific model.
  • Almost free retrieval, such as PLD, gives steady gains.

The resulting guidelines help practitioners choose acceleration methods for edge deployment.

Key points

  • Evaluation uses SpecBench with six tasks of 80 instances each (480 samples) on a single NVIDIA A5500. EAGLE-2 and Kangaroo components were trained on ShareGPT.
  • Independent drafting fails with the available family drafters (Qwen2.5-0.5B-Instruct, SmolLM2-135M-Instruct). Llama-3.2-1B was excluded because its family has no smaller model.
  • PLD reaches 1.25–1.52× speedup on small models. REST is weaker: 1.01–1.19× with a small ShareGPT datastore and 0.92–1.10× with a large (~12GB) UltraChat datastore, where retrieval latency outweighs the 7.3–12.0 ms verification time.
  • Tree verification adds 22–29% to EAGLE-2 performance on all small models without extra verification cost.
  • Recommended practice:
    • Avoid external drafters unless they are wide-shallow, low-latency designs.
    • Train model-specific drafters if budget allows.
    • Otherwise prefer retrieval methods.
    • Add tree verification where possible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Eindhoven University of TechnologyEindhoven · Netherlands8 works · 602 citations

Also on the byline: NXP (Germany).

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Luca Mainardi, Selcuk Sandikci and Joaquin Vanschoren. 2026. An Empirical Study of Speculative Decoding for Small Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.255

@inproceedings{mainardi2026empirical,
  title = {{An Empirical Study of Speculative Decoding for Small Language Models}},
  author = {Luca Mainardi, Selcuk Sandikci and Joaquin Vanschoren},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.255},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.255}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aeala. 2023. Sharegpt v4.3 unfiltered cleaned split. find
  2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631.
  3. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, and 1 others. 2025. Smollm2: When smol goes big–data-centric training of a small language model. arXiv preprint ar…
  4. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153. find
  5. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, De huai Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. ArXiv, abs/2401.10774.
  6. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, L. Sifre, and John M. Jumper. 2023. Accelerating large language model decoding with speculative sampling. ArXiv, abs/2302.01318. find
  7. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.
  8. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.
  9. Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. 2023. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252. find
  10. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
  11. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Yakun Sophia Shao. 2025. Speed: Speculative pipelined execution for efficient decoding. In Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, pages 19–32. Springer. find
  12. Wouter Kool, Herke Van Hoof, and Max Welling. 2020. Ancestral gumbel-top-k sampling for sampling without replacement. Journal of Machine Learning Research, 21(47):1–36. find
  13. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2022. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. find
  14. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. Eagle-2: Faster inference of language models with dynamic draft trees. In Conference on Empirical Methods in Natural Language Processing.
  15. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. Eagle: Speculative sampling requires rethinking feature uncertainty. ArXiv, abs/2401.15077.
  16. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840. — [arXiv:2503.01840](https://arxiv.org/abs/2503.01840) find
  17. Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024. Kangaroo: Lossless self-speculative decoding via double early exiting. ArXiv, abs/2404.18911. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) find
  18. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2023. Specinfer: Accelerating large language model serving with tree-based speculat… find
  19. Giovanni Monea, Armand Joulin, and Edouard Grave. 2023. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581. — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) find
  20. Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. 2023. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427. find
  21. Apoorv Saxena. 2023. Prompt lookup decoding. find
  22. Noam M. Shazeer. 2019. Fast transformer decoding: One write-head is all you need. ArXiv, abs/1911.02150. find
  23. Benjamin Spector and Christal Re. 2023. Accelerating llm inference with staged speculative decoding. ArXiv, abs/2308.04623. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) find
  24. Mitchell Stern, Noam M. Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Neural Information Processing Systems. find
  25. Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2023. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36:30222–30242. find
  26. Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2022. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Conference on Empirical Methods in Natural Language Processing.
  27. Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. ArXiv, abs/2401.07851.
  28. Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. 2024. Decoding speculative decoding. arXiv preprint arXiv:2402.01528. find
  29. Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023a. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487. — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) find
  30. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, and 25 others. 2024a. Qwen2.5 technical report. ArXiv,… find
  31. Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2024b. Multi-candidate speculative decoding. arXiv preprint arXiv:2401.06706. — [arXiv:2401.06706](https://arxiv.org/abs/2401.06706) find
  32. Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. 2023b. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding. ArXiv, abs/2307.05908. — [arXiv:2307.05908](https://arxiv.org/abs/2307.05908) find
  33. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168.
  34. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. ArXiv, abs/2401.02385.
  35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  36. Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2023. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461. — [arXiv:2310.08461](https://arxiv.org/abs/2310.08461) find
  37. A aim to accelerate, and a draft model M p used for speculation. We write x ∼ p to denote that x is sampled from distribution p. A.2 Autoregressive Decoding Transformer-based language models generate text through autoregressive decoding, where tokens are produced sequentially, each conditioned on al… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mainardi-2026-empirical-study-speculative-decoding-small/