Online Speculative Decoding
Paper · 2023 · Xiaoxuan Liu, Lanxiang Hu, Peter Bailis et al.
Summary
Speculative decoding speeds up LLM inference by having a small draft model propose tokens that a large target model checks in parallel. The gain depends on how often those proposals are accepted, and that rate falls when the draft model is much weaker or when user inputs are varied. The authors propose online speculative decoding (OSD), which keeps the draft model adapting to real serving traffic. When the target model rejects a draft token, both probability distributions are saved to a buffer. Once the buffer is large or old enough, the draft model is retrained with knowledge distillation, using forward KL, reverse KL or JSD as the distance. OSD can also maintain several draft models, each specialised for a language or topic, and route every query to the best-matching one. Tested on synthetic and real query data, OSD raises token acceptance by 0.1–0.65 and cuts latency by 1.42×–2.17×. This matters because services can serve LLMs faster without enlarging the draft model.
Key points
- Tested model pairs were LLaMA-160M drafting for Vicuna-7B and TinyLLaMA-1.1B drafting for Vicuna-33B. Latency was measured with llamacpp on one A100-80G GPU.
- For the latency experiments, the chosen distillation setup was forward KD on teacher-sampled data. Speedup peaks at 2.63× once acceptance exceeds 0.9.
- On the Spider SQL dataset, distillation improved acceptance for domain-typical tokens such as SELECT and WHERE. The 100 most common target tokens make up 72.2% of all generated tokens.
- On Chatbot Arena Conversations, separate draft models per language or topic achieved higher acceptance than a single shared draft model.
- OSD also works with Medusa-v1:
- On Spider, it raised Medusa's speedup from 1.34× to 2.01×, while standard speculative decoding with OSD reached 2.17×.
- On Arena, OSD added 0.35× to Medusa's speedup. Code is available at github.com/LiuXiaoxuanPKU/OSD (ICML 2024).
Fields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works
🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works
Cite
Cite as Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica and Hao Zhang. 2023. Online Speculative Decoding. arXiv:2310.07177.
@misc{liu2023online,
title = {{Online Speculative Decoding}},
author = {Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica and Hao Zhang},
year = {2023},
doi = {10.48550/arxiv.2310.07177},
eprint = {2310.07177},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2310.07177}
}References
- Huggingface transformers, 2023. URL https://huggingface.co/docs/transformers/ en/index. *Table 4. OSD and Medusa* Dataset Spider Chatbot Arena Extra Parameters (B) ↗
- Medusa-7B Medusa-7B + OSD Draft model + OSD 1.34× 2.01× 2.17× 2.03× 2.38× 1.51× 0.44 0.44 0.16 find
- Sharegpt, 2023. URL https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_ unfiltered. ↗
- Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649, 2023. \#\# 6. Conclusion
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/