Elamur Research⌕
☰
home / papers / 2026 / River-LLM: Large Language Model Seamless Exit Based on KV Share

River-LLM: Large Language Model Seamless Exit Based on KV Share

Paper · 2026 · Yingtao Shen and An Zou

Summary

Early Exit lets a decoder-only LLM stop at a shallow layer for easy tokens, but real speed gains mostly vanish because of KV Cache Absence. A token that leaves early never writes keys and values for the skipped layers, so later tokens have no history to attend to at those depths. Existing fixes, such as batched recomputation, State Propagation and KV masking, either add latency or reduce accuracy. The authors first compare these strategies in one shared test setup. On Llama3.2 1B with GSM8K, token-level exit could in theory cut layer computation 3.3×, yet the measured gains are small. They then propose River-LLM, a training-free method with a lightweight "KV-Shared Exit River." Its exit path shares KV with the backbone, so the missing cache entries are produced during the exit itself and do not need to be rebuilt afterwards. Exit decisions use the similarity between a decoder block's input and output states as a proxy for accumulated KV error. Across GSM8K, MATH and HumanEval on four models, River-LLM reaches roughly 1.5×–2.2× real throughput with modest accuracy loss.

Key points

  • Headline speedup: the paper claims 1.71×–2.16× wall-clock speedup, measured in tokens/s on an NVIDIA A40.
    • Llama3.2 1B shows the widest gains: 2.16× on GSM8K (33.2→29.3 accuracy), 1.88× on MATH and 1.71× on HumanEval.
    • The table also contains lower values, for example 1.53× for Phi4-mini on MATH.
  • Accuracy on larger models: Llama3.1 8B loses only 0.4–3.8 points, with 1.72×–1.78× speedup. Ministral3 8B on GSM8K scores 84.3 versus 84.5 for the backbone, at 1.77×.
  • Comparison with quantization: full quantization is usually somewhat faster than River-LLM but loses noticeably more accuracy. The authors also say River-LLM sets a better accuracy–speed Pareto frontier than masking, recomputation and State Propagation.
  • Low overhead:
    • The exit criterion costs O(d), about 100 µs per decision on Llama3.1 8B, or 0.0688% of per-token time.
    • Thanks to the shared KV and backbone offloading, memory use (4K–64K context) stays below the backbone and below Balcony and EE-LLM.
  • Limitations: the method is tested only up to 8B parameters. It also helps less on prefill-heavy tasks such as MMLU, where sequence-level exit is used instead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yingtao Shen and An Zou. 2026. River-LLM: Large Language Model Seamless Exit Based on KV Share. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1746

@inproceedings{shen2026river,
  title = {{River-LLM: Large Language Model Seamless Exit Based on KV Share}},
  author = {Yingtao Shen and An Zou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1746},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1746}
}

References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mehmet Emre Akbulut, Hazem Hesham Yousef Shalby, Fabrizio Pittorino, and Manuel Roveri. 2026. Infoq: Mixed-precision quantization via global information flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 19598–19606. find
  2. Hicham Badri and Appu Shaji. 2023. Half-quadratic quantization of large machine learning models. find
  3. Sébastien Bubeck and 1 others. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  4. Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos I. Ve-nieris. 2025. Progressive mixed-precision decoding for efficient LLM inference. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  5. Mark Chen and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  6. Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. In International Conference on Machine Learning, pages 7163–7189. PMLR. find
  7. Christopher Clark and 1 others. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  8. Peter Clark and 1 others. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  9. Karl Cobbe and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  10. Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. 2023. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference. arXiv preprint arXiv:2307.02628. — [arXiv:2307.02628](https://arxiv.org/abs/2307.02628) find
  11. Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. In ICLR 2020-Eighth International Conference on Learning Representations, pages 1–14. find
  12. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, and 1 others. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Associa…
  13. Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, and Yequan Wang. 2025. Not all layers of llms are necessary during inference. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 5083–5091. International Joint Con… find
  14. Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. 2021. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456. find
  15. Shwai He, Tao Ge, Guoheng Sun, Bowei Tian, Xiaoyang Wang, and Dong Yu. 2025a. Router-tuning: A simple and effective approach for dynamic depth. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1925–1938. find
  16. Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhenzhe Zheng, and Fan Wu. 2025b. Adaskip: Adaptive sublayer skipping for accelerating long-context llm inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24050–24058. find
  17. Dan Hendrycks and 1 others. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  18. Dan Hendrycks and 1 others. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
  19. Lianming Huang, Shangyu Wu, Yufei Cui, Ying Xiong, Xue Liu, Tei-Wei Kuo, Nan Guan, and Chun Jason Xue. 2024. Raee: A robust retrieval-augmented early exiting framework for efficient inference. arXiv preprint arXiv:2405.15198. — [arXiv:2405.15198](https://arxiv.org/abs/2405.15198) find
  20. Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, and Sujoy Paul. 2024. Mixture of nested experts: Adaptive processing of visual tokens. Advances in Neural Information Processing Systems, 37:58480– 58497. find
  21. Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh, Parsa Farinneya, Hossein Ra-jabzadeh, Aref Jafari, Boxing Chen, and Marzieh S. Tahaei. 2025. Balcony: A lightweight approach to dynamic inference of generative language models. In Proceedings of the 2025 Conference on Empirical Methods i… find
  22. Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Hui Qian, John Lui, and 1 others. 2024. D-llm: A token adaptive computing resource allocation strategy for large language models. Advances in Neural Information Processing Systems, 37:1725–1749. find
  23. Sangwoo Kwon, Seong Hoon Seo, Jae W. Lee, and Yeonhong Park. 2025. DP-LLM: Runtime model adaptation with dynamic layer-wise precision assignment. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  24. Xiangjie Li, Chenfei Lou, Yuchi Chen, Zhengping Zhu, Yingtao Shen, Yehan Ma, and An Zou. 2023. Predictive exit: Prediction of fine-grained early exits for computation-and energy-efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8657–8665. find
  25. Akide Liu and 1 others. 2024. Minicache: Kv cache compression for large language models. arXiv preprint arXiv:2405.14365. find
  26. Xuan Luo, Weizhi Wang, and Xifeng Yan. 2025. Diffskip: Differential layer skipping in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7221–7231. find
  27. Ruijie Miao, Yihan Yan, Xinshuo Yao, and Tong Yang. 2024. An efficient inference framework for early-exit large language models. arXiv preprint arXiv:2407.20272. — [arXiv:2407.20272](https://arxiv.org/abs/2407.20272) find
  28. Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. 2016. Conditional deep learning for energy-efficient and enhanced pattern recognition. In Design, Automation & Test in Europe Conference & Exhibition. IEEE. find
  29. Reiner Pope and 1 others. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems. find
  30. Baptiste Rozière and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  31. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472. find
  32. Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition. IEEE. find
  33. Jason Wei and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems.
  34. Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. 2025. Specee: Accelerating large language model inference with speculative early exiting. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pages 467–481. find
  35. Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Minghui Chen, Zheng Lin, and Weiping Wang. 2025. Dynamic early exit in reasoning models. arXiv preprint arXiv:2504.15895. find
  36. Rowan Zellers and 1 others. 2019. Hellaswag: Can a machine really finish your sentence? In ACL. find
  37. Yi Zhang, Kai Zhang, Zheyang Li, Wenming Tan, Ye Ren, and Jilin Hu. 2025. Beyond dynamic quantization: An efficient static hierarchical mix-precision framework for near-lossless LLM compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-river-llm-large-language-model/