Elamur Research⌕
☰
home / papers / 2024 / LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism

Paper · 2024 · Bingyang Wu, Shengyu Liu, Yinmin Zhong et al.

Summary

Long-context LLMs (Claude-3, Gemini-1.5 and LWM all accept up to 1M tokens) create very uneven resource demands. Requests differ widely in length, and a single request needs much more compute in prefill than in decoding. Current serving systems fix their parallel configuration before launch, so they waste GPUs, move key-value caches between groups and fragment memory. The authors propose Elastic Sequence Parallelism (ESP), which picks the degree of parallelism (DoP) for each request at every iteration. They build it into LoongServe, a serving system that extends Striped Attention from training to inference. LoongServe adds:

  • scaling mechanisms that cost no extra communication,
  • token-level placement of KV cache across instances,
  • an iteration-level scheduler with polynomial complexity.

On real-world datasets, LoongServe delivers up to 3.85× the throughput of chunked prefill and 5.81× that of prefill-decoding disaggregation, and it speeds up both phases at once. This matters because it treats parallelism as something to decide per request at runtime, not a fixed deployment setting.

Key points

  • For one 1M-token request on an LWM model, the KV cache alone is about 488GB, which is beyond any single current GPU. Attention cost also grows quadratically with input length.
  • Reported baselines are chunked prefill (SARATHI, DeepSpeed-FastGen) and prefill-decoding disaggregation (DistServe, SplitWise, TetriInfer). LoongServe reaches up to 3.85× and 5.81× their throughput, respectively.
  • Scaling overhead:
    • Scaling down adds under 2% overhead.
    • Scaling up costs under 10% for small batches.
    • For large batches, scaling up can halve per-iteration latency.
  • On ShareGPT at 25 requests/s, the global manager triggers about 7.12 scale-up operations per 10 seconds. This shows that elasticity is actually exercised under load.
  • The scheduler's analytical cost model stays within 10% of measured iteration times across SP/TP mixes (SP2TP4, SP4TP2, SP8TP1).

Interesting ideas

  • I1. Fold resharding into work you already do. Proactive scale-down reuses the communication that prefill attention already performs to move KV data to where decoding will run. Hiding redistribution inside a compute step's existing data movement could help any pipeline whose layout changes between stages.
  • I2. Multi-master decoding. Instead of migrating KV caches when DoP grows, LoongServe leaves them in place and lets several instances share decoding, overlapping partial communication with computation. "Move compute to the data, not data to compute" applies broadly to stateful serving.
  • I3. Token-granularity memory placement. Managing KV entries per token across instances, with no locality constraint, removes the fragmentation caused by isolated GPU groups. A similar approach suits any pooled-memory system with highly variable item sizes.
  • I4. Split a huge scheduling problem into ordered steps. The scheduler handles DoP choice, batching, KV placement and scaling in four sequential steps, keeping decisions polynomial within millisecond-scale budgets. Decomposing a joint optimization into ordered steps is a practical template for real-time schedulers.
  • I5. Validate the cost model separately. The authors evaluate how well their analytical latency model predicts reality, not only end-to-end throughput. This is a good evaluation habit for any model-driven scheduler.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,267 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China152 works · 13,166 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksHardware and Architecture · 167 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. arXiv:2404.09526.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.09526 · arXiv 2404.09526

@misc{wu2024loongserve,
  title = {{LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism}},
  author = {Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu and Xin Jin},
  year = {2024},
  doi = {10.48550/arxiv.2404.09526},
  eprint = {2404.09526},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.09526}
}

References

61 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt. (2022). ↗
  2. 2023. Bard, an experiment by Google. https://bard.google.com/. (2023). ↗
  3. 2023. Optimized primitives for collective multi-GPU communicatio Resources. https://github.com/NVIDIA/nccl. (2023). ↗
  4. 2023. ShareGPT Teams. https://sharegpt.com/. (2023). ↗
  5. 2024. DeepSpeed Model Implementations for Inference (MII). https://github.com/microsoft/DeepSpeed-MII. (2024). ↗
  6. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. In arXiv.
  7. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference. In Conference on Machine Learning and Systems. find
  8. Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. In arXiv.
  9. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In The Conference on Empirical Methods in Natural Language Processing (EMNLP).
  10. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-Eval: Instituting Standardized Evaluation for Long Context Language Models. In arXiv.
  11. Anthropic. 2024. Introducing the next generation of Claude. https://www.anthropic.com/news/claude-3-family. (2024). ↗
  12. William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Striped Attention: Faster Ring Attention for Causal Transformers. In arXiv. find
  13. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. In arXiv.
  14. Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In arXiv.
  15. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Neural Information Processing Systems. Our next-generation model: Gemini 1.5. find
  16. Google. 2024. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/. (2024). ↗
  17. Significant Gravitas. 2023. AutoGPT. https://github.com/Significant-Gravitas/AutoGPT. (2023). ↗
  18. Diandian Gu, Yihao Zhao, Yinmin Zhong, Yifan Xiong, Zhenhua Han, Peng Cheng, Fan Yang, Gang Huang, Xin Jin, and Xuanzhe Liu. 2023. ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning. In ACM ASPLOS. find
  19. Juncheng Gu, Mosharaf Chowdhury, Kang G Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Harry Liu, and Chuanxiong Guo. 2019. Tiresias: A GPU Cluster Manager for Distributed Deep Learning.. In USENIX NSDI. find
  20. Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. In arXiv.
  21. Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. In arXiv. find
  22. Changho Hwang, Taehyun Kim, Sunghyun Kim, Jinwoo Shin, and KyoungSoo Park. 2021. Elastic Resource Sharing for Distributed Deep Learning. In USENIX NSDI. find
  23. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. In arXiv.
  24. Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, and Gregory R. Ganger. 2023. Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling. In ACM SOSP. find
  25. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  26. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Reducing Activation Recomputation in Large Transformer Models. In arXiv.
  27. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Reducing activation recomputation in large transformer models. In Conference on Machine Learning and Systems.
  28. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In ACM SOSP.
  29. Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. Lightseq: Sequence level parallelism for distributed training of long context transformers. In arXiv. find
  30. Mingzhen Li, Wencong Xiao, Biao Sun, Hanyu Zhao, Hailong Yang, Shiru Ren, Zhongzhi Luan, Xianyan Jia, Yi Liu, Yong Li, Wei Lin, and Depei Qian. 2023. EasyScale: Accuracy-consistent Elastic Training for Deep Learning. In ACM SC. find
  31. Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2023. Sequence Parallelism: Long Sequence Training from System Perspective. In Annual Meeting of the Association for Computational Linguistics (ACL).
  32. Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In USENIX OSDI.
  33. Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. In arXiv. find
  34. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024. World Model on Million-Length Video and Language with RingAttention. In arXiv. find
  35. Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. In arXiv.
  36. Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Optimizing Speculative Decoding for Serving Large Language Models Using Goodput. In arXiv. find
  37. Yujing Ma, Florin Rusu, Kesheng Wu, and Alexander Sim. 2021. Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers. In arXiv. find
  38. Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In USENIX OSDI. find
  39. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations (ICLR). find
  40. Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. 2023. Faster Causal Attention Over Large Sequences Through Sparse Flash Attention. In arXiv. find
  41. Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. 2023. Splitwise: Efficient generative LLM inference using phase splitting. In arXiv.
  42. Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. One Queue Is All You Need: Resolving Head-of-Line Blocking in Large Language Model Serving. In arXiv. find
  43. Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. 2018. Optimus: an efficient dynamic resource scheduler for deep learning clusters. In EuroSys. find
  44. Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In USENIX OSDI. find
  45. Noam Shazeer. 2019. Fast Transformer Decoding: One Write-Head is All You Need. In arXiv. find
  46. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-billion Parameter Language Models using Model Parallelism. In arXiv. find
  47. Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. 2024. Linear Attention Sequence Parallelism. In arXiv. find
  48. LightLLM Team. 2023. LightLLM: A Light and Fast Inference Service for LLM. https://github.com/ModelTC/lightllm. (2023). ↗
  49. Qwen Team. 2024. Introducing Qwen1.5. https://qwenlm.github.io/ blog/qwen1.5/. (2024). ↗
  50. Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages.
  51. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv.
  52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. In arXiv.
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.
  54. Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models. In arXiv.
  55. Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In USENIX OSDI. find
  56. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR). find
  57. F. Frances Yao. 1980. Efficient dynamic programming using quadrangle inequalities. In ACM Symposium on Theory of Computing. find
  58. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-BasedGenerative Models. In USENIX OSDI. find
  59. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K. In arXiv.
  60. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Neural Information Processing Systems. find
  61. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In USENIX OSDI.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2024-loongserve-efficiently-serving-long-context/