Elamur Research⌕
☰
home / papers / 2024 / ServerlessLLM: Low-Latency Serverless Inference for Large Language Models

ServerlessLLM: Low-Latency Serverless Inference for Large Language Models

Paper · 2024 · Yao Fu, Leyang Xue, Yeqi Huang et al.

Summary

Serverless platforms let LLM owners pay only per request, but each cold start forces the system to pull a checkpoint of many gigabytes from remote storage and deserialize it onto GPUs. That startup step can take tens of seconds. ServerlessLLM attacks this delay by treating the DRAM and SSDs already sitting inside GPU servers as a local checkpoint cache. The system has three parts. The first is a checkpoint format built for fast loading, paired with a pipelined loader that spans several storage tiers. The second is live migration of running inferences, which lets a new request claim a server that already holds its model. The third is a cluster scheduler that predicts startup time on every server and picks the quickest one. Evaluation covered microbenchmarks plus workloads modeled on the Azure serverless trace, using OPT, LLaMA-2 and Falcon models. Against KServe and Ray Serve, latency dropped by 10–200X. This matters because keeping latency low under serverless sharing makes it cheaper to host many custom LLMs on a limited GPU pool.

Key points

  • Checkpoint loading runs 3.6–8.2X faster than Safetensors and PyTorch loaders on OPT, LLaMA-2 and Falcon, and 4.4X faster for LoRA adaptors.
  • End-to-end tests used Azure Trace–style request patterns with GSM8K and ShareGPT inputs. Baselines were KServe, Ray Serve, and Ray Serve with local caching, and ServerlessLLM reduced latency by 10–200X.
  • KServe started at a 128 s first-token latency, of which 114 s was spent downloading OPT-6.7B over 1 Gbps. With enhancements it reached 28 s, while only ServerlessLLM got below one second.
  • The motivating numbers come from two sources:
    • In the Azure Trace, more than 40% of functions have cold-start rates above 25%.
    • A 130 GB LLaMA-2-70B checkpoint needs at least 26 s to fetch even at 5 GB/s.
  • The code is open source at github.com/ServerlessLLM/ServerlessLLM.

Interesting ideas

  • I1. Lay out files for the loader rather than the trainer. The checkpoint is stored for sequential, chunked reads with direct tensor addressing, which avoids costly deserialization. The same idea applies to any large artifact loaded repeatedly, such as embeddings, indexes or corpora.
  • I2. Migrate tokens, not state. When an inference moves to another server, only the token sequence is sent, and the destination rebuilds the KV cache by recomputation. Trading cheap compute for expensive network transfer is a general pattern for moving stateful jobs.
  • I3. Use the hardware you already own as a cache. Servers' host memory and SSDs were mostly idle, so tiering checkpoints across them removed most remote downloads. Auditing underused local storage before buying faster networks is a cheap first step.
  • I4. Schedule on estimated startup time. Cost models estimate loading time from each storage tier and migration time, so placement follows predicted startup latency rather than simple load. This approach suits any cluster where data locality dominates job launch cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksHardware and Architecture · 167 worksComputer Networks and Communications · 100 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel and Luo Mai. 2024. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models. arXiv:2401.14351.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.14351 · arXiv 2401.14351

@misc{fu2024serverlessllm,
  title = {{ServerlessLLM: Low-Latency Serverless Inference for Large Language Models}},
  author = {Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel and Luo Mai},
  year = {2024},
  doi = {10.48550/arxiv.2401.14351},
  eprint = {2401.14351},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.14351}
}

References

96 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Amazon elasticache pricing. https://aws.amazon.com/elasticache/pricing/?nc=sn&loc=5. Accessed: 2024-05-31. ↗
  2. LLM serving systems. Recent advancements in LLM serving have improved inference latency and throughput. Orca [85] uses continuous batching for better GPU utilization during [2] Custom LLMs. https://deepinfra.com/docs/adv anced/custom_llms, 2024. Accessed on 2024-06-02. ↗
  3. [3] Generative AI tailored to you. https://fireworks.ai/, 2024. Accessed on 2024-06-02. ↗
  4. Rasley, and Yuxiong He. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22. IEEE Press, 2022. find
  5. [4] The leading enterprise AI platform. https://cohere.com/, 2024. Accessed on 2024-06-02. ↗
  6. [5] Run AI with an API. https://replicate.com/, 2024. Accessed on 2024-06-02. ↗
  7. [15] Lixiang Ao, George Porter, and Geoffrey M. Voelker. Faasnap: Faas made fast using snapshot-based vms. In Proceedings of the Seventeenth European Conference on Computer Systems, EuroSys ’22, page 730–746, New York, NY, USA, 2022. Association for Computing Machinery. find
  8. [6] Serverless endpoints for leading open-source models. https://www.together.ai/products\#inference, 2024. Accessed on 2024-06-02. ↗
  9. [7] Your data. your AI. your future. https://www.databr icks.com/, 2024. Accessed on 2024-06-02. ↗
  10. [16] The KServe Authors. Kserve. https://github.com/kserve/kserve, 2023. Accessed on 2024-01-22. ↗
  11. [8] Mania Abdi, Samuel Ginzburg, Xiayue Charles Lin, Jose Faleiro, Gohar Irfan Chaudhry, Inigo Goiri, Ricardo Bianchini, Daniel S Berger, and Rodrigo Fonseca. Palette load balancing: Locality hints for serverless functions. In Proceedings of the Eighteenth European Conference on Computer Systems, Eu… find
  12. [17] Jens Axboe. Flexible I/O Tester, 2022. Accessed on 2024-01-22. find
  13. [18] Wei Bai, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, Rebecca Chow, Jeff Cohen, Mahmoud Elhaddad, Vivek Ette, Igal Figlin, Daniel Firestone, Mathew George, Ilya German, Lakhmeet Ghai, Eric Green, Al… find
  14. [9] Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. SAND: Towards High-Performance serverless computing. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 923–935, Boston, MA, July 2018. USENIX Association. find
  15. [10] Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni. Batch: machine learning inference serving on serverless platforms with adaptive batching. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15. IEEE, 2020. find
  16. [11] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-th… find
  17. [12] Amazon. AWS S3. https://aws.amazon.com/s3/, 2023. Accessed on 2024-01-22. ↗
  18. [19] Anyscale Technical Blog. https://www.anyscale.com/blog/loading-llama-2-70b-20x-faste r-with-anyscale-endpoints, 2023. Accessed on 2024-01-22. ↗
  19. [13] Amazon. Serverless inference - minimizing cold starts. https://docs.aws.amazon.com/sagemaker/late st/dg/serverless-endpoints.html, 2023. Accessed on 2024-01-22. ↗
  20. [20] Banana.dev Technical Blog. https://www.banana.d ev/blog/turboboot, 2023. Accessed on 2024-01-22. ↗
  21. [21] Microsoft Official Blog. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. https://blogs.microsoft.com/blog/2 023/02/07/reinventing-search-with-a-new-a [14] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olat… ↗
  22. [22] James Cadden, Thomas Unger, Yara Awad, Han Dong, Orran Krieger, and Jonathan Appavoo. Seuss: skip redundant paths to make serverless fast. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys ’20, New York, NY, USA, 2020. Association for Computing Machinery. find
  23. [32] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022. ↗
  24. [23] Andrew A. Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. Reducing the carbon impact of generative AI inference (today and in 2035). In HotCarbon, pages 11:1–11:7. find
  25. [33] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In OSDI, pages 443–462. USENIX Association, 2020. find
  26. [24] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 199–216, Carlsbad, CA, July 2022. USENIX Associ… find
  27. [34] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  28. [35] HuggingFace. https://huggingface.co, 2023. Accessed on 2024-01-22. ↗
  29. [25] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021.
  30. [36] HuggingFace. Safetensors: ML Safer for All. https://github.com/huggingface/safetensors, 2023. Accessed on 2024-01-22. ↗
  31. [37] HuggingFace. Speed Comparison. https://huggin gface.co/docs/safetensors/speed, 2023. Accessed on 2024-01-22. find
  32. [26] GitHub Copilot. https://github.com/features/ copilot, 2023. Accessed on 2024-01-22. ↗
  33. [27] DataBricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blo g/llm-inference-performance-engineering-b est-practices, 2023. Accessed on 2024-01-22. ↗
  34. [38] Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed. Zookeeper: Wait-free coordination for internet-scale systems. In Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, USENIXATC’10, page 11, USA, 2010. USENIX Association. find
  35. [28] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen. Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming La… find
  36. [39] Jinwoo Jeong, Seungsu Baek, and Jeongseob Ahn. Fast and efficient model serving using multi-gpus with direct-host-access. In EuroSys, pages 249–265. ACM, 2023. find
  37. [40] Zhipeng Jia and Emmett Witchel. Boki: Stateful serverless computing with shared logs. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, page 691–707, New York, NY, USA, 2021. Association for Computing Machinery. find
  38. [29] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), jan 2022. find
  39. [41] Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. Pocket: Elastic ephemeral storage for serverless analytics. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 427–444, Carlsbad, CA, October 2018. USENIX As… find
  40. [30] Cloud Native Computing Foundation. etcd. https://etcd.io, 2023. Accessed on 2024-01-22. ↗
  41. [31] Yixiao Gao, Qiang Li, Lingbo Tang, Yongqing Xi, Pengcheng Zhang, Wenwen Peng, Bo Li, Yaohui Wu, Shaozong Liu, Lei Yan, et al. When cloud storage meets [42] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memor… — Efficient Memory Management for Large Language Model Serving with PagedAttention
  42. [51] MinIO. MinIO. https://min.io, 2023. Accessed on 2024-01-22. ↗
  43. [52] NVIDIA. NVIDIA DGX H100. https://resources.nvidia.com/en-us-dgx-systems/ai-enterpris e-dgx, 2023. Accessed on 2024-01-22. ↗
  44. [43] Jie Li, Laiping Zhao, Yanan Yang, Kunlin Zhan, and Keqiu Li. Tetris: Memory-efficient serverless inference through tensor sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022. find
  45. [53] Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. SOCK: Rapid task provisioning with Serverless-Optimized containers. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 57–70, Boston, MA, July 2018. USENIX Associatio… find
  46. [44] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In OSDI, pages 663–679. USENIX Association, 2023.
  47. [54] OpenAI. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt/, 2022. Accessed on 2024-01-22. ↗
  48. [55] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting, 2023.
  49. [45] Zijun Li, Jiagan Cheng, Quan Chen, Eryu Guan, Zizheng Bian, Yi Tao, Bin Zha, Qiang Wang, Weidong Han, and Minyi Guo. Rund: A lightweight secure container runtime for high-density deployment and high-concurrency startup in serverless computing. In USENIX Annual Technical Conference, pages 53–68.… find
  50. [56] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. find
  51. [46] Ashraf Mahgoub, Karthick Shankar, Subrata Mitra, Ana Klimovic, Somali Chaterji, and Saurabh Bagchi. SONIC: Application-aware data passing for chained serverless applications. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 285–301. find
  52. USENIX Association, July 2021. find
  53. [57] PyTorch. PyTorch Documentation: Saving and Loading Models. https://pytorch.org/tutorials/beginn er/saving_loading_models.html, 2023. Accessed on 2024-01-22. ↗
  54. [47] Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, Somali Chaterji, and Saurabh Bagchi. ORION and the three rights: Sizing, bundling, and prewarming for serverless DAGs. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 303–320, Carlsba… find
  55. [58] PyTorch. torch.load. https://pytorch.org/do cs/stable/generated/torch.load.html, 2023. Accessed on 2024-01-22. ↗
  56. [59] Francisco Romero, Gohar Irfan Chaudhry, Íñigo Goiri, Pragna Gopa, Paul Batum, Neeraja J. Yadwadkar, Rodrigo Fonseca, Christos Kozyrakis, and Ricardo Bianchini. Faa$t: A transparent auto-scaling cache for serverless applications. In Proceedings of the ACM Symposium on Cloud Computing, SoCC ’21,… find
  57. [48] Luo Mai, Guo Li, Marcel Wagenländer, Konstantinos Fertakis, Andrei-Octavian Brabete, and Peter Pietzuch. {KungFu}: Making training in distributed machine learning adaptive. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 937–954, 2020. find
  58. [60] Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 397–411, 2021. find
  59. [49] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022. ↗
  60. [61] Rohan Basu Roy, Tirthak Patel, and Devesh Tiwari. Ice-breaker: Warming serverless functions better with heterogeneity. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 753– 767, 2022. find
  61. [50] Microsoft. Azure ML. https://learn.microsoft.com/en-us/azure/machine-learning, 2023. Accessed on 2024-01-22. ↗
  62. [62] ONNX Runtime. API Reference, onnxruntime.backend.prepare. https://onnxruntime.ai/docs/api/python/api_summary.html, 2023. Accessed on 2024-01-22. ↗
  63. Hellerstein, and Alexey Tumanov. Cloudburst: stateful functions-as-a-service. Proc. VLDB Endow., 13(12):2438–2452, jul 2020. find
  64. [70] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjàvu: Kv-cache streaming for fast, fault-tolerant generative LLM serving, 2024. find
  65. [63] AWS SageMaker. Machine Learning Service - Amazon SageMaker. https://aws.amazon.com/pm/sagema ker/, 2023. ↗
  66. [71] Mistral AI Team. Better, faster, stronger. Mistral AI News, April 2024. Accessed: 2024-05-31. find
  67. [64] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 USENIX Annual Tech… find
  68. USENIX Association, July 2020. find
  69. [72] The Mosaic Research Team. Introducing DBRX: A new state-of-the-art open LLM. Mosaic AI Research, March 2024. Accessed: 2024-05-31. find
  70. [73] The Ray Team. Ray serve. https://docs.ray.io/ en/latest/serve/index.html, 2023. Accessed on 2024-01-22. ↗
  71. [65] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285, 2023.
  72. [74] TensorFlow. TensorFlow Documentation: Using the Saved Model Format. https://www.tensorflow.org/guide/saved_model, 2023. Accessed on 2024-01-22. ↗
  73. [75] TensorFlow. tf.saved_model.load. https://www.tens orflow.org/api_docs/python/tf/saved_model/ load, 2023. Accessed on 2024-01-22. ↗
  74. [66] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, S…
  75. [76] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fu… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  76. [67] Simon Shillaker and Peter Pietzuch. Faasm: Lightweight isolation for efficient stateful serverless computing. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 419–433. find
  77. USENIX Association, July 2020. find
  78. [68] Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangova… find
  79. [77] Dmitrii Ustiugov, Plamen Petrov, Marios Kogias, Edouard Bugnion, and Boris Grot. Benchmarking, analysis, and optimization of serverless function snapshots. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’2… find
  80. New York, NY, USA, 2021. Association for Computing Machinery. find
  81. [87] Minchen Yu, Zhifeng Jiang, Hok Chun Ng, Wei Wang, Ruichuan Chen, and Bo Li. Gillis: Serving large neural networks in serverless functions with automatic model partitioning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), pages 138–148. find
  82. [78] Marcel Wagenländer, Luo Mai, Guo Li, and Peter Pietzuch. Spotnik: Designing distributed machine learning for transient cloud resources. In 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20), 2020. find
  83. [88] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. FaaSwap: SLO-Aware, GPU-Efficient Serverless Inference via Model Swapping. CoRR, abs/2306.03622, 2023. find
  84. [79] Ao Wang, Shuai Chang, Huangshi Tian, Hongqi Wang, Haoran Yang, Huiba Li, Rui Du, and Yue Cheng. FaaS-Net: Scalable and fast provisioning of custom serverless container runtimes at alibaba cloud function compute. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 443–457. find
  85. USENIX Association, July 2021. find
  86. [89] Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. MArk: Exploiting cloud services for Cost-Effective, SLO-Aware machine learning inference serving. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 1049–1062, Renton, WA, July 2019. USENIX Association. find
  87. [80] Xingda Wei, Fangming Lu, Tianxia Wang, Jinyu Gu, Yuhan Yang, Rong Chen, and Haibo Chen. No provisioned concurrency: Fast RDMA-codesigned remote fork for serverless computing. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 497–517, Boston, MA, July 2023.… find
  88. [90] Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. SHEPHERD: Serving DNNs in the Wild. In NSDI, pages 787–808. USENIX Association, 2023. find
  89. [91] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-tr… — OPT: Open Pre-trained Transformer Language Models
  90. [81] Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models, 2023.
  91. [82] xAI. Grok-1, 2024. Accessed: 2024-05-31. find
  92. [83] Ming Xu. https://huggingface.co/datasets/sh ibing624/sharegpt_gpt4, 2023. ↗
  93. [92] Yanqi Zhang, Íñigo Goiri, Gohar Irfan Chaudhry, Rodrigo Fonseca, Sameh Elnikety, Christina Delimitrou, and Ricardo Bianchini. Faster and cheaper serverless computing on harvested resources. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, page 724–739,… find
  94. [84] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. Infless: a native serverless system for low-latency, high-throughput inference. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and O… find
  95. [85] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Associatio…
  96. [86] Minchen Yu, Tingjia Cao, Wei Wang, and Ruichuan Chen. Following the data, not the function: Rethinking function orchestration in serverless computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1489–1504, Boston, MA, April 2023. USENIX Association. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fu-2024-serverlessllm-low-latency-serverless-inference/