Elamur Research⌕
☰
home / papers / 2023 / Punica: Multi-Tenant LoRA Serving

Punica: Multi-Tenant LoRA Serving

Paper · 2023 · Lequn Chen, Zihao Ye, Yongji Wu et al.

Summary

Organisations that fine-tune one base LLM with LoRA for many tenants usually end up running each adapter as a separate model. That wastes GPUs, because every adapter shares the same frozen backbone. Punica is a serving system that keeps one copy of the pre-trained model on each GPU and fetches only the small LoRA matrices from remote storage when a request needs them. At the heart of the system is SGMV (Segmented Gather Matrix-Vector Multiplication), a CUDA kernel the authors built from scratch. SGMV lets requests for different adapters share one batch: the backbone part runs as an ordinary GEMM, and the per-adapter low-rank additions are computed together, with requests for the same adapter grouped. On top of this kernel, a scheduler packs work onto as few GPUs as possible and periodically migrates running requests to free hardware. The authors tested adapters built on Llama2 7B, 13B and 70B using NVIDIA A100 clusters. On the same hardware, Punica reached about 12x the throughput of state-of-the-art LLM serving systems, adding only around 2 ms latency per token. This makes serving many customised models far cheaper.

Key points

  • The SGMV kernel batches the LoRA add-on computation across different adapters and uses GPU Tensor Cores. The authors saw almost no performance difference between batching identical adapters and mixed ones.
  • Reported results: roughly 12x higher throughput than existing LLM serving systems, at about +2 ms per token. Experiments used Llama2 7B/13B/70B LoRA variants on A100 GPUs. The code is open source at github.com/punica-ai/punica.
  • Batching has very different effects in the two stages:
    • In prefill, the GPU is already saturated, so latency grows with batch size.
    • In decode, going from batch 1 to 32 raised step latency only from 11 to 13 ms for short sequences, and from 17 to 34 ms for long ones.
  • A LoRA adapter adds only about 0.1–1% of the model's weights, with rank typically far below the hidden size (e.g. 16 vs 4096). Loading an adapter on demand therefore takes only milliseconds.
  • The architecture has three parts:
    • frontends that expose a RESTful API;
    • a central scheduler;
    • per-server runners that control the GPUs and stream generated tokens back to users.

Interesting ideas

  • I1. Split shared and per-tenant computation. Run the shared backbone as one large batched operation, and handle the small tenant-specific parts with a dedicated gather-style kernel. This pattern could suit any setup where many variants share a common core, such as language-specific adapters on one multilingual model.
  • I2. Sort the batch by adapter. Placing requests for the same adapter next to each other turns scattered small multiplications into denser segments that use the hardware better. Sorting by key before computing is a cheap trick that carries over to many batched workloads.
  • I3. Optimise the stage that dominates cost. The authors focus engineering effort on the decode stage, where most serving cost lies, and use simple methods elsewhere, such as loading adapters only when needed. This is a sensible way to prioritise effort in any pipeline.
  • I4. Consolidate instead of spreading load. New requests go to GPUs that are already busy until those are full, and ongoing requests are migrated to pack the work together. This strategy works best when adding work to a running batch is nearly free, and it can cut idle hardware in shared research clusters.
  • I5. Measure how latency scales with batch size, per stage. Plotting latency against batch size separately for each stage quickly shows where batching pays off. It is an easy diagnostic to run before deciding how to deploy any model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,763 citations🎓Duke UniversityDurham · United States37 works · 15,861 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksHardware and Architecture · 167 works

Cite

Cite as Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze and Arvind Krishnamurthy. 2023. Punica: Multi-Tenant LoRA Serving. MLSys 2024.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2310.18547

@inproceedings{chen2023punica,
  title = {{Punica: Multi-Tenant LoRA Serving}},
  author = {Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze and Arvind Krishnamurthy},
  booktitle = {MLSys 2024},
  year = {2023},
  eprint = {2310.18547},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18547}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. find
  2. Cai, T., Li, Y., Geng, Z., Peng, H., and Dao, T. Medusa, September 2023. URL https://github.com/FasterDecoding/Medusa. ↗
  3. Chen, L., Deng, W., Canumalla, A., Xin, Y., Philipose, M., and Krishnamurthy, A. Symphony: Optimized model serving using centralized orchestration, 2023. find
  4. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I. Clipper: A Low-Latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pp. 613–627, Boston, MA, March 2017. USENIX Association. ISBN 978-1-931971-3… find
  5. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. find
  6. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
  7. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323, 2022. doi: 10.48550/ARXIV.2210.17323. URL https://doi.org/10.48550/arXiv.2210.17323. ↗
  8. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462. USENIX Association, November 2020. ISB… find
  9. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1–15, 2023. find
  10. Guo, D., Rush, A. M., and Kim, Y. Parameter-efficient transfer learning with diff pruning. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
  11. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for NLP. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 J… find
  12. PMLR, 2019. URL http://proceedings.mlr.press/v97/houlsby19a.html. ↗
  13. Hsueh, B. Y. FasterTransformer. https://github.com/NVIDIA/FasterTransformer, 2021. ↗
  14. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
  15. OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  16. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pp. 611–626, New York, NY, USA, 202…
  17. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202… find
  18. PMLR, 2023. URL https://proceedings.mlr.press/v202/leviathan23a.html. ↗
  19. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. find
  20. Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. CoRR, abs/2305.17118, 2023a. doi: 10.48550/ARXIV.2305.17118. URL https://doi.org/10.48550/arXiv.2… — [doi:10.48550/arxiv.2305.17118](https://doi.org/10.48550/arxiv.2305.17118) ↗
  21. Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., and Chen, B. Deja vu: Contextual sparsity for efficient llms at inference time. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conferenc…
  22. PMLR, 2023b. URL https://proceedings.mlr.press/v202/liu23am.html. ↗
  23. Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022. ↗
  24. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating generative LLM serving with speculative inference and token tree verification. CoRR, abs/2305.09781, 2023. doi: 10.48550/ARXIV.2305.09781. URL https://doi.or… ↗
  25. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conf… find
  26. ShareGPT. ShareGPT: Share your wildest ChatGPT conversations with one click., 2023. URL https://sharegpt.com/. ↗
  27. Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., and Sundaram, R. Nexus: A gpu cluster engine for accelerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP ’19, pp. 322–337, New York, NY, USA, 2019. Associa… — [doi:10.1145/3341301.3359658](https://doi.org/10.1145/3341301.3359658) find
  28. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single GPU. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Internat…
  29. PMLR, 2023. URL https://proceedings.mlr.press/v202/sheng23a.html. ↗
  30. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053. ↗
  31. Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. Cutlass, 1 2023. URL https://github.com/NVIDIA/cutlass/tree/v3.0.0. ↗
  32. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  33. Williams, S., Waterman, A., and Patterson, D. Roofline: An insightful visual performance model for multi-core architectures. Commun. ACM, 52(4):65–76, apr 2009. ISSN 0001-0782. doi: 10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785. ↗
  34. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-the-art natural… — Transformers: State-of-the-Art Natural Language Processing
  35. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 202… find
  36. PMLR, 2023a. URL https://proceedings.mlr.press/v202/xiao23c.html. ↗
  37. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. CoRR, abs/2309.17453, 2023b. doi: 10.48550/ARXIV.2309.17453. URL https://doi.org/10.48550/arXiv.2309.17453. ↗
  38. Ye, Z. FlashInfer: Kernel Library for LLM Serving. https://github.com/flashinfer-ai/flashinfer, 2023. ↗
  39. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
  40. Zaken, E. B., Goldberg, Y., and Ravfogel, S. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short…
  41. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C. W., Wang, Z., and Chen, B. H2 o: Heavy-hitter oracle for efficient generative inference of large language models. CoRR, abs/2306.14048, 2023. doi: 10.48550/ARXIV.2306.14048. URL https://doi.org/10.…
  42. Zhao, M., Lin, T., Mi, F., Jaggi, M., and Schütze, H. Masking as an efficient alternative to finetuning for pretrained language models. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, No… find
  43. Zhou, Z., Wei, X., Zhang, J., and Sun, G. PetS: A unified framework for Parameter-Efficient transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489–504, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-29-11. URL https://www.usenix.org/conference/atc22… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-punica-multi-tenant-lora-serving/