Elamur Research⌕
☰
home / papers / 2024 / DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Paper · 2024 · Jovan Stojkovic, Chaojie Zhang, Inigo Goiri et al.

Summary

Serving LLMs on GPUs under tight latency targets uses a lot of energy. Existing power-management schemes were built for conventional datacenter software and miss the specific traits of LLM inference. The authors first profile several open models on an NVIDIA DGX H100 server running vLLM, varying prompt and output lengths, load, model and SLO. They also analyze a week of Azure invocation traces from Coding and Conversation services. The cheapest configuration that still meets SLOs depends strongly on the request mix and keeps shifting over time. Building on this, they present DynamoLLM, an energy-management framework that continuously reconfigures the cluster using three knobs: instance count, tensor-parallelism degree and GPU frequency. Requests go to separate instance pools tuned for different length classes. Layered controllers divide the decisions, and reconfiguration costs are reduced or hidden. Replaying production traces on a large GPU cluster, DynamoLLM saves 53% energy and 38% operational carbon, and lowers customer cost by 61%, all within latency SLOs. Inference is a fast-growing source of datacenter power demand, which makes these savings important.

Key points

  • Profiled models: Llama2-13B, Mixtral-8x7B, Llama2-70B, Llama3-70B, Mixtral-8x22B and Falcon-180B. They ran on DGX H100 with vLLM, using TP2/TP4/TP8 and GPU clocks from 0.8 to 2.0 GHz.
  • Requests fall into 9 classes (SS … LL) by input/output length. The class cut-offs come from the 33rd, 66th and 100th percentiles of an Azure Conversation trace. TTFT/TBT SLOs are defined as five times the latency a request sees when it executes by itself on the system.
  • Prefill-heavy (long-input) requests are sensitive to GPU frequency. Short-input, long-output requests are memory-bound and can run at lower clocks with little slowdown. Larger models also respond differently to GPU count and frequency.
  • Energy per request also depends on system load. Load levels were tested at roughly 650, 2K and 4K tokens per second, so a configuration that is optimal at one moment can soon become wasteful.
  • Headline results on production-level traces: −53% energy, −38% operational carbon emissions, −61% customer cost, with SLOs met.

Interesting ideas

  • I1. Bucket requests by input and output size, and serve each bucket from its own tuned pool. Classifying jobs by shape before scheduling could help any batch or inference service with very mixed workloads.
  • I2. Data-driven bucket boundaries. Deriving class thresholds from trace percentiles, rather than picking round numbers, keeps the categories balanced and representative. The same trick is handy when designing benchmarks or evaluation splits.
  • I3. Pools that merge and split over time. Resizing pools as demand shifts trades fine-grained right-sizing against fragmentation. This is a useful pattern for any system that has to rebalance resources as demand changes.
  • I4. Layered controllers, each owning one knob. Each level makes its decision within limits set by the level above and passes constraints downward. This shrinks the search space and avoids a single central bottleneck, which is applicable to other multi-parameter tuning problems.
  • I5. SLOs defined relative to unloaded latency. Setting targets as a multiple of the latency of an isolated request gives a simple, model-agnostic way to compare configurations fairly across models and request types.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksHardware and Architecture · 167 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jovan Stojkovic, Chaojie Zhang, Inigo Goiri, Josep Torrellas and Esha Choukse. 2024. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. arXiv:2408.00741.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/hpca61900.2025.00102 · arXiv 2408.00741

@misc{stojkovic2024dynamollm,
  title = {{DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency}},
  author = {Jovan Stojkovic, Chaojie Zhang, Inigo Goiri, Josep Torrellas and Esha Choukse},
  year = {2024},
  doi = {10.1109/hpca61900.2025.00102},
  eprint = {2408.00741},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/hpca61900.2025.00102}
}

References

82 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. “CAISO carbon emission,” https://www.caiso.com/todays-outlook/emissions, 2024. ↗
  2. “WattTime,” https://watttime.org, 2024. ↗
  3. A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills,” 2023.
  4. K. Alizadeh, I. Mirzadeh, D. Belenko, K. Khatamifard, M. Cho, C. C. D. Mundo, M. Rastegari, and M. Farajtabar, “LLM in a flash: Efficient Large Language Model Inference with Limited Memory,” 2024.
  5. J. Bowne, “Using large language models in learning and teaching,” https://biomedicalsciences.unimelb.edu.au/study/dlh/assets/documents/large-language-models-in-education/llms-in-education, 2024. ↗
  6. S. Chen, C. Delimitrou, and J. F. Martı́nez, “PARTIES: QoS-Aware Resource Partitioning for Multiple Interactive Services,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19), 2019. find
  7. S. Chen, A. Jin, C. Delimitrou, and J. F. Martı́nez, “ReTail: Opting for Learning Simplicity to Enable QoS-Aware Power Management in the Cloud,” in Proceedings of the IEEE International Symposium on High-Performance Computer Architecture (HPCA ’22), 2022. find
  8. Cloud Price, “Standard ND96is H100 v5,” https://cloudprice.net/vm/Standard ND96is H100 v5, 2024. ↗
  9. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” 2022. find
  10. C. Delimitrou and C. Kozyrakis, “Amdahl’s Law for Tail Latency,” Commun. ACM, vol. 61, no. 8, jul 2018. find
  11. D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruehle, L. V. S. Lakshmanan, and A. Awadallah, “Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing,” in Proceedings of the International Conference on Learning Representations (ICLR ’24), 2024.
  12. A. Faiz, S. Kaneda, R. Wang, R. Osi, P. Sharma, F. Chen, and L. Jiang, “LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models,” arXiv preprint arXiv:2309.14393, 2024. find
  13. GitHub, “The world’s most widely adopted ai developer tool,” https://github.com/features/copilot, 2024. ↗
  14. M. Halpern, B. Boroujerdian, T. Mummert, E. Duesterwald, and V. Janapa Reddi, “One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-Off in Machine Learning Cloud Service APIs via Tolerance Tiers,” in Proceedings of the IEEE International Symposium on Performance Analysis of… find
  15. M. E. Haque, Y. He, S. Elnikety, T. D. Nguyen, R. Bianchini, and K. S. McKinley, “Exploiting Heterogeneity for Tail Latency and Energy Efficiency,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’17), 2017. find
  16. C.-H. Hsu, Q. Deng, J. Mars, and L. Tang, “SmoothOperator: Reducing Power Fragmentation and Improving Power Utilization in Large-Scale Datacenters,” in Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’18), 2018. find
  17. C.-H. Hsu, Y. Zhang, M. A. Laurenzano, D. Meisner, T. Wenisch, J. Mars, L. Tang, and R. G. Dreslinski, “Adrenaline: Pinpointing and reining in tail queries with quick voltage boosting,” in Proceedings of the IEEE 21st International Symposium on High Performance Computer Architecture (HPCA ’15), 2015… find
  18. A. Jahanshahi, M. Rezvani, and D. Wong, “WattWiser: Power and Resource-Efficient Scheduling for Multi-Model Multi-GPU Inference Servers,” in Proceedings of the 14th International Green and Sustainable Computing Conference (IGSC ’23), 2023. find
  19. Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, “ S 3 : Increasing GPU utilization during generative inference for higher throughput,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS ’23), 2023. find
  20. A. K. Kakolyris, D. Masouros, S. Xydis, and D. Soudris, “SLO-aware GPU DVFS for Energy-efficient LLM Inference Serving,” IEEE Computer Architecture Letters, 2024. find
  21. H. Kasture, D. B. Bartolini, N. Beckmann, and D. Sanchez, “Rubik: Fast analytical power management for latency-critical systems,” in Proceedings of the 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’15), 2015. find
  22. Y. G. Kim and C.-J. Wu, “AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning,” in Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’20), 2020. find
  23. T. Komoda, S. Hayashi, T. Nakada, S. Miwa, and H. Nakamura, “Power capping of cpu-gpu heterogeneous systems through coordinating dvfs and task mapping,” in Proceedings of the IEEE 31st International Conference on Computer Design (ICCD ’13), 2013. find
  24. N. Kulkarni, G. Gonzalez-Pumariega, A. Khurana, C. A. Shoemaker, C. Delimitrou, and D. H. Albonesi, “CuttleSys: Data-Driven Resource Management for Interactive Services on Reconfigurable Multicores,” in Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’20),… find
  25. A. G. Kumbhare, R. Azimi, I. Manousakis, A. Bonde, F. Frujeri, N. Mahalingam, P. A. Misra, S. A. Javadi, B. Schroeder, M. Fontoura, and R. Bianchini, “Prediction-Based Power Oversubscription in Cloud Platforms,” in Proceedings of the USENIX Annual Technical Conference (USENIX ATC ’21), 2021. find
  26. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), 2023.
  27. M. Lammertyn, “60+ ChatGPT Statistics And Facts You Need to Know in 2024,” https://blog.invgate.com/chatgpt-statistics, 2024. ↗
  28. LCG Consulting, “Energy Online: ERCOT Real Time Price,” https://energyonline.com/Data/GenericData.aspx?DataId=4&ERCOT Real-time Price, 2024. ↗
  29. S. Li, X. Wang, X. Zhang, V. Kontorinis, S. Kodakara, D. Lo, and P. Ranganathan, “Thunderbolt: Throughput-Optimized, Quality-of-Service-Aware Power Capping at Scale,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20), 2020. find
  30. Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica, “AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,” in Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’2…
  31. D. Lo, L. Cheng, R. Govindaraju, L. A. Barroso, and C. Kozyrakis, “Towards energy proportionality for large-scale latency-critical workloads,” in Proceedings of the ACM/IEEE 41st International Symposium on Computer Architecture (ISCA ’14), 2014. find
  32. Meta, “Llama2-13B,” https://huggingface.co/meta-llama/Llama-2-13b, 2024. ↗
  33. Meta, “Llama2-70B,” https://huggingface.co/meta-llama/Llama-2-70b, 2024. ↗
  34. Meta, “Llama3-70B,” https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct, 2024. ↗
  35. X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “SpotServe: Serving Generative Large Language Models on Preemptible Instances,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’24), 2024.
  36. Microsoft Azure, “ND H100 v5-series,” https://learn.microsoft.com/en-us/azure/virtual-machines/nd-h100-v5-series, 2024. ↗
  37. A. Mirhosseini and T. Wenisch, “µSteal: A Theory-Backed Framework for Preemptive Work and Resource Stealing in Mixed-Criticality Microservices,” in Proceedings of the ACM International Conference on Supercomputing (ICS ’21), 2021. find
  38. Mistral AI, “The Mixtral-8x22B Large Language Model,” https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1, 2024. ↗
  39. Mistral AI, “The Mixtral-8x7B Large Language Model,” https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1, 2024. ↗
  40. S. M. Nabavinejad, S. Reda, and M. Ebrahimi, “BatchSizer: Power-Performance Trade-off for DNN Inference,” in Proceedings of the 26th Asia and South Pacific Design Automation Conference (ASP-DAC ’21), 2021. find
  41. S. M. Nabavinejad, S. Reda, and M. Ebrahimi, “Coordinated batching and dvfs for dnn inference on gpu accelerators,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 10, 2022. find
  42. K. K. W. Ng, H. M. Demoulin, and V. Liu, “Paella: Low-latency Model Serving with Software-defined GPU Scheduling,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), 2023. find
  43. R. Nishtala, V. Petrucci, P. Carpenter, and M. Sjalander, “Twig: Multi-Agent Task Management for Colocated Latency-Critical Cloud Services,” in Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA ’20), 2020. find
  44. NVIDIA, “DGX H100: AI for Enterprise,” https://www.nvidia.com/en-us/data-center/dgx-h100/, 2024. ↗
  45. NVIDIA, “NVLink and NVLink Switch,” https://www.nvidia.com/en-us/data-center/nvlink/, 2024. ↗
  46. NVIDIA, “System Management Interface SMI,” https://developer.nvidia.com/system-management-interface, 2024. ↗
  47. NVIDIA, “TensorRT-LLM’s Documentation,” https://nvidia.github.io/TensorRT-LLM/, 2024. ↗
  48. H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’24), 2024. find
  49. P. Patel, E. Choukse, C. Zhang, I. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing Power Management Opportunities for LLMs in the Cloud,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’24),… find
  50. P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA ’24), 2024.
  51. T. Patel and D. Tiwari, “CLITE: Efficient and QoS-Aware Co-Location of Multiple Latency-Critical Jobs for Warehouse Scale Computers,” in Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA ’20), 2020. find
  52. C. Peng, X. Yang, A. Chen, K. Smith, N. PourNejatian, A. Costa, C. Martin, M. Flores, Y. Zhang, T. Magoc, G. Lipori, M. Duane, N. Ospina, M. Ahmed, W. Hogan, E. Shenkman, Y. Guo, J. Bian, and Y. Wu, “A study of generative large language model for medical research and healthcare,” npj Digital Medicin… find
  53. Python PuLP, “Optimization with PuLP,” https://coin-or.github.io/pulp/, 2024. ↗
  54. Python SciPy, “SciPy Library,” https://scipy.org/, 2024. ↗
  55. H. Qiu, W. Mao, A. Patke, S. Cui, S. Jha, C. Wang, H. Franke, Z. T. Kalbarczyk, T. Başar, and R. K. Iyer, “Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction,” in The 5th International Workshop on Cloud Intelligence / AIOps at ASPLOS 2024, 2024. find
  56. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  57. F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “INFaaS: Automated Model-less Inference Serving,” in Proceedings of the USENIX Annual Technical Conference (USENIX ATC ’21), 2021. find
  58. S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2023, pp. 1–9.
  59. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al., “BLOOM: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022. find
  60. J. Stojkovic, E. Choukse, C. Zhang, I. Goiri, and J. Torrellas, “Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference,” arXiv e-prints, p. arXiv:2403.20306, Mar. 2024. find
  61. J. Stojkovic, N. Iliakopoulou, T. Xu, H. Franke, and J. Torrellas, “EcoFaaS: Rethinking the Design of Serverless Environments for Energy Efficiency,” in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA ’24), 2024. find
  62. J. Stojkovic, P. Misra, I. Goiri, S. Whitlock, E. Choukse, M. Das, C. Bansal, J. Lee, Z. Sun, H. Qiu, R. Zimmermann, S. Samal, B. Warrier, A. Raniwala, and R. Bianchini, “SmartOClock: Workload- and Risk-Aware Overclocking in the Cloud,” in Proceedings of the 51st Annual International Symposium on Co… find
  63. K. Talamadupula, “A Guide to LLM Inference Performance Monitoring,” https://symbl.ai/developers/blog/a-guide-to-llm-inference-performance-monitoring/, 2024. ↗
  64. T. Tambe, C. Hooper, L. Pentecost, T. Jia, E.-Y. Yang, M. Donato, V. Sanh, P. Whatmough, A. M. Rush, D. Brooks, and G.-Y. Wei, “Edge-BERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference,” in Proceedings of the 54th Annual IEEE/ACM International Symposium on Microarchi… find
  65. Z. Tang, Y. Wang, Q. Wang, and X. Chu, “The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study,” in Proceedings of the Tenth ACM International Conference on Future Energy Systems (e-Energy ’19), 2019. find
  66. Technology Innovation Institute (TII), “Falcon-180B,” 2024. find
  67. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
  68. T. Varshney, “Build an llm-powered data agent for data analysis,” https://developer.nvidia.com/blog/build-an-llm-powered-data-agent-for-data-analysis/, 2024. ↗
  69. C. Wan, M. Santriaji, E. Rogers, H. Hoffmann, M. Maire, and S. Lu, “ALERT: Accurate learning for energy and timeliness,” in Proceedings of the USENIX Annual Technical Conference (USENIX ATC ’20), 2020. find
  70. F. Wang, W. Zhang, S. Lai, M. Hao, and Z. Wang, “Dynamic GPU Energy Optimization for Machine Learning Training Workloads,” IEEE Transactions on Parallel and Distributed Systems, 2022. find
  71. W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia, “AntMan: Dynamic scaling on GPU clusters for deep learning,” in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20), 2020. find
  72. J. You, J.-W. Chung, and M. Chowdhury, “Zeus: Understanding and optimizing GPU energy consumption of DNN training,” in Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’23), 2023.
  73. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models,” in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’22), 2022.
  74. J. Yu, J. Kim, and E. Seo, “Know Your Enemy To Save Cloud Energy: Energy-Performance Characterization of Machine Learning Serving,” in 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2023. find
  75. C. Zhang, A. G. Kumbhare, I. Manousakis, D. Zhang, P. A. Misra, R. Assis, K. Woolcock, N. Mahalingam, B. Warrier, D. Gauthier, L. Kunnath, S. Solomon, O. Morales, M. Fontoura, and R. Bianchini, “Flex: High-Availability Datacenters with Zero Reserved Power,” in Proceedings of the 48th Annual Internat… find
  76. Y. Zhang, W. Hua, Z. Zhou, G. E. Suh, and C. Delimitrou, “Sinan: ML-Based and QoS-Aware Resource Management for Cloud Microservices,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’21), 2021. find
  77. Y. Zhang, Q. Wang, Z. Lin, P. Xu, and B. Wang, “Improving gpu energy efficiency through an application-transparent frequency scaling policy with performance assurance,” in Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24), 2024. find
  78. Y. Z. Zhao, D. W. Wu, and J. Wang, “ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching,” in Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA ’24), 2024. find
  79. Z. Zheng, X. Ren, F. Xue, Y. Luo, X. Jiang, and Y. You, “Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS ’23), 2023. find
  80. L. Zhou, L. N. Bhuyan, and K. K. Ramakrishnan, “Gemini: Learning to Manage CPU Power for Latency-Critical Search Engines,” in Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’20), 2020. find
  81. Z. Zhou, X. Wei, J. Zhang, and G. Sun, “PetS: A unified framework for Parameter-Efficient transformers serving,” in Proceedings of the USENIX Annual Technical Conference (USENIX ATC ’22), 2022. find
  82. P. Zou, A. Li, K. Barker, and R. Ge, “Indicator-Directed Dynamic Power Management for Iterative Workloads on GPU-Accelerated Systems,” in Proceedings of the 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID ’20), 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stojkovic-2024-dynamollm-designing-llm-inference-clusters/