Elamur Research⌕
☰
home / papers / 2024 / Vidur: A Large-Scale Simulation Framework For LLM Inference

Vidur: A Large-Scale Simulation Framework For LLM Inference

Paper · 2024 · Amey Agrawal, Nitin Kedia, Jayashree Mohan et al.

Summary

Deploying a large language model involves many choices: the parallelism layout (TP, PP, number of replicas), the scheduler (Orca, vLLM, Sarathi-Serve), batch size, scheduler-specific knobs and the GPU SKU. Normally, each candidate setting has to be tested by running real workloads on real GPUs, which is slow and costly. The authors also show that the best choice depends on the model and the request trace together, so the search must be repeated whenever either changes.

Vidur is a simulator that replaces most of these experiments. It represents an LLM as just a few kinds of operators: ones that act per token, ones that act per sequence, and ones that handle communication. Only a limited range of input sizes is profiled on hardware, and a runtime estimator predicts the rest. Using these per-iteration estimates, Vidur replays workloads and reports request metrics (TTFT, TBT, latency, throughput) and cluster metrics (MFU, memory use). Its latency predictions stay within about 9% of real measurements. Vidur-Search uses the simulator to find the cheapest configuration that meets latency targets, replacing weeks of GPU experiments with hours of CPU time.

Key points

  • Three components:
    • Vidur, the simulator.
    • Vidur-Bench, a set of workload traces and scheduler implementations (vLLM, Orca, FasterTransformer, Sarathi-Serve) with profiling data for A100 and H100 GPUs.
    • Vidur-Search, the configuration search tool.
    • The code is open-source at github.com/microsoft/vidur.
  • Evaluation covers LLaMA2-7B, InternLM-20B, LLaMA2-70B and Qwen-72B on three traces (Chat-1M, Arxiv-4K, BWB-4K).
    • Prediction error is under 9% in general.
    • Error reaches 12.65% only for LLaMA2-7B at 95% of maximum capacity, where CPU overheads cause errors to cascade.
  • For LLaMA2-70B, Vidur-Search finds the best configuration in about one hour on a 96-core CPU machine (about $9.93/hour on Azure). The equivalent deployment-based search would take about 42K GPU hours, or roughly $218K.
  • Reusing the optimal configuration from one trace on another trace can raise cost by up to 2× (LLaMA2-70B).
  • The full what-if study over 12 model-trace pairs required 35,565 runs.
    • Projected GPU cost on real hardware: about $1.14M.
    • Simulated cost: about 12.5 CPU hours for $125.
    • Per-scenario savings range from roughly 3,800× to 33,000×.

Interesting ideas

  • I1. Profile a few points and predict the rest. Vidur breaks models into shared operator types, measures only a minimal set of input sizes, and fills in the gaps with a learned estimator. This approach could make cost estimation feasible for any system with a huge input space, such as planning inference costs for Ukrainian NLP models.
  • I2. Test fidelity under heavy load. Small per-step errors compound in stateful, queue-driven systems, so the authors check accuracy close to saturation (75% and 95% of capacity), not only at light load. Any simulator or forecasting tool should be validated in the regime where its errors snowball.
  • I3. Optimise for a model-trace pair, not the model alone. The best setting shifts with the workload, so a benchmark should include several realistic traffic patterns. The same lesson applies to evaluating NLP pipelines on different text domains.
  • I4. Measure the cost of misconfiguration. Applying the optimum from one workload to another and reporting the cost ratio shows how much a one-size-fits-all choice loses. This is a simple and convincing evaluation that can be reused when tuning any system.
  • I5. Report search cost in dollars. Comparing simulated and real exploration in money and GPU hours makes the benefit concrete for decision-makers. This framing is useful whenever compute budgets are tight.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksHardware and Architecture · 167 worksComputer Networks and Communications · 100 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework For LLM Inference. arXiv:2405.05465.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.05465 · arXiv 2405.05465

@misc{agrawal2024vidur,
  title = {{Vidur: A Large-Scale Simulation Framework For LLM Inference}},
  author = {Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee and Alexey Tumanov},
  year = {2024},
  doi = {10.48550/arxiv.2405.05465},
  eprint = {2405.05465},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.05465}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. arxiv.org e-print archive. https://arxiv.org/. Cupti: Cuda toolkit documentation. https://docs.nvidia.com/cuda/cupti/index.html. Faster Transformer. https://github.com/NVIDIA/FasterTransformer. Google duet ai. https://workspace.google.com/solutions/ai/. Microsoft copilot. https://www.microsoft.com/e… ↗
  2. Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023.
  3. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in llm inference with sarathi-serve. 2024.
  4. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang,… — Qwen Technical Report
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  6. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  7. Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E. cudnn: Efficient primitives for deep learning, 2014.
  8. Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
  9. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. find
  10. Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-decoding for long-context inference, 2023. find
  11. Duan, J., Li, X., Xu, P., Zhang, X., Yan, S., Liang, Y., and Lin, D. Proteus: Simulating the performance of distributed DNN training. CoRR, abs/2306.02267, 2023. doi: 10.48550/arXiv.2306.02267. URL https://doi.org/10.48550/arXiv.2306.02267. — [doi:10.48550/arxiv.2306.02267](https://doi.org/10.48550/arxiv.2306.02267) ↗
  12. Hooper, C., Kim, S., Mohammadzadeh, H., Genc, H., Keutzer, K., Gholami, A., and Shao, S. Speed: Speculative pipelined execution for efficient decoding, 2023. find
  13. Jiang, Y. E., Liu, T., Ma, S., Zhang, D., Cotterell, R., and Sachan, M. Discourse centric evaluation of machine translation with a densely annotated parallel corpus. In Proceedings of the 2023 Conference of the Association for Computational Linguistics: Human Language Technologies, pp. 1550–1565, To… find
  14. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on…
  15. ACM, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165. — [doi:10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165) ↗
  16. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. September 2023. URL https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need-ii-phi-1-5-technical-report/. ↗
  17. Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D., and Stoica, I. Terapipe: Token-level pipeline parallelism for training large-scale language models, 2021. find
  18. Lin, Z., Feng, L., Ardestani, E. K., Lee, J., Lundell, J., Kim, C., Kejariwal, A., and Owens, J. D. Building a performance model for deep learning recommendation model training on gpus. In 29th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2022, Bengaluru, In… find
  19. IEEE, 2022. doi: 10.1109/HiPC56025.2022.00019. URL https://doi.org/10.1109/HiPC56025.2022.00019. — [doi:10.1109/hipc56025.2022.00019](https://doi.org/10.1109/hipc56025.2022.00019) ↗
  20. NVIDIA Corporation. CUBLAS library. https://docs.nvidia.com/cuda/cublas/index.html, a. NVIDIA Corporation. Matrix multiplication background user’s guide. https://docs.nvidia.com/deeplearning/ performance/dl-performance-matrix-multiplication/index.html, b. Patel, D. and Ahmed, A. The inference cost… ↗
  21. Patel, P., Choukse, E., Zhang, C., Goiri, Í., Shah, A., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting. arXiv preprint arXiv:2311.18677, 2023.
  22. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference, 2022. find
  23. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  24. Sivathanu, M., Chugh, T., Singapuram, S. S., and Zhou, L. Astra: Exploiting predictability to optimize deep learning. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’19, pp. 909–923, New York, NY, USA, 201… — [doi:10.1145/3297858.3304072](https://doi.org/10.1145/3297858.3304072) find
  25. Team, I. Internlm: A multilingual language model with progressively enhanced capabilities, 2023. find
  26. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a.
  27. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, v…
  29. Williams, S., Waterman, A., and Patterson, D. Roofline: An insightful visual performance model for multicore architectures. Commun. ACM, 52(4):65–76, apr 2009. ISSN 0001-0782. doi: 10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785. ↗
  30. Xiao, W., Bhardwaj, R., Ramjee, R., Sivathanu, M., Kwatra, N., Han, Z., Patel, P., Peng, X., Zhao, H., Zhang, Q., Yang, F., and Zhou, L. Gandiva: Introspective cluster scheduling for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 595–610, Carlsb… find
  31. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
  32. Yu, G. X., Gao, Y., Golikov, P., and Pekhimenko, G. Habitat: A runtime-based computational performance predictor for deep neural network training. In Calciu, I. and Kuenning, G. (eds.), 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021, pp. 503–521. find
  33. USENIX Association, 2021. URL https://www.usenix.org/conference/atc21/presentation/yu. ↗
  34. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023.
  35. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024.
  36. Zhu, H., Phanishayee, A., and Pekhimenko, G. Daydream: Accurately estimating the efficacy of optimizations for DNN training. In Gavrilovska, A. and Zadok, E. (eds.), 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, pp. 337–352. find
  37. USENIX Association, 2020. URL https://www.usenix.org/conference/atc20/presentation/zhu-hongyu. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agrawal-2024-vidur-large-scale-simulation-framework/