Elamur Research⌕
☰
home / papers / 2024 / A Survey on Efficient Inference for Large Language Models

A Survey on Efficient Inference for Large Language Models

Paper · 2024 · Zixuan Zhou, Xuefei Ning, Ke Hong et al.

Summary

Running large language models is expensive. They need a lot of computation, memory traffic and storage, which raises latency, lowers throughput and makes deployment difficult on edge devices and costly in the cloud. This survey, by authors from Tsinghua University, Shanghai Jiaotong University and Infinigence-AI, reviews research on making LLM inference cheaper. It first identifies three root causes of inefficiency: very large parameter counts, attention whose cost grows quadratically with sequence length, and token-by-token auto-regressive generation. It then sorts existing techniques into three levels of optimization: data, model and system. Other surveys typically cover only model compression or a systems view. This one spans all three levels and adds its own comparative experiments on representative methods in key sub-fields such as quantization and serving systems, from which it draws practical recommendations. It also discusses specific application scenarios and outlines open research directions. The work offers practitioners a single map of the field and evidence for choosing between techniques.

Key points

  • The taxonomy has three tiers: data-level, model-level and system-level optimization. It is contrasted in Table 1 with earlier surveys, including Zhu et al., Wan et al. and Xu et al.
  • Inference is modeled as two phases. In the prefilling stage, the KV cache for the prompt is built and the first token is produced. In the decoding stage, tokens are generated one at a time while the KV cache is extended.
  • The survey defines a set of efficiency metrics:
    • first token latency, per-output token latency and generation latency;
    • model size, KV cache size and peak memory;
    • token throughput and request throughput.
  • A motivating example is LLaMA-2-70B. In FP16 its weights take about 140 GB of VRAM, which requires at least 6 RTX 3090Ti or 2 NVIDIA A100 (80 GB) GPUs. Each token takes roughly 100 ms on two A100s, so outputs of a few hundred tokens take more than 10 seconds.
  • Comparative experiments cover representative methods in sub-fields such as model quantization and LLM serving systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Peking UniversityBeijing · China315 works · 43,267 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong and Yu Wang. 2024. A Survey on Efficient Inference for Large Language Models. arXiv:2404.14294.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.14294 · arXiv 2404.14294

@misc{zhou2024survey,
  title = {{A Survey on Efficient Inference for Large Language Models}},
  author = {Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong and Yu Wang},
  year = {2024},
  doi = {10.48550/arxiv.2404.14294},
  eprint = {2404.14294},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.14294}
}

References

313 references, 110 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018. find
  2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  4. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022.
  5. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
  6. A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan et al., “Baichuan 2: Open large-scale language models,” arXiv preprint arXiv:2309.10305, 2023.
  7. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna.lmsys. org (accessed 14 April 2023), 2023. ↗
  8. D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, and H. Zhang, “How long can context length of open-source llms truly promise?” in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. find
  9. B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon et al., “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022. find
  10. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic et al., “The falcon series of open language models,” arXiv preprint arXiv:2311.16867, 2023.
  11. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” arXiv preprint arXiv:2103.10360, 2021.
  12. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. find
  13. J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” ACM Transactions on Knowledge Discovery from Data, 2023.
  14. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  15. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  16. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023.
  17. X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “A survey on model compression for large language models,” arXiv preprint arXiv:2308.07633, 2023.
  18. S. Park, J. Choi, S. Lee, and U. Kang, “A comprehensive survey of compression algorithms for language models,” arXiv preprint arXiv:2401.15347, 2024. — [arXiv:2401.15347](https://arxiv.org/abs/2401.15347) find
  19. W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He, “Model compression and efficient inference for large language models: A survey,” arXiv preprint arXiv:2402.09748, 2024. — [arXiv:2402.09748](https://arxiv.org/abs/2402.09748)
  20. Y. Tang, Y. Wang, J. Guo, Z. Tu, K. Han, H. Hu, and D. Tao, “A survey on transformer compression,” arXiv preprint arXiv:2402.05964, 2024. — [arXiv:2402.05964](https://arxiv.org/abs/2402.05964) find
  21. T. Ding, T. Chen, H. Zhu, J. Jiang, Y. Zhong, J. Zhou, G. Wang, Z. Zhu, I. Zharkov, and L. Liang, “The efficiency spectrum of large language models: An algorithmic survey,” arXiv preprint arXiv:2312.00678, 2023. — [arXiv:2312.00678](https://arxiv.org/abs/2312.00678) find
  22. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” arXiv preprint arXiv:2312.15234, 2023.
  23. Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury et al., “Efficient large language models: A survey,” arXiv preprint arXiv:2312.03863, vol. 1, 2023.
  24. M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y. Zhao, C. Yang, S. Wang et al., “A survey of resource-efficient llm and multimodal foundation models,” arXiv preprint arXiv:2401.08092, 2024. find
  25. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023.
  26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  27. Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan et al., “Llm inference unveiled: Survey and roofline model insights,” arXiv preprint arXiv:2402.16363, 2024.
  28. A. Golden, S. Hsia, F. Sun, B. Acun, B. Hosmer, Y. Lee, Z. DeVito, J. Johnson, G.-Y. Wei, D. Brooks et al., “Is flash attention stable?” arXiv preprint arXiv:2405.02803, 2024. — [arXiv:2405.02803](https://arxiv.org/abs/2405.02803) find
  29. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459– 9474, 2020.
  30. A. Chevalier, A. Wettig, A. Ajith, and D. Chen, “Adapting language models to compress contexts,” arXiv preprint arXiv:2305.14788, 2023.
  31. W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W. tau Yih, “Replug: Retrieval-augmented black-box language models,” 2023.
  32. A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learning to retrieve, generate, and critique through self-reflection,” 2023.
  33. D. Wingate, M. Shoeybi, and T. Sorensen, “Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models,” arXiv preprint arXiv:2210.03162, 2022. find
  34. J. Mu, X. L. Li, and N. Goodman, “Learning to compress prompts with gist tokens,” arXiv preprint arXiv:2304.08467, 2023.
  35. T. Ge, J. Hu, X. Wang, S.-Q. Chen, and F. Wei, “In-context autoencoder for context compression in a large language model,” arXiv preprint arXiv:2307.06945, 2023.
  36. F. Xu, W. Shi, and E. Choi, “Recomp: Improving retrieval-augmented lms with compression and selective augmentation,” arXiv preprint arXiv:2310.04408, 2023.
  37. W. Fei, X. Niu, P. Zhou, L. Hou, B. Bai, L. Deng, and W. Han, “Extending context window of large language models via semantic compression,” arXiv preprint arXiv:2312.09571, 2023. find
  38. W. Zhou, Y. E. Jiang, R. Cotterell, and M. Sachan, “Efficient prompting via dynamic in-context learning,” arXiv preprint arXiv:2305.11170, 2023. — [arXiv:2305.11170](https://arxiv.org/abs/2305.11170) find
  39. Y. Li, B. Dong, F. Guerin, and C. Lin, “Compressing context to enhance inference efficiency of large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6342–6353.
  40. F. Yin, J. Vig, P. Laban, S. Joty, C. Xiong, and C.-S. J. Wu, “Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning,” arXiv preprint arXiv:2306.01150, 2023. find
  41. H. Jung and K.-J. Kim, “Discrete prompt compression with reinforcement learning,” arXiv preprint arXiv:2308.08758, 2023. — [arXiv:2308.08758](https://arxiv.org/abs/2308.08758) find
  42. H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu, “Llmlingua: Compressing prompts for accelerated inference of large language models,” in The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  43. H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu, “Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression,” arXiv preprint arXiv:2310.06839, 2023.
  44. X. Huang, L. L. Zhang, K.-T. Cheng, and M. Yang, “Boosting llm reasoning: Push the limits of few-shot learning with reinforced in-context pruning,” arXiv preprint arXiv:2312.08901, 2023. — [arXiv:2312.08901](https://arxiv.org/abs/2312.08901) find
  45. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022.
  46. X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4582–4597. find
  47. X. Ning, Z. Lin, Z. Zhou, H. Yang, and Y. Wang, “Skeleton-of-thought: Large language models can do parallel decoding,” arXiv preprint arXiv:2307.15337, 2023.
  48. S. Jin, Y. Wu, H. Zheng, Q. Zhang, M. Lentz, Z. M. Mao, A. Prakash, F. Qian, and D. Zhuo, “Adaptive skeleton graph decoding,” arXiv preprint arXiv:2402.12280, 2024. — [arXiv:2402.12280](https://arxiv.org/abs/2402.12280) find
  49. M. Liu, A. Zeng, B. Wang, P. Zhang, J. Tang, and Y. Dong, “Apar: Llms can do auto-parallel auto-regressive decoding,” arXiv preprint arXiv:2401.06761, 2024. — [arXiv:2401.06761](https://arxiv.org/abs/2401.06761) find
  50. T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” 2024.
  51. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626.
  52. L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al., “Efficiently programming large language models using sglang,” arXiv preprint arXiv:2312.07104, 2023.
  53. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
  54. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk et al., “Graph of thoughts: Solving elaborate problems with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024… find
  55. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864, 2023.
  56. Q. Sun, Z. Yin, X. Li, Z. Wu, X. Qiu, and L. Kong, “Corex: Pushing the boundaries of complex reasoning through multi-model collaboration,” arXiv preprint arXiv:2310.00280, 2023. find
  57. T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” arXiv preprint arXiv:2402.01680, 2024.
  58. L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176, 2023.
  59. Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey, “What makes convolutional models great on long sequence modeling?” arXiv preprint arXiv:2210.09298, 2022. — [arXiv:2210.09298](https://arxiv.org/abs/2210.09298) find
  60. D. W. Romero, A. Kuzina, E. J. Bekkers, J. M. Tomczak, and M. Hoogendoorn, “Ckconv: Continuous kernel convolution for sequential data,” arXiv preprint arXiv:2102.02611, 2021. — [arXiv:2102.02611](https://arxiv.org/abs/2102.02611) find
  61. M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré, “Hyena hierarchy: Towards larger convolutional language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 043–28 078. find
  62. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023. find
  63. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” arXiv preprint arXiv:2307.08621, 2023.
  64. A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, “Hippo: Recurrent memory with optimal polynomial projections,” Advances in neural information processing systems, vol. 33, pp. 1474–1487, 2020. find
  65. A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré, “Combining recurrent, convolutional, and continuous-time models with linear state space layers,” Advances in neural information processing systems, vol. 34, pp. 572–585, 2021. find
  66. A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396, 2021. find
  67. A. Gupta, A. Gu, and J. Berant, “Diagonal state spaces are as effective as structured state spaces,” Advances in Neural Information Processing Systems, vol. 35, pp. 22 982–22 994, 2022. find
  68. A. Gu, K. Goel, A. Gupta, and C. Ré, “On the parameterization and initialization of diagonal state space models,” Advances in Neural Information Processing Systems, vol. 35, pp. 35 971–35 983, 2022. find
  69. H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur, “Long range language modeling via gated state spaces,” in International Conference on Learning Representations, 2023. find
  70. D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré, “Hungry hungry hippos: Towards language modeling with state space models,” arXiv preprint arXiv:2212.14052, 2022. find
  71. R. Hasani, M. Lechner, T.-H. Wang, M. Chahine, A. Amini, and D. Rus, “Liquid structural state-space models,” arXiv preprint arXiv:2209.12951, 2022. — [arXiv:2209.12951](https://arxiv.org/abs/2209.12951) find
  72. J. T. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” arXiv preprint arXiv:2208.04933, 2022. — [arXiv:2208.04933](https://arxiv.org/abs/2208.04933) find
  73. J. Pilault, M. Fathi, O. Firat, C. Pal, P.-L. Bacon, and R. Goroshin, “Block-state transformers,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  74. J. Wang, J. N. Yan, A. Gu, and A. M. Rush, “Pretraining without attention,” arXiv preprint arXiv:2212.10544, 2022. find
  75. A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023. find
  76. J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos, “Can mamba learn how to learn? a comparative study on in-context learning tasks,” arXiv preprint arXiv:2402.04248, 2024. find
  77. N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019. find
  78. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” arXiv preprint arXiv:2305.13245, 2023.
  79. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020. find
  80. G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu, and P. Fung, “Lightweight and efficient end-to-end speech recognition using low-rank transformer,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6144–6148. find
  81. A. Gupta, Y. Yuan, Y. Zhou, and C. Mendis, “Flurka: Fast fused low-rank & kernel attention,” arXiv preprint arXiv:2306.15799, 2023. — [arXiv:2306.15799](https://arxiv.org/abs/2306.15799) find
  82. X. Ma, X. Kong, S. Wang, C. Zhou, J. May, H. Ma, and L. Zettlemoyer, “Luna: Linear unified nested attention,” Advances in Neural Information Processing Systems, vol. 34, pp. 2441–2453, 2021. find
  83. J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in International conference on machine learning. PMLR, 2019, pp. 3744–3753. find
  84. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in International conference on machine learning. PMLR, 2020, pp. 5156–5165. find
  85. K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser et al., “Rethinking attention with performers,” in International Conference on Learning Representations, 2020.
  86. H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong, “Random feature attention,” in International Conference on Learning Representations, 2022. find
  87. P. Kacham, V. Mirrokni, and P. Zhong, “Polysketchformer: Fast transformers via sketches for polynomial kernels,” arXiv preprint arXiv:2310.01655, 2023. — [arXiv:2310.01655](https://arxiv.org/abs/2310.01655) find
  88. W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” The Journal of Machine Learning Research, vol. 23, no. 1, pp. 5232– 5270, 2022. find
  89. Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 877–890.
  90. Z.-F. Gao, P. Liu, W. X. Zhao, Z.-Y. Lu, and J.-R. Wen, “Parameter-efficient mixture-of-experts architecture for pre-trained language models,” in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 3263–3273. find
  91. A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” arXiv preprint arXiv:2212.05055, 2022.
  92. M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in International Conference on Machine Learning. PMLR, 2021, pp. 6265–6274.
  93. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon et al., “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems, vol. 35, pp. 7103–7114, 2022. find
  94. B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus, “St-moe: Designing stable and transferable sparse expert models,” arXiv preprint arXiv:2202.08906, 2022. find
  95. D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei, “Stablemoe: Stable routing strategy for mixture of experts,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 7085–7095. find
  96. T. Chen, Z. Zhang, A. K. JAISWAL, S. Liu, and Z. Wang, “Sparse moe as the new dropout: Scaling dense and self-slimmable transformers,” in The Eleventh International Conference on Learning Representations, 2022. find
  97. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in International Conference on Machine Learning. PMLR, 2022, pp. 5547–5569.
  98. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in International Conference on Learning Representations, 2016. find
  99. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. find
  100. C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al., “Tutel: Adaptive mixture-of-experts at scale,” Proceedings of Machine Learning and Systems, vol. 5, 2023.
  101. D. P. Bertsekas, “Auction algorithms for network flow problems: A tutorial introduction,” Computational optimization and applications, vol. 1, pp. 7–66, 1992. find
  102. Z. Dai, G. Lai, Y. Yang, and Q. Le, “Funnel-transformer: Filtering out sequential redundancy for efficient language processing,” Advances in neural information processing systems, vol. 33, pp. 4271– 4282, 2020. find
  103. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision mamba: Efficient visual representation learning with bidirectional state space model,” arXiv preprint arXiv:2401.09417, 2024.
  104. W. Hua, Z. Dai, H. Liu, and Q. Le, “Transformer quality in linear time,” in International Conference on Machine Learning. PMLR, 2022, pp. 9099–9117.
  105. AI21, “Jamba: Ai21’s groundbreaking ssm-transformer model,” March 2024. [Online]. Available: https://www.ai21.com/blog/announcing-jamba ↗
  106. W. He, K. Han, Y. Tang, C. Wang, Y. Yang, T. Guo, and Y. Wang, “Densemamba: State space models with dense hidden connection for efficient large language models,” arXiv preprint arXiv:2403.00818, 2024. — [arXiv:2403.00818](https://arxiv.org/abs/2403.00818) find
  107. Q. Anthony, Y. Tokpanov, P. Glorioso, and B. Millidge, “Blackmamba: Mixture of experts for state-space models,” arXiv preprint arXiv:2402.01771, 2024. — [arXiv:2402.01771](https://arxiv.org/abs/2402.01771) find
  108. M. Pióro, K. Ciebiera, K. Król, J. Ludziejewski, and S. Jaszczur, “Moe-mamba: Efficient selective state space models with mixture of experts,” arXiv preprint arXiv:2401.04081, 2024. — [arXiv:2401.04081](https://arxiv.org/abs/2401.04081) find
  109. S. Zhai, W. Talbott, N. Srivastava, C. Huang, H. Goh, R. Zhang, and J. Susskind, “An attention free transformer,” arXiv preprint arXiv:2105.14103, 2021. — [arXiv:2105.14103](https://arxiv.org/abs/2105.14103) find
  110. T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler, “Confident adaptive language modeling,” Advances in Neural Information Processing Systems, vol. 35, pp. 17 456–17 472, 2022. find
  111. L. Del Corro, A. Del Giorno, S. Agarwal, B. Yu, A. Awadallah, and S. Mukherjee, “Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference,” arXiv preprint arXiv:2307.02628, 2023. — [arXiv:2307.02628](https://arxiv.org/abs/2307.02628) find
  112. W. Liu, P. Zhou, Z. Wang, Z. Zhao, H. Deng, and Q. Ju, “Fastbert: a self-distilling bert with adaptive inference time,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 6035–6044.
  113. J. Kong, J. Wang, L.-C. Yu, and X. Zhang, “Accelerating inference for pretrained language models by unified multi-perspective early exiting,” in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 4677–4686. find
  114. K. Liao, Y. Zhang, X. Ren, Q. Su, X. Sun, and B. He, “A global past-future early exit method for accelerating inference of pre-trained language models,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  115. J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, “Deebert: Dynamic early exiting for accelerating bert inference,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 2246–2251.
  116. W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, “Bert loses patience: Fast and robust inference with early exit,” Advances in Neural Information Processing Systems, vol. 33, pp. 18 330–18 341, 2020. find
  117. T. Sun, X. Liu, W. Zhu, Z. Geng, L. Wu, Y. He, Y. Ni, G. Xie, X.-J. Huang, and X. Qiu, “A simple hash-based early exiting approach for language understanding and generation,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2409– 2421. find
  118. Y. Huang, Y. Chen, Z. Yu, and K. McKeown, “In-context learning distillation: Transferring few-shot learning ability of pre-trained language models,” arXiv preprint arXiv:2212.10670, 2022. — [arXiv:2212.10670](https://arxiv.org/abs/2212.10670) find
  119. J. Zhao, W. Zhao, A. Drozdov, B. Rozonoyer, M. A. Sultan, J.-Y. Lee, M. Iyyer, and A. McCallum, “Multistage collaborative knowledge distillation from large language models,” arXiv preprint arXiv:2311.08640, 2023. — [arXiv:2311.08640](https://arxiv.org/abs/2311.08640) find
  120. C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister, “Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,” arXiv preprint arXiv:2305.02301, 2023.
  121. L. H. Li, J. Hessel, Y. Yu, X. Ren, K.-W. Chang, and Y. Choi, “Symbolic chain-of-thought distillation: Small models can also” think” step-by-step,” arXiv preprint arXiv:2306.14050, 2023.
  122. L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn, “Teaching small language models to reason,” arXiv preprint arXiv:2212.08410, 2022.
  123. H. Chen, S. Wu, X. Quan, R. Wang, M. Yan, and J. Zhang, “Mcc-kd: Multi-cot consistent knowledge distillation,” arXiv preprint arXiv:2310.14747, 2023. find
  124. N. Ho, L. Schmid, and S.-Y. Yun, “Large language models are reasoning teachers,” arXiv preprint arXiv:2212.10071, 2022.
  125. K. Shridhar, A. Stolfo, and M. Sachan, “Distilling reasoning capabilities into smaller language models,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 7059– 7073.
  126. X. Zhu, B. Qi, K. Zhang, X. Long, and B. Zhou, “Pad: Program-aided distillation specializes large models in reasoning,” arXiv preprint arXiv:2305.13888, 2023. find
  127. P. Wang, Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren, “Scott: Self-consistent chain-of-thought distillation,” arXiv preprint arXiv:2305.01879, 2023.
  128. Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson, “Disco: distilling counterfactuals with large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 5514– 5528. find
  129. M. Wu, A. Waheed, C. Zhang, M. Abdul-Mageed, and A. F. Aji, “Lamini-lm: A diverse herd of distilled models from large-scale instructions,” arXiv preprint arXiv:2304.14402, 2023.
  130. Y. Jiang, C. Chan, M. Chen, and W. Wang, “Lion: Adversarial distillation of proprietary large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 3134–3154.
  131. Y. Gu, L. Dong, F. Wei, and M. Huang, “Knowledge distillation of large language models,” arXiv preprint arXiv:2306.08543, 2023.
  132. R. Agarwal, N. Vieillard, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem, “Gkd: Generalized knowledge distillation for autoregressive sequence models,” arXiv preprint arXiv:2306.13649, 2023.
  133. C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao, “Less is more: Task-aware layer-wise distillation for language model compression,” in International Conference on Machine Learning. PMLR, 2023, pp. 20 852–20 867. find
  134. I. Timiryasov and J.-L. Tastet, “Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty,” arXiv preprint arXiv:2308.02019, 2023. find
  135. C. Zhang, Y. Yang, J. Liu, J. Wang, Y. Xian, B. Wang, and D. Song, “Lifting the curse of capacity gap in distilling language models,” arXiv preprint arXiv:2305.12129, 2023. find
  136. L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu, “Dynabert: Dynamic bert with adaptive width and depth,” Advances in Neural Information Processing Systems, vol. 33, pp. 9782–9793, 2020. find
  137. S. Padmanabhan, Y. Onoe, M. J. Zhang, G. Durrett, and E. Choi, “Propagating knowledge updates to lms through distillation,” arXiv preprint arXiv:2306.09306, 2023. find
  138. Y. Yin, C. Chen, L. Shang, X. Jiang, X. Chen, and Q. Liu, “Autotinybert: Automatic hyper-parameter optimization for efficient pre-trained language models,” arXiv preprint arXiv:2107.13686, 2021. find
  139. J. Xu, X. Tan, R. Luo, K. Song, J. Li, T. Qin, and T.-Y. Liu, “Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 1933– 1943. find
  140. A. Klein, J. Golebiowski, X. Ma, V. Perrone, and C. Archambeau, “Structural pruning of large language models via neural architecture search,” 2023. find
  141. M. Javaheripi, G. de Rosa, S. Mukherjee, S. Shah, T. Religa, C. C. Teodoro Mendes, S. Bubeck, F. Koushanfar, and D. Dey, “Litetransformersearch: Training-free neural architecture search for efficient language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 254–24 267, 202… find
  142. D. D. Xu, S. Mukherjee, X. Liu, D. Dey, W. Wang, X. Zhang, A. Awadallah, and J. Gao, “Few-shot task-agnostic neural architecture search for distilling large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 28 644–28 656, 2022. find
  143. A. Kaushal, T. Vaidhya, and I. Rish, “Lord: Low rank decomposition of monolingual code llms for one-shot compression,” arXiv preprint arXiv:2309.14021, 2023. find
  144. M. Xu, Y. L. Xu, and D. P. Mandic, “Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition,” arXiv preprint arXiv:2307.00526, 2023. — [arXiv:2307.00526](https://arxiv.org/abs/2307.00526) find
  145. Y. Li, Y. Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao, “Losparse: Structured compression of large language models based on low-rank and sparse approximation,” arXiv preprint arXiv:2306.11222, 2023. — [arXiv:2306.11222](https://arxiv.org/abs/2306.11222) find
  146. R. Saha, V. Srivastava, and M. Pilanci, “Matrix compression via randomized low rank and low precision factorization,” arXiv preprint arXiv:2310.11028, 2023. — [arXiv:2310.11028](https://arxiv.org/abs/2310.11028) find
  147. Z. Yao, X. Wu, C. Li, S. Youn, and Y. He, “Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,” arXiv preprint arXiv:2303.08302, 2023. — [arXiv:2303.08302](https://arxiv.org/abs/2303.08302) find
  148. R. Chand, Y. Prabhu, and P. Kumar, “Dsformer: Effective compression of text-transformers by dense-sparse weight factorization,” arXiv preprint arXiv:2312.13211, 2023. — [arXiv:2312.13211](https://arxiv.org/abs/2312.13211) find
  149. Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun, “Asvd: Activation-aware singular value decomposition for compressing large language models,” arXiv preprint arXiv:2312.05821, 2023. — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) find
  150. R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019.
  151. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” arXiv preprint arXiv:2309.17453, 2023. find
  152. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020.
  153. M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang et al., “Big bird: Transformers for longer sequences,” Advances in neural information processing systems, vol. 33, pp. 17 283–17 297, 2020.
  154. S. Dai, H. Genc, R. Venkatesan, and B. Khailany, “Efficient transformer inference with statically structured sparse attention,” in 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, 2023, pp. 1–6. find
  155. Anonymous, “SemSA: Semantic sparse attention is hidden in large language models.” 2023. [Online]. Available: https://openreview.net/forum?id=eG9AkHtYYH ↗
  156. H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, pp. 97–110.
  157. L. Ren, Y. Liu, S. Wang, Y. Xu, C. Zhu, and C. Zhai, “Sparse modular activation for efficient sequence modeling,” arXiv preprint arXiv:2306.11197, 2023. — [arXiv:2306.11197](https://arxiv.org/abs/2306.11197) find
  158. S. Anagnostidis, D. Pavllo, L. Biggio, L. Noci, A. Lucchi, and T. Hoffmann, “Dynamic context pruning for efficient and interpretable autoregressive transformers,” arXiv preprint arXiv:2305.15805, 2023. — [arXiv:2305.15805](https://arxiv.org/abs/2305.15805) find
  159. N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” arXiv preprint arXiv:2001.04451, 2020.
  160. M. Pagliardini, D. Paliotta, M. Jaggi, and F. Fleuret, “Faster causal attention over large sequences through sparse flash attention,” arXiv preprint arXiv:2306.01160, 2023. — [arXiv:2306.01160](https://arxiv.org/abs/2306.01160) find
  161. A. Roy, M. Saffar, A. Vaswani, and D. Grangier, “Efficient content-based sparse attention with routing transformers,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 53–68, 2021.
  162. Y. Tay, D. Bahri, L. Yang, D. Metzler, and D.-C. Juan, “Sparse sinkhorn attention,” in International Conference on Machine Learning. PMLR, 2020, pp. 9438–9447. find
  163. Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
  164. A. Feng, I. Li, Y. Jiang, and R. Ying, “Diffuser: efficient transformers with multi-hop attention diffusion for long sequences,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, pp. 12 772–12 780. find
  165. E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” 2023. find
  166. M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” arXiv preprint arXiv:2306.11695, 2023. find
  167. H. Shao, B. Liu, and Y. Qian, “One-shot sensitivity-aware mixed sparsity pruning for large language models,” arXiv preprint arXiv:2310.09499, 2023. — [arXiv:2310.09499](https://arxiv.org/abs/2310.09499) find
  168. A. Syed, P. H. Guo, and V. Sundarapandiyan, “Prune and tune: Improving efficient pruning techniques for massive language models,” 2023. find
  169. X. Wei, Y. Zhang, Y. Li, X. Zhang, R. Gong, J. Guo, and X. Liu, “Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling,” arXiv preprint arXiv:2304.09145, 2023. — [arXiv:2304.09145](https://arxiv.org/abs/2304.09145) find
  170. P. Xu, W. Shao, M. Chen, S. Tang, K. Zhang, P. Gao, F. An, Y. Qiao, and P. Luo, “Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation,” in The Twelfth International Conference on Learning Representations, 2023. find
  171. E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh, “The optimal bert surgeon: Scalable and accurate second-order pruning for large language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 4163–418… find
  172. W. Kwon, S. Kim, M. W. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami, “A fast post-training pruning framework for transformers,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 101–24 116, 2022. find
  173. Y. Zhang, H. Bai, H. Lin, J. Zhao, L. Hou, and C. V. Cannistraci, “An efficient plug-and-play post-training pruning strategy in large language models,” 2023. find
  174. X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” Advances in neural information processing systems, vol. 36, 2024.
  175. M. Xia, T. Gao, Z. Zeng, and D. Chen, “Sheared llama: Accelerating language model pre-training via structured pruning,” arXiv preprint arXiv:2310.06694, 2023. find
  176. E. Kurtić, E. Frantar, and D. Alistarh, “Ziplm: Inference-aware structured pruning of language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
  177. M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “Loraprune: Pruning meets low-rank parameter-efficient fine-tuning,” 2023. find
  178. T. Chen, T. Ding, B. Yadav, I. Zharkov, and L. Liang, “Lorashear: Efficient large language model structured pruning and knowledge recovery,” arXiv preprint arXiv:2310.18356, 2023. find
  179. S. Ashkboos, M. L. Croci, M. G. d. Nascimento, T. Hoefler, and J. Hensman, “Slicegpt: Compress large language models by deleting rows and columns,” arXiv preprint arXiv:2401.15024, 2024. find
  180. Q. Zhang, S. Zuo, C. Liang, A. Bukharin, P. He, W. Chen, and T. Zhao, “Platon: Pruning large transformer models with upper confidence bound of weight importance,” in International Conference on Machine Learning. PMLR, 2022, pp. 26 809–26 823. find
  181. M. Xia, Z. Zhong, and D. Chen, “Structured pruning learns compact and accurate models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 1513–1528.
  182. C. Tao, L. Hou, H. Bai, J. Wei, X. Jiang, Q. Liu, P. Luo, and N. Wong, “Structured pruning for efficient generative pre-trained language models,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 10 880–10 895. find
  183. X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li, “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” arXiv preprint arXiv:2402.14800, 2024.
  184. A. Muzio, A. Sun, and C. He, “Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts,” arXiv preprint arXiv:2404.05089, 2024. — [arXiv:2404.05089](https://arxiv.org/abs/2404.05089) find
  185. P. Dong, L. Li, Z. Tang, X. Liu, X. Pan, Q. Wang, and X. Chu, “Pruner-zero: Evolving symbolic pruning metric from scratch for large language models,” in International Conference on Machine Learning (ICML), 2024. find
  186. Y. Zhang, L. Zhao, M. Lin, S. Yunyun, Y. Yao, X. Han, J. Tanner, S. Liu, and R. Ji, “Dynamic sparse no training: Training-free fine-tuning for sparse llms,” in International Conference on Learning Representations (ICLR), 2024. find
  187. S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,” in The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. find
  188. L. Li, Q. Li, B. Zhang, and X. Chu, “Norm tweaking: High-performance low-bit quantization of large language models,” arXiv preprint arXiv:2309.02784, 2023. — [arXiv:2309.02784](https://arxiv.org/abs/2309.02784) find
  189. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  190. Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “Qa-lora: Quantization-aware low-rank adaptation of large language models,” arXiv preprint arXiv:2309.14717, 2023. — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) find
  191. Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,” arXiv preprint arXiv:2310.08659, 2023. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) find
  192. E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022. find
  193. G. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y. Lee, D. Lee et al., “Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models,” in The Twelfth International Conference on Learning Representations, 2023.
  194. J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” arXiv preprint arXiv:2306.00978, 2023. find
  195. C. Lee, J. Jin, T. Kim, H. Kim, and E. Park, “Owq: Lessons learned from activation outliers for weight quantization in large language models,” arXiv preprint arXiv:2306.02272, 2023.
  196. T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse-quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078, 2023. find
  197. S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “Squeezellm: Dense-and-sparse quantization,” arXiv preprint arXiv:2306.07629, 2023.
  198. J. Chee, Y. Cai, V. Kuleshov, and C. De Sa, “Quip: 2-bit quantization of large language models with guarantees,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. find
  199. Y. J. Kim, R. Henry, R. Fahim, and H. H. Awadalla, “Finequant: Unlocking efficiency with fine-grained weight-only quantization for llms,” arXiv preprint arXiv:2308.09723, 2023. find
  200. K. Behdin, A. Acharya, A. Gupta, S. Keerthi, and R. Mazumder, “Quantease: Optimization-based quantization for language models–an efficient and intuitive algorithm,” arXiv preprint arXiv:2309.01885, 2023. find
  201. S. Li, X. Ning, K. Hong, T. Liu, L. Wang, X. Li, K. Zhong, G. Dai, H. Yang, and Y. Wang, “Llm-mq: Mixed-precision quantization for efficient llm deployment,” 2023. find
  202. Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,” in Advances in Neural Information Processing Systems, 2022.
  203. Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” 2023.
  204. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339, 2022. find
  205. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 38 087–38 099. find
  206. Z. Yao, X. Wu, C. Li, S. Youn, and Y. He, “Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,” arXiv preprint arXiv:2303.08302, 2023. — [arXiv:2303.08302](https://arxiv.org/abs/2303.08302) find
  207. Z. Yuan, L. Niu, J. Liu, W. Liu, X. Wang, Y. Shang, G. Sun, Q. Wu, J. Wu, and B. Wu, “Rptq: Reorder-based post-training quantization for large language models,” arXiv preprint arXiv:2304.01089, 2023. — [arXiv:2304.01089](https://arxiv.org/abs/2304.01089) find
  208. C. Guo, J. Tang, W. Hu, J. Leng, C. Zhang, F. Yang, Y. Liu, M. Guo, and Y. Zhu, “Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–15. find
  209. X. Wu, Z. Yao, and Y. He, “Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats,” arXiv preprint arXiv:2307.09782, 2023. find
  210. W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo, “Omniquant: Omnidirectionally calibrated quantization for large language models,” in The Twelfth International Conference on Learning Representations, 2023.
  211. J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “Qllm: Accurate and efficient low-bitwidth quantization for large language models,” in The Twelfth International Conference on Learning Representations, 2023. find
  212. Y. Zhao, C.-Y. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” arXiv preprint arXiv:2310.19102, 2023.
  213. W. Huang, Y. Liu, H. Qin, Y. Li, S. Zhang, X. Liu, M. Magno, and X. Qi, “Billm: Pushing the limit of post-training quantization for llms,” 2024. find
  214. S. Li, X. Ning, L. Wang, T. Liu, X. Shi, S. Yan, G. Dai, H. Yang, and Y. Wang, “Evaluating quantized large language models,” arXiv preprint arXiv:2402.18158, 2024. — [arXiv:2402.18158](https://arxiv.org/abs/2402.18158) find
  215. Y. Ma, H. Li, X. Zheng, F. Ling, X. Xiao, R. Wang, S. Wen, F. Chao, and R. Ji, “Affinequant: Affine transformation quantization for large language models,” in International Conference on Learning Representations (ICLR), 2024. find
  216. A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa, “Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks,” arXiv preprint arXiv:2402.04396, 2024.
  217. S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “Quarot: Outlier-free 4-bit inference in rotated llms,” arXiv preprint arXiv:2404.00456, 2024.
  218. Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort, “Spinquant–llm quantization with learned rotations,” arXiv preprint arXiv:2405.16406, 2024. — [arXiv:2405.16406](https://arxiv.org/abs/2405.16406) find
  219. C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,” arXiv preprint arXiv:2401.18079, 2024.
  220. Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” arXiv preprint arXiv:2402.02750, 2024.
  221. E. Frantar and D. Alistarh, “Optimal brain compression: A framework for accurate post-training quantization and pruning,” in Advances in Neural Information Processing Systems, 2022. find
  222. N. Vaidya, F. Oh, and N. Comly, “Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available,” [Online], 2023, https://github.com/NVIDIA/TensorRT-LLM. ↗
  223. InternLM, “Lmdeploy,” 2024. [Online]. Available: https://github.com/InternLM/lmdeploy ↗
  224. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
  225. B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal brain surgeon and general network pruning,” in IEEE international conference on neural networks. IEEE, 1993, pp. 293–299. find
  226. Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage,” Advances in neural information processing systems, vol. 2, 1989. find
  227. D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta, “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,” Nature communications, vol. 9, no. 1, p. 2383, 2018. find
  228. B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in International Conference on Learning Representations, 2016.
  229. X. Wang, Y. Zheng, Z. Wan, and M. Zhang, “Svd-llm: Truncation-aware singular value decomposition for large language model compression,” arXiv preprint arXiv:2403.07378, 2024. — [arXiv:2403.07378](https://arxiv.org/abs/2403.07378) find
  230. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback, 2022,” URL https://arxiv.org/abs/2203.02155, vol. 13, 2022. ↗
  231. Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang, “Dynamic neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7436–7456, 2021. find
  232. C. Xu and J. McAuley, “A survey on dynamic neural networks for natural language processing,” in Findings of the Association for Computational Linguistics: EACL 2023, 2023, pp. 2370–2381. find
  233. X. He, I. Keivanloo, Y. Xu, X. He, B. Zeng, S. Rajagopalan, and T. Chilimbi, “Magic pyramid: Accelerating inference with early exiting and token pruning,” Image, 2023. find
  234. TogetherAI, “Paving the way to efficient architectures: Stripedhyena-7b, open source models offering a glimpse into a world beyond transformers,” December 2023. [Online]. Available: https://www.together.ai/blog/stripedhyena-7b ↗
  235. A. Jaiswal, Z. Gan, X. Du, B. Zhang, Z. Wang, and Y. Yang, “Compressing llms: The truth is rarely pure and never simple,” arXiv preprint arXiv:2310.01382, 2023. — [arXiv:2310.01382](https://arxiv.org/abs/2310.01382) find
  236. Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286. find
  237. C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv preprint arXiv:2302.01318, 2023. find
  238. Y. Zhou, K. Lyu, A. S. Rawat, A. K. Menon, A. Rostamizadeh, S. Kumar, J.-F. Kagy, and R. Agarwal, “Distillspec: Improving speculative decoding via knowledge distillation,” arXiv preprint arXiv:2310.08461, 2023. — [arXiv:2310.08461](https://arxiv.org/abs/2310.08461) find
  239. J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra, “Draft & verify: Lossless large language model acceleration via self-speculative decoding,” arXiv preprint arXiv:2309.08168, 2023.
  240. X. Liu, L. Hu, P. Bailis, I. Stoica, Z. Deng, A. Cheung, and H. Zhang, “Online speculative decoding,” arXiv preprint arXiv:2310.07177, 2023.
  241. G. Monea, A. Joulin, and E. Grave, “Pass: Parallel speculative sampling,” arXiv preprint arXiv:2311.13581, 2023. — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) find
  242. Z. He, Z. Zhong, T. Cai, J. D. Lee, and D. He, “Rest: Retrieval-based speculative decoding,” arXiv preprint arXiv:2311.08252, 2023.
  243. X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, R. Y. Y. Wong, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia, “Specinfer: Accelerating generative llm serving with speculative inference and token tree verification,” arXiv preprint arXiv:2305.09781, 2023.
  244. B. Spector and C. Re, “Accelerating llm inference with staged speculative decoding,” arXiv preprint arXiv:2308.04623, 2023. — [arXiv:2308.04623](https://arxiv.org/abs/2308.04623) find
  245. Z. Chen, X. Yang, J. Lin, C. Sun, J. Huang, and K. C.-C. Chang, “Cascade speculative drafting for even faster llm inference,” arXiv preprint arXiv:2312.11462, 2023. — [arXiv:2312.11462](https://arxiv.org/abs/2312.11462) find
  246. Y. Fu, P. Bailis, I. Stoica, and H. Zhang, “Breaking the sequential dependency of llm inference using lookahead decoding,” November 2023. [Online]. Available: https://lmsys.org/blog/2023-11-21-lookahead-decoding/ ↗
  247. Y. Li, C. Zhang, and H. Zhang, “Eagle: Lossless acceleration of llm decoding by feature extrapolation,” December 2023. [Online]. Available: https://sites.google.com/view/eagle-llm ↗
  248. Z. Sun, A. T. Suresh, J. H. Ro, A. Beirami, H. Jain, and F. Yu, “Spectr: Fast speculative decoding via optimal transport,” arXiv preprint arXiv:2310.15141, 2023. — [arXiv:2310.15141](https://arxiv.org/abs/2310.15141) find
  249. F. Liu, Y. Tang, Z. Liu, Y. Ni, K. Han, and Y. Wang, “Kangaroo: Lossless self-speculative decoding via double early exiting,” arXiv preprint arXiv:2404.18911, 2024. — [arXiv:2404.18911](https://arxiv.org/abs/2404.18911) find
  250. ggerganov, “Inference of meta’s llama model (and others) in pure c/c++,” 2024. [Online]. Available: https://github.com/ggerganov/llama.cpp ↗
  251. Y. Song, Z. Mi, H. Xie, and H. Chen, “Powerinfer: Fast large language model serving with a consumer-grade gpu,” arXiv preprint arXiv:2312.12456, 2023.
  252. J. He and J. Zhai, “Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines,” arXiv preprint arXiv:2403.11421, 2024. find
  253. K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y. Dong, and Y. Wang, “Flashdecoding++: Faster large language model inference on gpus,” 2024. find
  254. T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” in Proceedings of Machine Learning and Systems (MLSys), 2023. find
  255. T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems, vol. 35, pp. 16 344–16 359, 2022. find
  256. T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691, 2023.
  257. Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “Bytetransformer: A high-performance transformer boosted for variable-length inputs,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2023, pp. 344–355. find
  258. R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley et al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Computing, Networking, Storag… find
  259. T. Dao, D. Haziza, F. Massa, and G. Sizov, “Flash-decoding for long-context inference,” [Online], 2023, https://crfm.stanford.edu/2023/10/12/flashdecoding.html. ↗
  260. HuggingFace, “Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax.” [Online], 2024, https://github.com/huggingface/transformers. ↗
  261. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
  262. Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “Glm: General language model pretraining with autoregressive blank infilling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 320–335.
  263. Sensetime, “Openppl: A high-performance deep learning inference platform,” [Online], 2023, https://openppl.ai/home. ↗
  264. NVIDIA, “cublas: Basic linear algebra on nvidia gpus,” [Online], 2017, https://developer.nvidia.com/cublas. ↗
  265. ——, “Cutlass: Cuda templates for linear algebra subroutines,” [Online], 2017, https://github.com/NVIDIA/cutlass. ↗
  266. S. Wang, “Fastgemv: High-speed gemv kernels,” [Online], 2023, https://github.com/wangsiping97/FastGEMV. ↗
  267. P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019, pp. 10–19.
  268. C. Zhang et al., “Beyond the speculative game: A survey of speculative execution in large language models,” arXiv preprint arXiv:2404.14897, 2024. — [arXiv:2404.14897](https://arxiv.org/abs/2404.14897) find
  269. H. Xia et al., “Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,” arXiv preprint arXiv:2401.07851, 2024.
  270. M. Stern, N. Shazeer, and J. Uszkoreit, “Blockwise parallel decoding for deep autoregressive models,” Advances in Neural Information Processing Systems, vol. 31, 2018. find
  271. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814. find
  272. P. Patel, E. Choukse, C. Zhang, Íñigo Goiri, A. Shah, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” arXiv preprint arXiv:2311.18677, 2023.
  273. C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y. Bao, N. Sun, and Y. Shan, “Inference without interference: Disaggregate llm inference for mixed downstream workloads,” arXiv preprint arXiv:2401.11181, 2024. — [arXiv:2401.11181](https://arxiv.org/abs/2401.11181) find
  274. Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” arXiv preprint arXiv:2401.09670, 2024.
  275. X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “Spotserve: Serving generative large language models on preemptible instances,” arXiv preprint arXiv:2311.15566, 2023.
  276. B. Lin, T. Peng, C. Zhang, M. Sun, L. Li, H. Zhao, W. Xiao, Q. Xu, X. Qiu, S. Li, Z. Ji, Y. Li, and W. Lin, “Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,” arXiv preprint arXiv:2401.02669, 2024. find
  277. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation, 2022, pp. 521–538.
  278. ModelTC, “Lightllm,” February 2024. [Online]. Available: https://github.com/ModelTC/lightllm/ ↗
  279. C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhandari, R. Y. Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko, and Y. He, “Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference,” arXiv preprint arXiv:2401.08671, 2024.
  280. B. Wu, Y. Zhong, Z. Zhang, G. Huang, X. Liu, and X. Jin, “Fast distributed inference serving for large language models,” arXiv preprint arXiv:2305.05920, 2023.
  281. Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, and D. Zhuo, “Fairness in serving large language models,” arXiv preprint arXiv:2401.00588, 2024. find
  282. A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, and R. Ramjee, “Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills,” arXiv preprint arXiv:2308.16369, 2023.
  283. A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, , and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” arXiv preprint arXiv:2403.02310, 2024.
  284. Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, “S 3 : Increasing gpu utilization during generative inference for higher throughput,” arXiv preprint arXiv:2306.06000, 2023. find
  285. Z. Ye, “flashinfer,” March 2024. [Online]. Available: https://github.com/flashinfer-ai/flashinfer ↗
  286. NVIDIA, “Fastertransformer: About transformer related optimization, including bert, gpt,” [Online], 2017, https://github.com/NVIDIA/FasterTransformer. ↗
  287. H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “Exegpt: Constraint-aware resource scheduling for llm inference,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2024, pp. 369–384. find
  288. B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” arXiv preprint arXiv:2406.03243, 2024.
  289. B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin, “Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism,” arXiv preprint arXiv:2404.09526, 2024.
  290. B. Li, S. Pandey, H. Fang, Y. Lyv, J. Li, J. Chen, M. Xie, L. Wan, H. Liu, and C. Ding, “Ftrans: Energy-efficient acceleration of transformers using fpga,” arXiv preprint arXiv:2007.08563, 2020. find
  291. T. J. Ham, Y. Lee, S. H. Seo, S. Kim, H. Choi, S. J. Jun, and J. W. Lee, “Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,” in ACM/IEEE 48th Annual International Symposium on Computer Architecture, 2021, pp. 692–705. find
  292. H. Fan, T. Chau, S. I. Venieris, R. Lee, A. Kouris, W. Luk, N. D. Lane, and M. S. Abdelfattah, “Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design,” in IEEE/ACM International Symposium on Microarchitecture, 2022, pp. 599–615. find
  293. Y. Qin, Y. Wang, D. Deng, Z. Zhao, X. Yang, L. Liu, S. Wei, Y. Hu, and S. Yin, “Fact: Ffn-attention co-optimized transformer architecture with eager correlation prediction,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–14. find
  294. H. Chen, J. Zhang, Y. Du, S. Xiang, Z. Yue, N. Zhang, Y. Cai, and Z. Zhang, “Understanding the potential of fpga-based spatial acceleration for large language model inference,” arXiv preprint arXiv:2312.15159, 2023. find
  295. S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, and J.-Y. Kim, “Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation,” in IEEE Hot Chips 34 Symposium, 2022. find
  296. S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang et al., “Flightllm: Efficient large language model inference with a complete mapping flow on fpga,” arXiv preprint arXiv:2401.03868, 2024. find
  297. S. teams, “Sharegpt,” 2023. [Online]. Available: https://sharegpt.com/ ↗
  298. J. Xie, Z. Chen, R. Zhang, X. Wan, and G. Li, “Large multimodal agents: A survey,” arXiv preprint arXiv:2402.15116, 2024. — [arXiv:2402.15116](https://arxiv.org/abs/2402.15116) find
  299. I. Lee, N. Jiang, and T. Berg-Kirkpatrick, “Exploring the relationship between model architecture and in-context learning ability,” arXiv preprint arXiv:2310.08049, 2023. — [arXiv:2310.08049](https://arxiv.org/abs/2310.08049) find
  300. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff et al., “Pythia: A suite for analyzing large language models across training and scaling,” in International Conference on Machine Learning. PMLR, 2023, pp. 2397–2430. find
  301. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023.
  302. Y. Tang, F. Liu, Y. Ni, Y. Tian, Z. Bai, Y.-Q. Hu, S. Liu, S. Jui, K. Han, and Y. Wang, “Rethinking optimization and architecture for tiny language models,” arXiv preprint arXiv:2402.02791, 2024.
  303. Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, “Textbooks are all you need ii: phi-1.5 technical report,” arXiv preprint arXiv:2309.05463, 2023.
  304. S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al., “Textbooks are all you need,” arXiv preprint arXiv:2306.11644, 2023. find
  305. P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” arXiv preprint arXiv:2401.02385, 2024.
  306. C. Zhang, D. Song, Z. Ye, and Y. Gao, “Towards the law of capacity gap in distilling language models,” arXiv preprint arXiv:2311.07052, 2023.
  307. X. Geng and H. Liu, “Openllama: An open reproduction of llama,” May 2023. [Online]. Available: https://github.com/openlm-research/open llama ↗
  308. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta et al., “Stable lm 2 1.6 b technical report,” arXiv preprint arXiv:2402.17834, 2024.
  309. “Minicpm: Unveiling the potential of end-side large language models,” 2024. find
  310. Z. Liu, C. Zhao, F. Iandola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi et al., “Mobilellm: Optimizing sub-billion parameter language models for on-device use cases,” arXiv preprint arXiv:2402.14905, 2024. find
  311. M. team, “MLC-LLM,” 2023. [Online]. Available: https://github.com/mlc-ai/mlc-llm ↗
  312. Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, p. 100211, 2024.
  313. Y. Li, H. Wen, W. Wang, X. Li, Y. Yuan, G. Liu, J. Liu, W. Xu, X. Wang, Y. Sun et al., “Personal llm agents: Insights and survey about the capability, efficiency and security,” arXiv preprint arXiv:2401.05459, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/