Elamur Research
☰
home / papers / 2023 / LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.emnlp-main.825.pdf

Summary

Prompts for large language models keep getting longer because of chain-of-thought reasoning, in-context examples, and retrieved documents. This raises cost and latency, and models reachable only through an API cannot be quantized or pruned by the user. LLMLingua addresses this by shortening the prompt itself before it reaches a black-box LLM. A small language model scores the prompt's content by perplexity and the method works in three stages. First, a budget controller assigns different compression levels to the instruction, the demonstrations, and the question, and drops entire low-value demonstrations. Second, an iterative token-level procedure removes less informative tokens while accounting for how the remaining tokens depend on each other. Third, instruction tuning aligns the small model's distribution with that of the target LLM. The authors tested the method on GSM8K, BBH, ShareGPT, and Arxiv-March23, covering reasoning, in-context learning, conversation, and summarization. It outperformed the prior Selective-Context approach and allowed very high compression with minimal quality loss. This matters for cheaper, faster inference and for fitting longer contexts.

Key points

  • The method reaches up to 20x compression with only about a 1.5-point performance drop. Results are state-of-the-art on all four datasets: GSM8K, BBH, ShareGPT, and Arxiv-March23.
  • It has three modules: the Budget Controller (perplexity-based, demonstration-level), Iterative Token-level Compression, and instruction-tuning-based distribution Alignment. Small compressors include Alpaca-7B and GPT2-Alpaca.
  • On a V100-32G GPU, end-to-end latency on GSM8K improves by 1.7x–5.7x at compression ratios between 2x and 10x. The authors estimate roughly 4x compute savings at 5x compression when using a 7B compressor with a 175B-scale target model.
  • GPT-4 can largely reconstruct compressed prompts, including full reasoning chains, whereas GPT-3.5-Turbo struggles to do so. Compression also tends to shorten the generated output.
  • Performance drops sharply at extreme ratios (about 25x–30x on GSM8K). Tokenizer mismatches between the small model and the target LLM can also lead to underestimating prompt length. Code is available at aka.ms/LLMLingua.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

181 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.825

@inproceedings{jiang2023llmlingua,
  title = {{LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models}},
  author = {Jiang, Huiqiang and Wu, Qianhui and Lin, Chin-Yew and Yang, Yuqing and Qiu, Lili},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.825},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.825}
}

References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Sharegpt. https://sharegpt.com/. ↗
  2. Udit Arora, William Huang, and He He. 2021. Types of out-of-distribution texts and how to detect them. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10687–10701, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  3. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations. find
  4. Harrison Chase. 2022. LangChain. find
  5. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. ArXiv preprint, abs/2305.14788. find our card
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168. find our card
  8. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. 2023. Language modeling is compression. ArXiv preprint, abs/2309.10668. find
  9. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems. find
  10. Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning. find
  11. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations. find
  12. Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023a. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. ArXiv preprint, abs/2305.17306. find
  13. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023b. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations. find our card
  14. Tao Ge, Jing Hu, Li Dong, Shaoguang Mao, Yan Xia, Xun Wang, Si-Qing Chen, and Furu Wei. 2022. Extensible prompts for language models. ArXiv preprint, abs/2212.00616. — [arXiv:2212.00616](https://arxiv.org/abs/2212.00616) find
  15. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context compression in a large language model. ArXiv preprint, abs/2307.06945. find
  16. Henry Gilbert, Michael Sandborn, Douglas C Schmidt, Jesse Spencer-Smith, and Jules White. 2023. Semantic compression with large language models. ArXiv preprint, abs/2304.12512. — [arXiv:2304.12512](https://arxiv.org/abs/2304.12512) find
  17. Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating BERT inference via progressive word-vector elimination. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July… find
  18. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  19. Gyuwan Kim and Kyunghyun Cho. 2021. Length-adaptive transformer: Train once with length drop, use anytime with search. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… find
  20. Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794. find
  21. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261. find our card
  22. Yucheng Li. 2023. Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering. ArXiv preprint, abs/2304.12102. find
  23. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford\_alpaca. find
  24. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  25. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  26. Kimberly T Mai, Toby Davies, and Lewis D Griffin. 2022. Self-supervised losses for one-class textual anomaly detection. ArXiv preprint, abs/2204.05695. — [arXiv:2204.05695](https://arxiv.org/abs/2204.05695) find
  27. Ali Modarressi, Hosein Mohebbi, and Mohammad Taher Pilehvar. 2022. AdapLeR: Speeding up inference by adaptive length reduction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1–15, Dublin, Ireland. Association for Computation… find
  28. Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. ArXiv preprint, abs/2304.08467. find our card
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  30. Richard Clark Pasco. 1976. Source coding algorithms for fast data compression. Ph.D. thesis, Citeseer. find
  31. Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems. find
  32. Jorma J Rissanen. 1976. Generalized kraft inequality and arithmetic coding. IBM Journal of research and development, 20(3):198–203. find
  33. Claude E Shannon. 1951. Prediction and entropy of printed english. Bell system technical journal, 30(1):50–64. find
  34. Ilya Sutskever. 2023. A theory of unsupervised learning. https://simons.berkeley.edu/talks/ ilya-sutskever-openai-2023-08-14. ↗
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find our card
  36. David Wingate, Mohammad Shoeybi, and Taylor Sorensen. 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621–5634, Abu Dhabi, United Arab Emirates. Assoc… find
  37. Qianhui Wu, Huqiang Jiang, Haonan Yin, Börje F. Karlsson, and Chin-Yew Lin. 2023. Multi-level knowledge distillation for out-of-distribution detection in text. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (Long Papers). find
  38. Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. find
  39. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304.12244. find our card
  40. Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Inference with reference: Lossless acceleration of large language models. ArXiv preprint, abs/2304.04487. — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) find
  41. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Ne… find our card
  42. Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. 2023. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. ArXiv preprint, abs/2304.14979. find
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  44. A.2 Other Implementation Details Wangchunshu Zhou, Yuchen Eleanor Jiang, Ryan Cotterell, and Mrinmaya Sachan. 2023. Efficient prompting via dynamic in-context learning. ArXiv preprint, abs/2305.11170. — [arXiv:2305.11170](https://arxiv.org/abs/2305.11170) find