Elamur Research
☰
home / papers / 2023 / Learning to Compress Prompts with Gist Tokens

Learning to Compress Prompts with Gist Tokens

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2304.08467

Summary

Long prompts, such as system instructions attached to every request, take up space in a model's context window. They also have to be re-encoded or cached again and again, which costs compute and storage. Task-specific finetuning or distillation removes the prompt, but it requires a separately trained model for each task. The authors propose gisting: a language model learns to condense any instruction into the activations of a few special "gist" tokens, which can then be stored and reused. Training is ordinary instruction tuning with one change. The attention mask is altered so that tokens after the gist tokens cannot see the original prompt, which forces its content through the gist bottleneck. Because the model itself produces the compressed prefix, it generalizes zero-shot to unseen instructions. Experiments on LLaMA-7B and FLAN-T5-XXL, trained on the Alpaca+ dataset, show prompts shrunk by as much as 26x while output quality remains close to uncompressed baselines. This yields notable compute and storage savings, which matters for anyone serving many cached prompts at scale.

Key points

  • Models and efficiency gains. Gisting was tested on a decoder-only model (LLaMA-7B) and an encoder-decoder model (FLAN-T5-XXL).
    • Prompt compression reached up to 26x.
    • FLOPs dropped by up to 40%.
    • Wall-clock time improved by up to 4.2%.
  • Alpaca+ dataset. It merges Self-Instruct and Stanford Alpaca data, which were generated by text-davinci-001 and text-davinci-003.
    • It contains 130,321 examples, with 104,664 unique tasks and 48,530 unique inputs.
    • About 59% of tasks have no input.
  • Evaluation splits. Three held-out sets were used:
    • 1000 Seen prompts;
    • 1000 Unseen prompts;
    • 252 hand-written Human prompts from Self-Instruct, used as an out-of-distribution test.
  • Implementation. The method needs only a single new vocabulary token plus a modified attention mask, roughly 10 lines of PyTorch. It adds no training cost beyond standard instruction finetuning.
  • Storage. In LLaMA-7B, each cached token takes about 1.05 MB of KV storage, so gisting lets about 26x more prompts be cached in the same space. The authors caution that compression can drop nuances of the original instruction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 130 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.08467 · arXiv 2304.08467

@misc{mu2023learning,
  title = {{Learning to Compress Prompts with Gist Tokens}},
  author = {Mu, Jesse and Li, Xiang Lisa and Goodman, Noah},
  year = {2023},
  doi = {10.48550/arxiv.2304.08467},
  eprint = {2304.08467},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08467}
}

References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find our card
  2. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020. find
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. find
  4. C. Chen. Transformer inference arithmetic. transformer-inference-arithmetic/, 2022. find
  5. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality. https://vicuna.lmsys.org/, 2023. ↗
  6. R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. find
  7. E. Choi, Y. Jo, J. Jang, and M. Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint arXiv:2206.11349, 2022. — [arXiv:2206.11349](https://arxiv.org/abs/2206.11349) find
  8. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  9. Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019. find
  10. W. J. Dixon and F. J. Massey Jr. Introduction to statistical analysis. McGraw-Hill, 1951. find
  11. X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley. edu/blog/2023/04/03/koala/. ↗
  12. F. Gilardi, M. Alizadeh, and M. Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023. find our card
  13. D. Ha, A. Dai, and Q. V. Le. HyperNetworks. In International Conference on Learning Representations, 2017. find
  14. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. find
  15. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019. https://kipp.ly/blog/ ↗
  16. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. find
  17. F. Huang, H. Kwak, and J. An. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. arXiv preprint arXiv:2302.07736, 2023. find
  18. B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021. find
  19. X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021. find
  20. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. find
  21. P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating Wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018. find our card
  22. OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt, 2022. ↗
  23. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730–27744, 2022. find
  24. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019. find our card
  25. J. Phang, Y. Mao, P. He, and W. Chen. HyperTuning: Toward adapting large language models without back-propagation. arXiv preprint arXiv:2211.12485, 2022. — [arXiv:2211.12485](https://arxiv.org/abs/2211.12485) find
  26. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. find
  27. J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. find our card
  28. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified Text-to-Text Transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. find
  29. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020. find
  30. C. Snell, D. Klein, and R. Zhong. arXiv:2209.15189, 2022. find
  31. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford\_alpaca, 2023. ↗
  32. Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022. Learning by distilling context. arXiv preprint find our card
  33. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. find
  34. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. find
  35. J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou. Is ChatGPT a good NLG evaluator? A preliminary study. arXiv preprint arXiv:2303.04048, 2023. find our card
  36. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
  37. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Languag… find
  38. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. find
  39. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. find our card
  40. D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621–5634, Abu Dhabi, United Arab Emirates, Dec. 2022. Association f… find
  41. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In… find our card
  42. Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. find
  43. H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv, N. Duan, and W. Chen. Poolingformer: Long document modeling with pooling attention. In International Conference on Machine Learning, pages 12437–12446. PMLR, 2021. find