Elamur Research⌕
☰
home / papers / 2024 / Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Paper · 2024 · Xindi Wang, Mahsa Salmani, Parsa Omidi et al.

Summary

Transformer-based LLMs such as GPT-3, PaLM and LLaMA struggle when inputs exceed the lengths seen during training. Self-attention cost grows steeply with sequence length, memory needs can outstrip high-end GPUs, and keeping long-range dependencies coherent is difficult. This survey, by Huawei Technologies Canada with academic co-authors, sorts the research on extending LLM context into five families: methods that extrapolate to longer lengths, approaches that approximate attention, transformer variants that drop attention entirely, techniques for compressing models, and designs that account for hardware. It also tags each technique by stage of use: training from scratch, fine-tuning an existing model, or inference alone with no parameter changes. The paper offers no new model or dataset. Its contribution is the structured overview, covering positional-encoding methods (ALiBi, xPOS, CLEX, YaRN, positional interpolation), input segmentation (PCW, StreamingLLM), prompt compression, and memory or IO techniques (Ring Attention, PagedAttention, Flash-Decoding). The authors list open problems, including compute cost, interpretability and use of external knowledge. They suggest curriculum learning on progressively longer inputs and call for dedicated long-context benchmarks. For practitioners, it helps choose a method based on deployment stage.

Key points

  • The taxonomy (Figure 1) has five branches. Hardware-aware methods include IO-awareness, multi-device distributed attention and memory management. Model compression covers quantization and pruning.
  • Prompt compression: LLMLingua uses small models (GPT-2 small, LLaMA-7B) to drop unneeded tokens, reaching up to 20× compression with little quality loss. LongLLMLingua reports up to 21.4% better retrieval-augmented generation while using only about 25% of the tokens.
  • StreamingLLM exploits the "attention sink" effect by keeping the initial token alongside a sliding window. This lets models trained on finite windows handle unbounded streams without fine-tuning.
  • Positional interpolation rescales position indices linearly to fit the original pre-training window. It proved more stable than direct extrapolation and needed fewer fine-tuning steps.
  • On the systems side, Flash-Decoding gives up to 8× faster generation on very long sequences. LLM-in-Flash stores weights in flash memory and can run models up to twice the size of available DRAM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Vector InstituteToronto · Canada33 works · 4,535 citations🎓Western UniversityLondon · Canada7 works · 278 citations🏢Huawei Technologies (Canada)Markham · Canada3 works · 210 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh and Armaghan Eshaghi. 2024. Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models. arXiv:2402.02244.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.02244 · arXiv 2402.02244

@misc{wang2024beyond,
  title = {{Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models}},
  author = {Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh and Armaghan Eshaghi},
  year = {2024},
  doi = {10.48550/arxiv.2402.02244},
  eprint = {2402.02244},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.02244}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Si…
  2. Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference…
  3. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, E… — PaLM: Scaling Language Modeling with Pathways
  4. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022. find
  5. Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-Decoding for long-context inference, 2023. find
  6. Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, and William Cohen. FiDO: Fusion-in-decoder optimized for stronger performance and faster inference. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11534–11547, Toronto, Cana… find
  7. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, volume 35, pages 30318–30332, 2022. find
  8. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. LongNet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. find
  9. Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR, 23–29 Jul 2023. [Frantar et al., 2023] El… find
  10. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore, December 2023. Association for Computational Lingu…
  11. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Kri… find
  12. Vishwanath Saragadam, Randall Balestriero, Ashok Veeraraghavan, and Richard G Baraniuk. DeepTensor: Low-rank tensor decomposition with deep network priors. arXiv preprint arXiv:2204.03145, 2022. — [arXiv:2204.03145](https://arxiv.org/abs/2204.03145) find
  13. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orlea…
  14. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. find
  15. Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on ReLU and softmax in transformer. arXiv preprint arXiv:2302.06461, 2023. — [arXiv:2302.06461](https://arxiv.org/abs/2302.06461) find
  16. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  17. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. find
  18. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive Network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
  19. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14590–14604, Toronto, Ca…
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim-othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. ArXiv, abs/…
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017.
  22. Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 1, 2023. [Wang et al., 2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Lin…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-limits-survey-techniques-extend-context/