Elamur Research⌕
☰
home / papers / 2024 / LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Paper · 2024 · Hongye Jin, Xiaotian Han, Jingfeng Yang et al.

Summary

Large language models that use RoPE usually fail on inputs longer than their pretraining sequence length. Perplexity spikes and outputs become erratic. In this ICML 2024 paper, the authors argue that the problem is not a lack of long-text understanding. Instead, they blame a "positional O.O.D." effect: at inference, the model meets relative distances it never saw during training.

Their fix, SelfExtend, needs no fine-tuning and only a small code change. It mixes two kinds of attention, both computed with the model's own weights:

  • Neighbor attention handles tokens inside a fixed local window, as usual.
  • Grouped attention handles distant tokens. Their positions are compressed by floor division by a group size, so every relative distance stays within the pretrained range.

The authors test the method on Llama-2, Mistral, SOLAR and Phi-2. Tasks cover language modeling on PG-19, synthetic passkey retrieval and real-world LongBench tasks. SelfExtend keeps perplexity stable on longer inputs and often matches or beats long-context models built by fine-tuning. This matters because it is a cheap way to widen context windows without scarce long-text data or costly retraining.

Key points

  • Core mechanism: a FLOOR (//) operation with group size G_s maps large, unseen relative positions to ones seen in pretraining. Exact nearby positions are kept inside a neighbor window, for example 1024 tokens.
  • PG-19 perplexity: Llama-2-7b-chat is stable at 4k (4096) tokens but explodes at 6k (6144). With grouped attention, it stays low and stable on longer sequences.
  • Passkey retrieval: SelfExtend on Llama-2-chat-7b was compared with fine-tuned models: vicuna-1.5-7b-16k, llama-2-7b-32k, Yarn-llama-2-7b-64k and Llama-2-7b-Longlora-16k. Group sizes tested were 8, 12, 16 and 100,000. A harder variant used passkeys longer than the usual 5 digits, and SelfExtend kept its advantage without tuning.
  • LongBench: longer contexts generally helped, with steady gains on NarrativeQA and Qmsum. Some datasets peaked and then declined, because large group sizes blur positional detail. Lcc stayed flat, likely because its code tasks are local and its inputs short.
  • Limitations: naive implementations add extra attention cost, which blocked kernels such as Flash Attention largely offset. Quality drops with very large group sizes, so context cannot grow indefinitely. Code is available at github.com/datamllab/LongLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen and Xia Hu. 2024. LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning. arXiv:2401.01325.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.01325 · arXiv 2401.01325

@misc{jin2024llm,
  title = {{LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning}},
  author = {Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen and Xia Hu},
  year = {2024},
  doi = {10.48550/arxiv.2401.01325},
  eprint = {2401.01325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.01325}
}

References

43 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. amazon. Mistrallite model. https://huggingface.co/ amazon/MistralLite, 2023. [Online; accessed 29-December-2023… ↗
  2. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. find
  3. An, C., Gong, S., Zhong, M., Li, M., Zhang, J., Kong, L., and Qiu, X. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023. our card
  4. Bai, T., Luo, J., Zhao, J., Wen, B., and Wang, Q. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01356, 2021. — [arXiv:2102.01356](https://arxiv.org/abs/2102.01356) find
  5. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… find
  6. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. gkamradt. Llmtest needleinahaystack: Doing simple retrieval from llm models. https:/… our card
  7. Anothropic. Long context prompting for claude 2.1. https://www.anthropic.com/news/ claude-2-1-prompting, 2023. ↗
  8. Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. context lengths in llms. arXiv preprint arXiv:2308.10882, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) find
  9. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. find
  10. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. our card
  11. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. find
  12. Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Giorno, A. D., Eldan, R., Gopi, S., Gunasekar, S., Javaheripi, M., Kauffmann, P., Lee, Y. T., Li, Y., Nguyen, A., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Santacroce, M., Behl, H. S., Kalai, A. T., Wang… find
  13. Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019. our card
  14. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. our card
  15. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. find
  16. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. our card
  17. Jiang, H., Wu, Q., , Luo, X., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. LongLLMLingua: Accelerating and enhancing llms in long context scenarios via prompt compression. ArXiv preprint, abs/2310.06839, 2023b. URL https://arxiv.org/abs/2310.06839. ↗ our card
  18. Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021. — [arXiv:2108.13624](https://arxiv.org/abs/2108.13624) find
  19. Shi, H., Gao, J., Ren, X., Xu, H., Liang, X., Li, Z., and Kwok, J. T.-Y. Sparsebert: Rethinking the importance analysis in self-attention. In International Conference on Machine Learning, pp. 9547–9557. PMLR, 2021. find
  20. Ke, G., He, D., and Liu, T.-Y. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020. find
  21. Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166, 2023. our card
  22. Su, J. Rectified rotary position embeddings. https://github.com/bojone/rerope, 2023. ↗
  23. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. RoFormer: Enhanced transformer with rotary position embedding, 2022. arXiv: 2104.09864. our card
  24. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. our card
  25. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024. our card
  26. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. A length-extrapolatable transformer. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… our card
  27. Mohtashami, A. and Jaggi, M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. find
  28. Team, M. N. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www. mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. find
  29. Pal, A., Karkhanis, D., Roberts, M., Dooley, S., Sundararajan, A., and Naidu, S. Giraffe: Adventures in expanding Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towar… — [arXiv:2108.13624](https://arxiv.org/abs/2108.13624) find
  30. Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  31. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023. our card
  32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017. our card
  33. Zhu, D., Yang, N., Wang, L., Song, Y., Wu, W., Wei, F., and Li, S. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400, 2023. find
  34. Wu, K., Peng, H., Chen, M., Fu, J., and Chao, H. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10033–10041, 2021. find
  35. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. find
  36. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. our card
  37. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020. our card
  38. Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., and Hu, X. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712, 2023. our card
  39. Yin Song and Chen Wu and Eden Duthie. amazon/Mistral-Lite, 2023. URL https://huggingface.co/amazon/ MistralLite. ↗
  40. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020. our card
  41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  42. Zhang, J., Chao, H., Dhurandhar, A., Chen, P.-Y., Tajer, A., Xu, Y., and Yan, P. When neural networks fail to generalize? a model sensitivity perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11219–11227, 2023. find
  43. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/