Elamur Research⌕
☰
home / papers / 2025 / LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

Paper · 2025 · Zhiyuan Hu, Yuliang Liu, Jinman Zhao et al.

Summary

Large language models handle inputs poorly once these exceed the context length seen during pretraining. Continuing training on full-length sequences fixes this but is expensive, since attention cost grows quadratically with length. LongRecipe is a training framework that extends the context window without training on genuinely long inputs.

First, it compares the logits of a base model with those of a long-context-tuned model. This reveals which token types change most, and numerals dominate. Sentences containing such tokens are selected and upsampled into shorter training samples. Position Index Transformation then gives these samples skipped but locally continuous position indices, so they imitate long-range token distances. Earlier methods fall short here: randomized encodings break local structure, and PoSE uses only two chunks. Pretraining data replay and model merging help retain general skills.

On Llama3-8B, Mistral-7B and Qwen2-7B, the method reaches effective windows of 80k–128k. It uses roughly 30% of the target length and over 85% less compute than full-length training, with little loss on general tasks. This makes long-context adaptation practical on a single GPU.

Key points

  • Evaluation covers Multi-Needle In A Haystack, RULER and LongBench for long context, plus MMLU, GSM8K and HumanEval for general ability. The average gain is about 5.5% across four metrics at 80k and 128k windows.
  • An open-source model was extended from 8k to 128k with one day of training on a single 80G H100. Its long-context performance is reported as near GPT-4-Turbo.
  • In the token analysis, NUM tokens make up about 0.4 of the top-20% logit-shift tokens. That is roughly 2.67× the share of PRON/AUX and about 4× that of CCONJ/ADP.
  • The constructed data keeps an average continuous segment of 88 tokens, versus about 0 for RPES. Its Long Dependency Score is 17.88, compared with 12.07 for PoSE/RPES data.
  • Code is released at github.com/zhiyuanhubj/LongRecipe. The authors plan long-context SFT and versions with 512k and 1M token windows.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang and Bryan Hooi. 2025. LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.581

@inproceedings{hu2025longrecipe,
  title = {{LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models}},
  author = {Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang and Bryan Hooi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.581},
  url = {https://doi.org/10.18653/v1/2025.acl-long.581}
}

References

51 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.
  2. Gradient AI. 2024. Gradient ai, llama3 series. find
  3. Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for Hig… find
  4. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) find
  5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  7. James P Callan. 1994. Passage-level evidence in document retrieval. In SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pages 302–310. Springer. find
  8. Cerebras. 2023. Slimpajama: A 627b token, cleaned and deduplicated version of redpajama. find
  9. Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. 2024a. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2405.17915. find
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find
  11. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024b. LongloRA: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations.
  12. Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399. find
  13. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  14. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  16. Hugging Face. Accelerate. https://huggingface.co/docs/accelerate/en/package_reference/ accelerator. ↗
  17. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171. gkamradt. 2023. Llmtest_needleinahaystack: Doing simple retrieval from llm models. https://github.com/gkamr… ↗
  18. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka… — ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
  19. Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. 2024. Contextual position encoding: Learning to count what’s important. arXiv preprint arXiv:2405.18719. — [arXiv:2405.18719](https://arxiv.org/abs/2405.18719) find
  20. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. Preprint, arXiv:2308.16137.
  21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  22. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.
  23. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509.
  24. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325.
  25. Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1–35. find
  26. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332.
  27. Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889.
  28. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
  29. Meta.AI. 2024. Llama 3.1 model card. find
  30. Mistral.AI. 2024. La plateforme. find
  31. OpenAI: Josh Achiam et al. 2023. GPT-4 technical report. arXiv:2303.08774.
  32. Bowen Peng and Jeffrey Quesnelle. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. https://www.reddit.com/r/LocalLLaMA/ comments/14lz7j5/ntkaware_scaled_rope_ allows_llama_models_to_have. ↗
  33. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. find
  34. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  35. Machel Reid et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530.
  36. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo… find
  37. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  38. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2023. A length-extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14590–14604, Toron…
  39. Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. 2024a. Resonance RoPE: Improving context length generalization of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 586– 598, Bangkok, Thailand and virtual meeting. Association… find
  40. Y. Wang, D. Ma, and D. Cai. 2024b. With greater text comes greater necessity: Inference-time training helps long text generation. Preprint, arXiv:2401.11504. — [arXiv:2401.11504](https://arxiv.org/abs/2401.11504) find
  41. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning. find
  42. Thomas Wolf et al. 2019. Huggingface’s Transformers: State-of-the-art natural language processing. arXiv:1910.03771.
  43. Tong Wu, Yanpeng Zhao, and Zilong Zheng. 2024. An efficient recipe for long context extension via middle-focused positional encoding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  44. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  45. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024b. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666. find
  46. Shuo Yang, Siwen Luo, and Soyeon Caren Han. 2025a. Multimodal commonsense knowledge distillation for visual question answering (student abstract). In Proceedings of the AAAI conference on artificial intelligence, volume 39, pages 29545–29547. find
  47. Shuo Yang, Siwen Luo, Soyeon Caren Han, and Eduard Hovy. 2025b. Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answering. arXiv preprint arXiv:2503.18491. find
  48. Alex Young et al. 2024. Yi: Open foundation models by 01.AI. arXiv:2403.04652.
  49. Zhipu. 2024. Longwriter. find
  50. Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2023. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400.
  51. Zilin Zhu. Zigzag-ring-attention. https://github.com/zhuzilin/ring-flash-attention. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2025-longrecipe-recipe-efficient-long-context/