Elamur Research
☰
home / papers / 2024 / RULER: What's the Real Context Size of Your Long-Context Language Models?

RULER: What's the Real Context Size of Your Long-Context Language Models?

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2404.06654

Summary

Long-context language models are usually checked with the needle-in-a-haystack (NIAH) test. That test only shows whether a model can find one planted fact, so it says little about deeper use of long inputs. The NVIDIA authors built RULER, a fully synthetic benchmark whose input length and difficulty can be set freely. Because the inputs are generated, models cannot lean on memorized knowledge. RULER has 13 tasks in four groups:

  • extended NIAH variants with several keys, values or queries;
  • variable tracking, a multi-hop task that stands in for coreference chains;
  • common- and frequent-word extraction, which stand in for aggregation and summarization;
  • QA over SQuAD-style documents padded with distractors.

The authors tested 17 models, among them Gemini-1.5, GPT-4 and 15 open-source systems, at lengths from 4K to 128K tokens. Nearly all models score close to perfectly on plain NIAH, yet they decline sharply on harder tasks as inputs grow. Only about half stay above an acceptable threshold at 32K, even though every model advertises 32K or more. The results show that advertised context sizes overstate how much context a model can really use. RULER offers a cheap, controllable behavioral check, though the authors note it does not replace realistic evaluations.

Key points

  • Results: Gemini-1.5 and GPT-4 stayed on top under both length-weighted average scores. Most other models fell below the threshold before reaching their claimed context limit.
  • Yi-34B-200K case study: At long lengths the model gave incomplete answers and failed to locate the relevant content. On QA its accuracy sank toward its no-context baseline because of hallucination. Across several models, longer inputs also led to more copying from the context and more reliance on parametric knowledge.
  • Ablations:
    • Training on longer contexts usually helps, but not always: LWM-1M scored below the 512K version at 256K.
    • Larger models hold up better: Yi-34B outperformed Yi-6B.
  • Non-Transformer models: RWKV-v5 and Mamba-2.8B-slimpj degraded heavily by 8K and trailed Llama2-7B up to 4K.
  • Code and limitations: The code is public at github.com/hsiehjackson/RULER. The authors list several limitations:
    • no control over needle position (depth);
    • unverified correlation with realistic tasks;
    • no short-context evaluation;
    • limited testing of prompt robustness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

373 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 366 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.06654 · arXiv 2404.06654

@misc{hsieh2024ruler,
  title = {{RULER: What's the Real Context Size of Your Long-Context Language Models?}},
  author = {Hsieh, Cheng-Ping and Sun, Simeng and Kriman, Samuel and Acharya, Shantanu and Rekesh, Dima and Jia, Fei and Zhang, Yang and Ginsburg, Boris},
  year = {2024},
  doi = {10.48550/arxiv.2404.06654},
  eprint = {2404.06654},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.06654}
}

References

97 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. find our card
  2. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024. find our card
  3. AI21. Introducing jamba: Ai21’s groundbreaking ssm-transformer model, 2024. URL https://www.ai21.com/blog/announcing-jamba. ↗
  4. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In ICLR, 2024. find our card
  5. Anthropic. Long context prompting for Claude 2.1. Blog, 2023. URL https://www.anthropic. com/index/claude-2-1-prompting. ↗
  6. Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic. com/news/claude-3-family. ↗
  7. Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. In ICLR, 2024. find
  8. Yushi Bai et al. LongBench: A bilingual, multitask benchmark for long context understanding. arXiv:2308.14508, 2023. find our card
  9. Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024. find
  10. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024. find
  11. Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling Transformer to 1M tokens and beyond with RMT. arXiv:2304.11062, 2023. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) find
  12. David Castillo, Joseph Davidson, Finlay Gray, José Solorzano, and Marek Rosa. Introducing GoodAI LTM benchmark. Blog, 2024. URL https://www.goodai.com/ introducing-goodai-ltm-benchmark/. ↗
  13. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. In ICLR, 2023. find
  14. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongLoRA: Efficient fine-tuning of long-context large language models. In ICLR, 2024. find our card
  15. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse Transformers. arXiv:1904.10509, 2019. find
  16. Cohere. Command r, 2024. model-details. find
  17. URL https://docs.cohere.com/docs/command-r-plus\#Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arxiv:2307.08691, 2023. ↗
  18. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS, 2022. find
  19. Databricks. Introducing dbrx: A new state-of-the-art open llm, 2024. URL https://www. databricks.com/blog/introducing-dbrx-new-state-art-open-llm. find
  20. Jiayu Ding et al. LongNet: Scaling Transformers to 1,000,000,000 tokens. arXiv:2307.02486, 2023. find
  21. Yiran Ding et al. LongRoPE: Extending LLM context window beyond 2 million tokens. arXiv:2402.13753, 2024. find our card
  22. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv:2309.13345, 2023. find
  23. Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards language modeling with state space models. In ICLR, 2023a. find
  24. Daniel Y. Fu et al. Simple hardware-efficient long convolutions for sequence modeling. ICML, 2023b. find
  25. Yao Fu et al. Data engineering for scaling language models to 128k context. arXiv:2402.10171, 2024. find
  26. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka… find our card
  27. Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. arXiv preprint arXiv:2407.00402, 2024. find
  28. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. arXiv:1410.5401, 2014. find
  29. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752, 2023. find
  30. Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022. find
  31. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv:2308.16137, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) find
  32. John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proc of the National Academy of Sciences of the United States of America, 79 8: 2554–8, 1982. find
  33. Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. Transactions of the ACL, 11:284–299, 2023. find
  34. Sam Ade Jacobs et al. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence Transformer models. arXiv:2309.14509, 2023. find our card
  35. Sebastian Jaszczur et al. Sparse is enough in scaling transformers. In NeurIPS, 2021. find
  36. Albert Q Jiang et al. Mixtral of experts. arXiv:2401.04088, 2024. find
  37. Huiqiang Jiang et al. LongLlmLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. arXiv:2310.06839, 2023. find our card
  38. Gregory Kamradt. Needle In A Haystack - pressure testing LLMs. Github, 2023. URL https://github.com/gkamradt/LLMTest NeedleInAHaystack/tree/main. ↗
  39. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A” novel” challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024. find
  40. Lauri Karttunen. Discourse referents. In COLING, 1969. find
  41. George Kingsley Zipf. Selected studies of the principle of relative frequency in language. Harvard university press, 1932. find
  42. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-ina-haystack. arXiv preprint arXiv:2406.10149, 2024. find
  43. Woosuk Kwon et al. Efficient memory management for large language model serving with paged attention. In Proc. of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. find
  44. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121, 2024. find
  45. Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848, 2024. find
  46. Dacheng Li, Rulin Shao, et al. How long can open-source LLMs truly promise on context length?, 2023a. URL https://lmsys.org/blog/2023-06-29-longchat. ↗
  47. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv:2311.04939, 2023b. find our card
  48. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise Transformers for near-infinite context. In ICLR, 2023. find our card
  49. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with Ring Attention. arxiv:2402.08268, 2024a. find
  50. Jiaheng Liu et al. E2-LLM: Efficient and extreme length extension of large language models. arXiv:2401.06951, 2024b. find
  51. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025, 2024c. — [arXiv:2406.06025](https://arxiv.org/abs/2406.06025) find
  52. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the ACL, 12:157–173, 2024d. find our card
  53. Pedro Henrique Martins, Zita Marinho, and Andre Martins. ∞-former: Infinite memory Transformer. In Proc. of the 60th Annual Meeting of the ACL (Volume 1: Long Papers), 2022. find
  54. Meta.AI. Llama 3 model card. 2024a. URL https://github.com/meta-llama/llama3/blob/ main/MODEL CARD.md. ↗
  55. Meta.AI. Llama 3.1 model card. 2024b. URL https://github.com/meta-llama/ llama-models/blob/main/models/llama3 1/MODEL CARD.md. ↗
  56. Mistral.AI. La plateforme, 2023. URL https://mistral.ai/news/la-plateforme/. ↗
  57. Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for Transformers. In Workshop on Efficient Systems for Foundation Models @ ICML, 2023. find
  58. Vincent Ng. Supervised noun phrase coreference research: The first fifteen years. In Proc. of the 48th Annual Meeting of the ACL, 2010. find our card
  59. Catherine Olsson et al. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. ↗
  60. OpenAI: Josh Achiam et al. GPT-4 technical report. arXiv:2303.08774, 2023. find our card
  61. Bo Peng et al. RWKV: Reinventing RNNs for the transformer era. In EMNLP, 2023. find
  62. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In ICLR, 2024. find
  63. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In ICML, 2023. find
  64. Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022. find
  65. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proc. of the 56th Annual Meeting of the ACL (Volume 2: Short Papers), 2018. find our card
  66. Machel Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530, 2024. find our card
  67. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proc. of the 58th Annual Meeting of the ACL, 2020. find
  68. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In EMNLP, 2023. find
  69. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with rotary position embedding. arXiv:2104.09864, 2023. find
  70. Simeng Sun, Katherine Thai, and Mohit Iyyer. ChapterBreak: A challenge dataset for long-range language models. In Proc. of the 2022 Conference of the North American Chapter of the ACL: Human Language Technologies, 2022. find
  71. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to Transformer for large language models. arXiv:2307.08621, 2023a. find
  72. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable Transformer. In Proc. of the 61st Annual Meeting of the ACL (Volume 1: Long Papers), 2023b. find
  73. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254, 2024. find
  74. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In ICLR, 2024. find
  75. Together.AI. Preparing for the era of 32k context: Early learnings and explorations, 2023a. URL https://www.together.ai/blog/llama-2-7b-32k. ↗
  76. Together.AI. Llama-2-7b-32k-instruct — and fine-tuning for llama-2 models with together api, 2023b. URL https://www.together.ai/blog/llama-2-7b-32k-instruct. ↗
  77. Hugo Touvron et al. Llama 2: arXiv:2307.09288, 2023. find
  78. Open foundation and fine-tuned chat models. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the ACL, 10: 539–554, 2022. find our card
  79. Szymon Tworkowski et al. Focused Transformer: Contrastive training for context scaling. NeurIPS, 36, 2024. find
  80. Teun A. van Dijk and Walter Kintsch. Strategies of discourse comprehension. In Academic Press, 1983. find
  81. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. find
  82. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. NeurIPS, 36, 2024. find
  83. Thomas Wolf et al. Huggingface’s Transformers: State-of-the-art natural language processing. arXiv:1910.03771, 2019. find our card
  84. Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-augmented Transformer for sequence modeling. In Findings of the ACL: AACL-IJCNLP, 2022. find
  85. X.AI. Announcing grok-1.5, 2024. URL https://x.ai/blog/grok-1.5. ↗
  86. Chaojun Xiao et al. InfLLM: Unveiling the intrinsic capacity of LLMs for understanding extremely long sequences with training-free memory. arXiv:2402.04617, 2024a. find our card
  87. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024b. find
  88. Wenhan Xiong et al. Effective long-context scaling of foundation models. arXiv:2309.16039, 2023. find our card
  89. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In ICLR, 2024a. find our card
  90. Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint arXiv:2407.16695, 2024b. — [arXiv:2407.16695](https://arxiv.org/abs/2407.16695) find
  91. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. find
  92. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018. find our card
  93. Alex Young et al. Yi: Open foundation models by 01.AI. arXiv:2403.04652, 2024. find our card
  94. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136, 2024. find
  95. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending LLM’s context with activation beacon. arXiv:2401.03462, 2024a. find
  96. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞bench: Extending long context evaluation beyond 100k tokens. arXiv:2402.13718, 2024b. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) find
  97. Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of LLMs via positional skip-wise training. In ICLR, 2024. find