Elamur Research⌕
☰
home / papers / 2024 / BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

Paper · 2024 · Yuri Kuratov, Aydar Bulatov, Petr Anokhin et al.

Summary

Context windows of LLMs now reach hundreds of thousands or millions of tokens. Yet common evaluations either stop around 40K tokens (Longbench, L-Eval) or use needle-in-a-haystack retrieval, which strong models already solve perfectly. BABILong fills this gap with the 20 synthetic bAbI reasoning tasks, which cover fact chaining, induction, deduction, counting and lists/sets. Their sentences are hidden inside PG19 book text: background sentences are added in their original order until a sample reaches the target length. Because samples are generated, length can be set freely and test items cannot leak into training corpora. The authors evaluate more than 30 systems, including open models (LLama-3.1, Qwen-2.5, Mistral and others), GPT-4, Gemini 1.5 Pro, RAG pipelines, Mamba, RMT and ARMT. Most LLMs draw useful information from only roughly a tenth to a fifth of their window, and accuracy drops as more facts must be combined. Small fine-tuned recurrent models, however, solve the tasks at extreme lengths. The result is a scalable, contamination-resistant stress test for future long-context models.

Key points

  • Resource: tasks QA1–QA20 are built on bAbI, with PG19 as background text (Wiki was also tried). Ready-made splits run from 0K to 10M tokens, and models were tested on samples up to 50M tokens. Code is at github.com/booydar/babilong, the dataset is RMT-team/babilong on Hugging Face, and there is a public leaderboard.
  • Context use: GPT-4 relies on only about 10% of its 128K window, while Gemini 1.5 Pro stays strong up to about 64K. Among pre-trained open models, LLama-3.1 and Qwen-2.5 score best.
  • RAG: retrieval reaches only about 60% on single-fact questions, regardless of length. It breaks down when the order of facts matters.
  • Fine-tuning: small fine-tuned models solve the tasks, namely RMT/ARMT on GPT-2 (137M) and Mamba (130M). RMT/ARMT handle up to 50M tokens, a record for a single model. Mamba is hard to run beyond 128K. Fine-tuning also helps GPT-3.5-Turbo and Mistral-7B, but their windows cap at 16K and 32K.
  • Limitations: the small bAbI vocabulary makes facts easy for fine-tuned models to spot. The tasks are English-only, and the RAG retriever and prompts were not optimized.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin and Mikhail Burtsev. 2024. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. arXiv:2406.10149.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.10149 · arXiv 2406.10149

@misc{kuratov2024babilong,
  title = {{BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack}},
  author = {Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin and Mikhail Burtsev},
  year = {2024},
  doi = {10.48550/arxiv.2406.10149},
  eprint = {2406.10149},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.10149}
}

References

89 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
  2. Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024.
  3. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md. ↗
  4. An, C., Gong, S., Zhong, M., Li, M., Zhang, J., Kong, L., and Qiu, X. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023.
  5. Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic.com/news/claude-3-family. ↗
  6. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.
  7. Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., Tang, J., Dong, Y., and Li, J. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058, 2024.
  8. Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  9. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206–2240. PMLR, 2022. find
  10. Bulatov, A., Kuratov, Y., and Burtsev, M. Recurrent memory transformer. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 11079–11091. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/… ↗
  11. Bulatov, A., Kuratov, Y., Kapushev, Y., and Burtsev, M. Beyond attention: Breaking the limits of transformer context length with recurrent memory. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17700–17708, Mar. 2024. doi: 10.1609/aaai.v38i16.29722. URL https://ojs.aaai.org/in… — [doi:10.1609/aaai.v38i16.29722](https://doi.org/10.1609/aaai.v38i16.29722) ↗
  12. Chase, H. LangChain, October 2022. URL https://github.com/langchain-ai/langchain. ↗
  13. Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2023.
  14. Chevalier, A., Wettig, A., Ajith, A., and Chen, D. Adapting language models to compress contexts. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829–3846, Singapore, December 2023. Association for Computatio…
  15. Clark, P., Tafjord, O., and Richardson, K. Transformers as soft reasoners over language. In Bessiere, C. (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3882–3890. ijcai.org, 2020. doi: 10.24963/ijcai.2020/537. URL https://doi.org/10.…
  16. Cohere. Command r: Retrieval-augmented generation at production scale, March 2024. URL https://cohere.com/blog/command-r. ↗
  17. Didolkar, A., Gupta, K., Goyal, A., Gundavarapu, N. B., Lamb, A. M., Ke, N. R., and Bengio, Y. Temporal latent bottleneck: Synthesis of fast and slow processing mechanisms in sequence learning. Advances in Neural Information Processing Systems, 35:10505–10520, 2022. find
  18. Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023.
  19. Dong, Z., Tang, T., Li, J., Zhao, W. X., and Wen, J.-R. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.13345, 2023.
  20. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. 2024.
  21. Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320–335, 2022.
  22. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  23. Fan, A., Lavril, T., Grave, E., Joulin, A., and Sukhbaatar, S. Addressing some limitations of transformers with feedback memory. arXiv preprint arXiv:2002.09402, 2020. — [arXiv:2002.09402](https://arxiv.org/abs/2002.09402) find
  24. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. find
  25. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  26. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. find
  27. Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021. find
  28. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020. — REALM: Retrieval-Augmented Language Model Pre-Training
  29. Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Benson, L., Sun, L., Zubova, E., Qiao, Y., Burtell, M., Peng, D., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., Zhang, R., Joty, S. R., Fabbri, A. R., Kryscinski, W., Lin, X. V., Xiong, C., and Radev, D. FOLIO: nat… — FOLIO: Natural Language Reasoning with First-Order Logic
  30. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
  31. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735. ↗
  32. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.
  33. Hutchins, D., Schlag, I., Wu, Y., Dyer, E., and Neyshabur, B. Block-recurrent transformers. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=uloenYmLCAo. ↗
  34. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  35. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  36. Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2019. find
  37. Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. find
  38. Lei, F., Liu, Q., Huang, Y., He, S., Zhao, J., and Liu, K. S3eval: A synthetic, scalable, systematic evaluation suite for large language models. arXiv preprint arXiv:2310.15147, 2023. — [arXiv:2310.15147](https://arxiv.org/abs/2310.15147) find
  39. Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T. L., and Bansal, M. Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning, 2020. find
  40. Li, D., Shao, R., Xie, A., Sheng, Y., Zheng, L., Gonzalez, J., Stoica, I., Ma, X., and Zhang, H. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. find
  41. Li, J., Wang, M., Zheng, Z., and Zhang, M. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939, 2023b.
  42. Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024.
  43. Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024.
  44. Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024a.
  45. Liu, Z., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catanzaro, B. Chatqa: Building gpt-4 level conversational qa models. arXiv preprint arXiv:2401.10225, 2024b.
  46. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  47. Ni, X., Cai, H., Wei, X., Wang, S., Yin, D., and Li, P. XL 2 Bench: A benchmark for extremely long context understanding with long-range dependencies. arXiv preprint arXiv:2404.05446, 2024. — [arXiv:2404.05446](https://arxiv.org/abs/2404.05446) find
  48. OpenAI. Gpt-4 technical report, 2023a.
  49. OpenAI. New models and developer products announced at devday, 2023b. URL https://openai.com/index/new-models-and-developer-products-announced-at-devday/. ↗
  50. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023a. find
  51. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023b. find
  52. Poulis, A., Tsalapati, E., and Koubarakis, M. Transformer-based language models for reasoning in the description logic alcq. arXiv preprint arXiv:2410.09613, 2024. — [arXiv:2410.09613](https://arxiv.org/abs/2410.09613) find
  53. Qiu, Z., Li, J., Huang, S., Zhong, W., and King, I. Clongeval: A chinese benchmark for evaluating long-context large language models. arXiv preprint arXiv:2403.03514, 2024. find
  54. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. find
  55. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SylKikSYDH. ↗
  56. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, 2016.
  57. Reddy, V., Koncel-Kedziorski, R., Lai, V. D., and Tanner, C. Docfinqa: A long-context financial reasoning dataset. arXiv preprint arXiv:2401.06915, 2024. find
  58. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  59. Rodkin, I., Kuratov, Y., Bulatov, A., and Burtsev, M. Associative recurrent memory transformer. In ICML Workshop Next Generation of Sequence Modeling Architectures, 2024. URL https://arxiv.org/abs/2407.04841. ↗
  60. Rubin, O. and Berant, J. Long-range language modeling with self-retrieval. arXiv preprint arXiv:2306.13421, 2023. — [arXiv:2306.13421](https://arxiv.org/abs/2306.13421) find
  61. Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023,…
  62. Saparov, A. and He, H. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
  63. OpenReview.net, 2023. URL https://openreview.net/pdf?id=qFVVBzXxR2V. ↗
  64. Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pp. 9355–9366. PMLR, 2021.
  65. Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007–12021, 2022.
  66. Shaham, U., Ivgi, M., Efrat, A., Berant, J., and Levy, O. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196, 2023.
  67. Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
  68. Song, D., Chen, S., Chen, G. H., Yu, F., Wan, X., and Wang, B. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532, 2024a. — [arXiv:2404.18532](https://arxiv.org/abs/2404.18532) find
  69. Song, M., Zheng, M., and Luo, X. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models, 2024b.
  70. Sorokin, A., Buzun, N., Pugachev, L., and Burtsev, M. Explain my surprise: Learning efficient long-term memory by predicting uncertain outcomes. Advances in Neural Information Processing Systems, 35:36875–36888, 2022. find
  71. Sun, S., Thai, K., and Iyyer, M. Chapterbreak: A challenge dataset for long-range language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3704–3714, 2022. find
  72. Tafjord, O., Dalvi, B., and Clark, P. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume…
  73. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL — Long Range Arena: A Benchmark for Efficient Transformers https://openreview.net/forum?id=qVyeW-grC2k. ↗
  74. Team, Q. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/. ↗
  75. Tian, J., Li, Y., Chen, W., Xiao, L., He, H., and Jin, Y. Diagnosing the first-order logical reasoning ability through LogicNLI. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Eve…
  76. Voelker, A., Kajić, I., and Eliasmith, C. Legendre memory units: Continuous-time representation in recurrent neural networks. Advances in neural information processing systems, 32, 2019. find
  77. Wang, C., Duan, H., Zhang, S., Lin, D., and Chen, K. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. arXiv preprint arXiv:2404.06480, 2024a. find
  78. Wang, C., Ning, R., Pan, B., Wu, T., Guo, Q., Deng, C., Bao, G., Wang, Q., and Zhang, Y. Novelqa: A benchmark for long-range novel question answering. arXiv preprint arXiv:2403.12766, 2024b.
  79. Wang, S., Bai, Y., Zhang, L., Zhou, P., Zhao, S., Zhang, G., Wang, S., Chen, R., Xu, H., and Sun, H. Xl3m: A training-free framework for llm length extension based on segment-wise inference. arXiv preprint arXiv:2405.17755, 2024c. — [arXiv:2405.17755](https://arxiv.org/abs/2405.17755) find
  80. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 20…
  81. Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., and Yu, Z. Memformer: A memory-augmented transformer for sequence modeling. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pp. 308–318, Online only, November 2022a. Association for Computational Linguistics. URL http… find
  82. Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=TrjbxzRcnf-. ↗
  83. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, 2018.
  84. Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
  85. Yuan, T., Ning, X., Zhou, D., Yang, Z., Li, S., Zhuang, M., Tan, Z., Yao, Z., Lin, D., Li, B., et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136, 2024.
  86. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: Transformers for longer sequences. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing…
  87. Zhang, L., Li, Y., Liu, Z., Liu, J., Yang, M., et al. Marathon: A race through the realm of long context with large language models. arXiv preprint arXiv:2312.09542, 2023. find
  88. Zhang, P., Liu, Z., Xiao, S., Shao, N., Ye, Q., and Dou, Z. Soaring from 4k to 400k: Extending llm’s context with activation beacon. arXiv preprint arXiv:2401.03462, 2024a.
  89. Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., et al. ∞ bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024b. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuratov-2024-babilong-testing-limits-llms-long/