Elamur Research⌕
☰
home / papers / 2024 / LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

Paper · 2024 · Tao Yuan, Xuefei Ning, Dong Zhou et al.

Summary

Long-context LLMs now advertise input windows of 128k–256k tokens or more. Popular evaluation suites, however, average only about 5k–21k words. They often reuse unmodified public texts that models may have memorized, and they score answers with brittle n-gram metrics. LV-Eval addresses these gaps with a bilingual (English/Chinese) QA benchmark. It is built from existing public datasets and organized into five length tiers, from 16k to 256k words. Every tier uses identical question-answer pairs, so only the context size changes, which allows controlled comparison across lengths.

The construction relies on three techniques:

  • Supporting documents are buried among distracting ones.
  • GPT-4 drafts misleading statements, which annotators edit before inserting them into the context.
  • Human-written substitution rules alter keywords and phrases to reduce reliance on memorized knowledge.

Scoring first checks recall of annotated answer keywords. It then computes F1 after removing blacklisted filler words.

Across 15 tested LLMs, Moonshot-v1, Qwen-2.5-72B and Llama-3.1-70B score highest, mostly below 64k. All models drop sharply when misleading content is present. The benchmark offers a harder, less leakage-prone test of long-context claims.

Key points

  • LV-Eval contains 11 QA datasets (6 English, 5 Chinese), split into single-hop and multi-hop tasks. Examples include cmrc-mixup, dureader-mixup, hotpotwikiqa-mixup, loogle-SD/CR/MR-mixup and the needle-style factrecall-en/zh.
  • Contexts average about 102k words, with a maximum near 387k. LongBench, LooGLE, L-Eval, BAMBOO and ZeroSCROLLS average roughly 5k–21k.
  • The core techniques are CFI, which inserts confusing facts into the context; KPR, which replaces selected keywords and phrases; and a two-stage metric built on keyword recall.
  • GLM-4-9B-128k, Yi-6B-200k and Llama3-8B-1M lose accuracy more gradually as length grows, but they do not necessarily beat shorter-context models in absolute score.
  • Ablations show large effects from CFI and KPR. On factrecall-zh at 16k, ChatGLM3-6B-32k falls from 63.19 without both techniques to 0 with both. Data and evaluation code are available at github.com/infinigence/LVEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan and Yu Wang. 2024. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K. arXiv:2402.05136.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05136 · arXiv 2402.05136

@misc{yuan2024lv,
  title = {{LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K}},
  author = {Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan and Yu Wang},
  year = {2024},
  doi = {10.48550/arxiv.2402.05136},
  eprint = {2402.05136},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05136}
}

References

39 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. AI@Meta. Llama 3 model card. https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md, 2024. ↗
  3. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023.
  4. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023a.
  5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023b. bloc97. Ntk-aware scaled rope allows llama models to ex…
  6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p… find
  7. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for chinese machine reading comprehension. arXiv preprint arXiv:1810.07366, 2018.
  8. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. find
  9. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.13345, 2023. gradient.ai. Llama-3 8b gradient instruct 1048k model card. https://huggingface.co/ grad… ↗
  10. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) find
  11. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020.
  12. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.
  13. Greg Kamradt. Pressure testing gpt-4-128k with long context recall. https://twitter.com/ GregKamradt/status/1722386725635580292, 2023. ↗
  14. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2019. find
  15. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. find
  16. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939, 2023b.
  17. Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint arXiv:2304.13343, 2023. — [arXiv:2304.13343](https://arxiv.org/abs/2304.13343) find
  18. Pedro Henrique Martins, Zita Marinho, and André FT Martins. ∞-former: Infinite memory transformer-former: Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5468–5485, 2022. find
  19. Meta. Llama-3.1-70b-instruct model card. Llama-3.1-70B-Instruct, 2024. https://huggingface.co/meta-llama/moonshot. moonshot model site. https://www.moonshot.cn/, 2024. ↗
  20. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023a. find
  21. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023b. find
  22. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. find
  23. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
  24. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196, 2023.
  25. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  26. Hongxuan Tang, Hongyu Li, Jing Liu, Yu Hong, Hua Wu, and Haifeng Wang. Dureader robust: A chinese dataset towards evaluating robustness and generalization of machine reading comprehension in real-world applications. arXiv preprint arXiv:2004.11142, 2020.
  27. BlueLM Team. Bluelm: An open multilingual 7b language model. https://github.com/ vivo-ai-lab/BlueLM, 2023. togetherAI. Llama-2-7b-32k-instruct — and fine-tuning for llama-2 models with together api. https://www.together.ai/blog/llama-2-7b-32k-instruct, ↗
  28. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  29. Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2021. find
  30. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. find
  31. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023.
  32. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. find
  33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018.
  34. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420, 2023. find
  35. Yi. A series of large language models trained from scratch by developers at 01-ai. https://github.com/01-ai/Yi, 2023. yuyijiong. Long-instruction-en2zh. https://huggingface.co/datasets/yuyijiong/ Long-instruction-en2zh, 2023. ↗
  36. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
  37. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  38. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023a. find
  39. Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint arXiv:2305.13304, 2023b.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2024-lv-eval-balanced-long-context/