📄 Data Foundations of Long-Context Language Models: A Survey

Paper · 2026 · Zechen Sun, Yuyang Sun, Zhaochen Su et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Soochow UniversitySuzhou · China63 works · 4,507 citations🎓Nanjing UniversityNanjing · China74 works · 7,814 citations

🧭 Fields, topics and concepts

Cite

Cite as Zechen Sun, Yuyang Sun, Zhaochen Su, Zecheng Tang, Juntao Li, Ao Zhou, Wenliang Chen and Min Zhang. 2026. Data Foundations of Long-Context Language Models: A Survey. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.775

@article{sun2026data,
  title = {{Data Foundations of Long-Context Language Models: A Survey}},
  author = {Zechen Sun, Yuyang Sun, Zhaochen Su, Zecheng Tang, Juntao Li, Ao Zhou, Wenliang Chen and Min Zhang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.775},
  url = {https://doi.org/10.1162/tacl.a.775}
}

References

161 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. 2025. Longhealth: A question answering benchmark with long clinical documents. Journal of Healthcare Informatics Research, pages 1–17. find
2. Chandra Vamsi Krishna Alla, Harish Naidu Gad-dam, and Manohar Kommi. 2026. Budgetmem: Learning selective memory policies for cost-efficient long-context processing in language models. find
3. Ido Amos, Jonathan Berant, and Ankit Gupta. 2024. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In International Conference on Learning Representations. find
4. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024a. L-eval: Instituting standardized evaluation for long context language models.
5. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14388–14411. find
6. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. 2024b. Why does the effective context length of llms fall short? find
7. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024c. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188. find
8. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024a. Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395.
9. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024b. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti…
10. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025a. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Associ…
11. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025b. Longwriter: Unleashing 10,000+ word generation from long context llms. In The Thirteenth International Conference on Learning Representations.
12. George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense. 2024. Lost in the middle, and in-between: Enhancing language models’ ability to reason over long contexts in multi-hop qa. find
13. Amanda Bertsch, Adithya Pratapa, Teruko Mitamura, Graham Neubig, and Matthew R Gormley. 2025. Oolong: Evaluating long context reasoning and aggregation capabilities. arXiv preprint arXiv:2511.02817. — [arXiv:2511.02817](https://arxiv.org/abs/2511.02817) find
14. Bernd Bohnet, Kevin Swersky, Rosanne Liu, Pranjal Awasthi, Azade Nova, Javier Snaider, Hanie Sedghi, Aaron T Parisi, Michael Collins, Angeliki Lazaridou, et al. 2024. Long-span question-answering: Automatic question generation and qa-system ranking via side-by-side evaluation. arXiv preprint arXiv:2… — [arXiv:2406.00179](https://arxiv.org/abs/2406.00179) find
15. Guanzheng Chen, Michael Qizhe Shieh, and Lidong Bing. 2026a. Longrlvr: Long-context reinforcement learning requires verifiable context rewards. arXiv preprint arXiv:2603.02146. — [arXiv:2603.02146](https://arxiv.org/abs/2603.02146) find
16. Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024a. Fintextqa: A dataset for long-form financial question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6025–6047. find
17. Jianghao Chen, Wei Sun, Qixiang Yin, Zhixing Tan, and Jiajun Zhang. 2025a. Ace-rl: Adaptive constraint-enhanced reward for long-form generation reinforcement learning. arXiv preprint arXiv:2509.04903. — [arXiv:2509.04903](https://arxiv.org/abs/2509.04903) find
18. Jianghao Chen, Junhong Wu, Yangyifan Xu, and Jiajun Zhang. 2025b. LADM: Long-context training data selection with attention-based dependency measurement for LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3076–3090, Vien…
19. Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. 2024b. Long context is not long at all: A prospector of long-dependency data for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
20. Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen, and Dahua Lin. 2025c. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. In Proceedings of the 63rd Annual Meeting of the Association…
21. Ziyang Chen, Xing Wu, Junlong Jia, Chaochen Gao, Qi Fu, Debing Zhang, and Songlin Hu. 2026b. Longbench pro: A more realistic and comprehensive bilingual long-context evaluation benchmark. arXiv preprint arXiv:2601.02872. — [arXiv:2601.02872](https://arxiv.org/abs/2601.02872) find
22. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
23. Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, Tuan A. Tran, Arina Turkatenko, and Carleigh Wood. 2025. LCFO: Long context and long form output dataset and benchmarking. In Findi… find
24. Hao Cui, Zahra Shamsi, Gowoon Cheon, Xue-jian Ma, Shutong Li, Maria Tikhanovskaya, Peter Christian Norgaard, Nayantara Mudur, Martyna Beata Plomecka, Paul Raccuglia, et al. 2025. Curie: Evaluating llms on multitask scientific long-context understanding and reasoning. In The Thirteenth International…
25. Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yi-An Ma, and Yeyun Gong. 2025. Beyond length: Quantifying long-range information for long-context llm pretraining data. arXiv preprint arXiv:2510.25804. — [arXiv:2510.25804](https://arxiv.org/abs/2510.25804) find
26. Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, and Furu Wei. 2025. Longreasonarena: A long reasoning benchmark for large language models. arXiv preprint arXiv:2508.19363. — [arXiv:2508.19363](https://arxiv.org/abs/2508.19363) find
27. Guancheng Du, Yong Hu, Wenqing Wang, Yaming Yang, and Jiaheng Gao. 2025. Sagascale: A realistic, scalable, and high-quality long-context benchmark built from full-length novels. arXiv preprint arXiv:2601.09723. — [arXiv:2601.09723](https://arxiv.org/abs/2601.09723) find
28. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2023. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1):110–120. find
29. Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, and Wei Han. 2024. Retrieval meets reasoning: Dynamic in-context editing for long-text understanding. arXiv preprint arXiv:2406.12331. — [arXiv:2406.12331](https://arxiv.org/abs/2406.12331) find
30. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning, pages 14125–14134.
31. Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. 2025a. Longmagpie: A self-synthesis method for generating large-scale long-context instructions. arXiv preprint arXiv:2505.17134. — [arXiv:2505.17134](https://arxiv.org/abs/2505.17134) find
32. Chaochen Gao, W Xing, Qi Fu, and Songlin Hu. 2025b. Quest: Query-centric data synthesis approach for long-context scaling of large language model. In The Thirteenth International Conference on Learning Representations. find
33. Chaochen Gao, W Xing, Zijia Lin, Debing Zhang, and Songlin Hu. 2025c. Nextlong: Toward effective long-context training without long documents. In Forty-second International Conference on Machine Learning. find
34. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.
35. Yunfan Gao, Yun Xiong, Wenlong Wu, Zijing Huang, Bohan Li, and Haofen Wang. 2025d. U-niah: Unified rag and llm evaluation for long context needle-in-a-haystack. arXiv preprint arXiv:2503.00353. — [arXiv:2503.00353](https://arxiv.org/abs/2503.00353) find
36. Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhu Chen, and Ge Zhang. 2024. Longins: A challenging long-context instruction-based exam for llms. arXiv preprint arXiv:2406.17588. gkamradt. 2023. — [arXiv:2406.17588](https://arxiv.org/abs/2406.17588) find
37. Llmtest-needleinahaystack. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack . ↗
38. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
39. GLM-5-Team. 2026. Glm-5: from vibe coding to agentic engineering.
40. Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16576–16… find
41. Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. 2025. RAPID: Efficient retrieval-augmented long text generation with writing planning and information discovery. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16742… find
42. Zhihan Guo, Jiele Wu, Wenqian Cui, Yifei Zhang, Minda Hu, Yufei Wang, and Irwin King. 2025. From general reward to targeted reward: Improving open-ended long-context generation models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5151–5166. find
43. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Proceedings of the 62nd A… find
44. Linda He, WANG Jue, Maurice Weber, Shang Zhu, Ben Athiwaratkun, and Ce Zhang. 2025. Scaling instruction-tuned llms to million-token contexts via hierarchical synthetic data generation. In The Thirteenth International Conference on Learning Representations. find
45. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189. find
46. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? In First Conference on Language Modeling.
47. Chi-Yang Hsu, Alexander Braylan, Yiheng Su, Omar Alonso, and Matthew Lease. 2025. Instance-level performance prediction for long-form generation tasks. arXiv e-prints, pages arXiv–2509. find
48. Ting-Yao Hu, Hema Swetha Koppula, Hadi Pouransari, Cem Koc, Oncel Tuzel, and Raviteja Vemulapalli. 2025a. Learning from self critique and refinement for faithful llm summarization. arXiv preprint arXiv:2512.05387. — [arXiv:2512.05387](https://arxiv.org/abs/2512.05387) find
49. Ziyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. 2025b. Openreward: Learning to reward long-form agentic tasks via reinforcement learning. arXiv preprint arXiv:2510.24636. — [arXiv:2510.24636](https://arxiv.org/abs/2510.24636) find
50. Junlong Jia, Xing Wu, Chaochen Gao, Ziyang Chen, Zijia Lin, Zhongzhi Li, Weinong Wang, Haotian Xu, Donghui Jin, Debing Zhang, et al. 2025. Litelong: Resource-efficient long-context data synthesis for llms. arXiv preprint arXiv:2509.15568. — [arXiv:2509.15568](https://arxiv.org/abs/2509.15568) find
51. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Fables: Evaluating faithfulness and content selection in book-length summarization. In First Conference on Language Modeling. find
52. Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1–35. find
53. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. Booksum: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6536–6558. find
54. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. 2024. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
55. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: testing the limits of llms with long context reasoning-in-a-haystack. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 10651… find
56. Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885–9903.
57. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. find
58. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024a. Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16304–16333.
59. Jiaxi Li, Xingxing Zhang, Xun Wang, Xiaolong Huang, Li Dong, Liang Wang, Si-Qing Chen, Wei Lu, and Furu Wei. 2025a. Wildlong: Synthesizing realistic long-context instruction data at scale. arXiv preprint arXiv:2502.16684. — [arXiv:2502.16684](https://arxiv.org/abs/2502.16684) find
60. Kuan Li, Liwen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Shuai Wang, and Minhao Cheng. 2025b. Lara: Benchmarking retrieval-augmented generation and long-context llms–no silver bullet for lc or rag routing. In Forty-second International Conference on Machine Learning. find
61. Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024b. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963. — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) find
62. Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, and Wai Lam. 2024c. Large language models can self-improve in long-context reasoning. arXiv preprint arXiv:2411.08147. — [arXiv:2411.08147](https://arxiv.org/abs/2411.08147) find
63. Taiji Li, Hao Chen, Fei Yu, and Yin Zhang. 2025c. Hera: Improving long document summarization using large language models with context packaging and reordering. find
64. Wenhao Li, Bangcheng Sun, Weihao Ye, Tianyi Zhang, Daohai Yu, Fei Chao, and Rongrong Ji. 2026. Ccf: A context compression framework for efficient long-sequence language modeling. find
65. Xuanyu Lei, Chenliang Li, Yuning Wu, Kaiming Liu, Weizhou Shen, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. 2025. Writing-rl: Advancing long-form writing via adaptive curriculum reinforcement learning. arXiv preprint arXiv:2506.05760.
66. Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, and Jordan Lee Boyd-Graber. 2025d. Semantically-aware rewards for open-ended r1 training in free-form generation. arXiv preprint arXiv:2506.15068. — [arXiv:2506.15068](https://arxiv.org/abs/2506.15068) find
67. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
68. Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089. — [arXiv:2501.15089](https://arxiv.org/abs/2501.15089) find
69. Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, and Arman Cohan. 2025a. MDCure: A scalable pipeline for multi-document instruction-following. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29258–29296, Vienn…
70. Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zho… — A Comprehensive Survey on Long Context Language Modeling
71. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
72. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024b. Regmix: Data mixture as regression for language model pre-training. In The Thirteenth International Conference on Learning Representations.
73. Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024c. Longgenbench: Long-context generation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 865–883. find
74. Xiaoran Liu, Kai Lv, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. 2024d. LongWanjuan: Towards systematic measurement for long text quality. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5709–5725, Miami, Florida, USA. Association for Computational Li… find
75. Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. 2024e. Chatqa: Building gpt-4 level conversational qa models. CoRR. find
76. Yida Lu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Cunxiang Wang, Xiaotao Gu, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. 2025. Longsafety: Evaluating long-context safety of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo… find
77. Seiji Maekawa, Hayate Iso, and Nikita Bhutani. 2025. Holistic reasoning with long-context lms: A benchmark for database operations on massive textual data. In The Thirteenth International Conference on Learning Representations. find
78. Mounika Marreddy, Subba Reddy Oota, Venkata Charan Chinni, Manish Gupta, and Lucie Flek. 2025. USDC: A dataset of U ser Stance and Dogmatism in long Conversations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23715–23759, Vienna, Austria. Association for Computationa… find
79. MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaish… — MiniMax-01: Scaling Foundation Models with Lightning Attention
80. Purbesh Mitra and Sennur Ulukus. 2025. Semantic soft bootstrapping: Long context reasoning in llms without reinforcement learning. arXiv preprint arXiv:2512.05105. — [arXiv:2512.05105](https://arxiv.org/abs/2512.05105) find
81. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A Rossi, Seunghyun Yoon, and Hinrich Schuetze. 2025a. Nolima: Long-context evaluation beyond literal matching. In Forty-second International Conference on Machine Learning. find
82. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A Rossi, Seunghyun Yoon, and Hinrich Schuetze. 2025b. Nolima: Long-context evaluation beyond literal matching. In International Conference on Machine Learning, pages 44554–44570. PMLR. find
83. Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, and Piji Li. 2024. Xl 2 bench: A benchmark for extremely long context understanding with long-range dependencies. find
84. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
85. Miao Peng, Weizhou Shen, Nuo Chen, Chenliang Li, Ming Yan, and Jia Li. 2026. Incentivizing in-depth reasoning over long contexts with process advantage shaping. arXiv preprint arXiv:2601.12465.
86. Bowen Ping, Zijun Chen, Yiyao Yu, Tingfeng Hui, Junchi Yan, and Baobao Chang. 2026. Longr: Unleashing long-context reasoning via reinforcement learning with dense utility rewards. arXiv preprint arXiv:2602.05758. — [arXiv:2602.05758](https://arxiv.org/abs/2602.05758) find
87. Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, and Shanghang Zhang. 2025. LongDPO: Unlock better long-form generation abilities for LLMs via critique-augmented stepwise information. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7613–7632, Vienna, Austria.… find
88. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191. — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) find
89. Stefano Rando, Luca Romani, Alessio Sampieri, Yuta Kyuragi, Luca Franco, Fabio Galasso, Tatsunori Hashimoto, and John Yang. 2025. Longcodebench: Evaluating coding llms at 1m context windows. arXiv preprint arXiv:2505.07897. — [arXiv:2505.07897](https://arxiv.org/abs/2505.07897) find
90. Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. 2024. Docfinqa: A long-context financial reasoning dataset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 445–458. find
91. Yiming Ren, Junjie Wang, Yuxin Meng, Yihang Shi, Zhiqiang Lin, Ruihang Chu, Yiran Xu, Ziming Li, Yunfei Zhao, Zihan Wang, et al. 2026. Sin-bench: Tracing native evidence chains in long-context multimodal scientific interleaved literature. arXiv preprint arXiv:2601.10108. — [arXiv:2601.10108](https://arxiv.org/abs/2601.10108) find
92. Jonathan Roberts, Kai Han, and Samuel Albanie. 2025. Needle threading: Can llms follow threads through near-million-scale haystacks? In The Thirteenth International Conference on Learning Representations. find
93. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
94. Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, et al. 2025. Expertlongbench: Benchmarking language models on expert-level long-form generation tasks with structured checklists. arXiv preprint arXiv:2506.0124… — [arXiv:2506.01241](https://arxiv.org/abs/2506.01241) find
95. Alireza Salemi, Julian Killingback, and Hamed Zamani. 2025. Expert: Effective and explainable evaluation of personalized long-form text generation. find
96. Alfred Shen and Aaron Shen. 2026. Gated sparse attention: Combining computational efficiency with training stability for long-context language models. find
97. Weizhou Shen, Ziyi Yang, Chenliang Li, Zhiyuan Lu, Miao Peng, Huashan Sun, Yingcheng Shi, Shengyi Liao, Shaopeng Lai, Bo Zhang, et al. 2025. Qwenlong-l1. 5: Post-training recipe for long-context reasoning and memory management. arXiv preprint arXiv:2512.12967.
98. Dachuan Shi, Yonggan Fu, Xiangchi Yuan, Zhongzhi Yu, Haoran You, Sixu Li, Xin Dong, Jan Kautz, Pavlo Molchanov, Yingyan, and Lin. 2025. Lacache: Ladder-shaped kv caching for efficient long-context modeling of large language models. find
99. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. 2024. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations.
100. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025a. PRMBench: A fine-grained and challenging benchmark for process-level reward models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25299–25346, Vienna,… find
101. Mingyang Song, Mao Zheng, and Xuan Luo. 2025b. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3753– 3763, Abu Dhabi, UAE. Association f…
102. Mingyang Song, Mao Zheng, and Xuan Luo. 2025c. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3753– 3763.
103. Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Yu Zhao, Henryk Michalewski, Łukasz Kuciński, and Piotr Miłoś. 2025. Structured packing in llm training improves long context utilization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25201–25209. find
104. Zechen Sun, Yuyang Sun, Zecheng Tang, Juntao Li, Wenpeng Hu, Wenliang Chen, Zhunchen Luo, Guotong Geng, and Min Zhang. 2026a. Is-cot: Breaking the long-form generation collapse via interleaved structural thinking.
105. Zechen Sun, Zecheng Tang, Juntao Li, Wenpeng Hu, Wenliang Chen, Zhunchen Luo, and Qiaoming Zhu. 2026b. Cgmis: Concept-graph based multi-hop instructions synthesis for enhancing long-context reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33153–33161. find
106. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, et al. 2024. Prox-yqa: An alternative framework for evaluating long-form text generation with large language models. In Proceedings of the 62nd Annual Meeting of the Association fo… find
107. Zecheng Tang, Baibei Ji, Ruoxi Sun, Haitian Wang, WangJie You, Zhang Yijun, Wenpeng Zhu, Ji Qi, Juntao Li, and Min Zhang. 2026. Memoryrewardbench: Benchmarking reward models for long-term memory management in large language models. arXiv preprint arXiv:2601.11969. — [arXiv:2601.11969](https://arxiv.org/abs/2601.11969) find
108. Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. 2025. Logo—long context alignment via efficient preference optimization. In Forty-second International Conference on Machine Learning. find
109. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng Chen, Guanduo Chen, Huarong Chen, Jia Chen, Jiahao Chen, Jianlong Chen, Jun Chen, Kefan Chen, Liang Chen, Ruijue Chen, Xinhao Chen, Yanru Chen, Yanxu Chen, Yicun Chen, Yimin Chen, Yingjiang Chen, Yuanku… — Kimi K2.5: Visual Agentic Intelligence
110. Junfeng Tian, Da Zheng, Yang Chen, Rui Wang, Colin Zhang, and Debing Zhang. 2025. Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
111. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
112. Austin Veselka. 2026. How to train your long-context visual document model. arXiv preprint arXiv:2602.15257. — [arXiv:2602.15257](https://arxiv.org/abs/2602.15257) find
113. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640.
114. Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang, Jingren Zhou, and Ming Yan. 2025a. Qwenlong-l1: Towards long-context large reasoning models with reinforcement learning.
115. Kaiyang Wan, Honglin Mu, Rui Hao, Haoran Luo, Tianle Gu, and Xiuying Chen. 2025b. A cognitive writing perspective for constrained long-form text generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9832–9844, Vienna, Austria. Association for Computational Linguis… find
116. Luanbo Wan and Weizhi Ma. 2025. Storybench: A dynamic benchmark for evaluating long-term memory with multi turns. arXiv preprint arXiv:2506.13356. — [arXiv:2506.13356](https://arxiv.org/abs/2506.13356) find
117. Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. 2024a. Novelqa: A benchmark for long-range novel question answering. arXiv e-prints, pages arXiv– 2403. find
118. Haozhe Wang, Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, et al. 2025a. Reverse-engineered reasoning for open-ended generation. arXiv preprint arXiv:2509.06160. — [arXiv:2509.06160](https://arxiv.org/abs/2509.06160) find
119. Zhunchen Luo, Wei Luo, Xiaoying Bai, and Guotong Geng. 2025c. Define: A decomposed and fine-grained annotated dataset for long-form article generation. find
120. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024b. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. In EMNLP.
121. Qianyue Wang, Jinwu Hu, Zhengping Li, Yufeng Wang, Daiyuan Li, Yu Hu, and Mingkui Tan. 2025d. Generating long-form story using dynamic hierarchical outlining with memory-enhancement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin… find
122. Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Ar-maghan Eshaghi. 2024c. Beyond the limits: A survey of techniques to extend the context length in large language models. In IJCAI.
123. Zhchao Wang, Bin Bi, Yanqi Luo, Sitaram Asur, and Claire Na Cheng. 2025e. Diversity enhances an llm’s performance in rag and long-context task. arXiv preprint arXiv:2502.09017. — [arXiv:2502.09017](https://arxiv.org/abs/2502.09017) find
124. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, et al. 2024. Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:80756–80827.
125. Lanrui Wang, Mingyu Zheng, Hongyin Tang, Zheng Lin, Yanan Cao, Jingang Wang, Xunliang Cai, and Weiping Wang. 2025b. Needleinatable: Exploring long-context capability of large language models towards long-structured tables. find
126. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Tanglifu Tanglifu, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. Light-r1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In Proceedings of the 63rd Annual Me… find
127. Ming Wang, Fang Wang, Minghao Hu, Li He, Haiyang Wang, Jun Zhang, Tianwei Yan, Li Li, Chen Wu and Yin Song. 2025. Scaling context, not parameters: Training a compact 7B language model for efficient long-context processing. In Proceedings of the 63rd Annual Meeting of the Association for Computationa… find
128. Longyun Wu, Dawei Zhu, Guangxiang Zhao, Zhuocheng Yu, Junfeng Ran, Xiangyu Wong, Lin Sun, and Sujian Li. 2025a. LongAttn: Selecting long-context training data via token-level attention. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19367–19380, Vienna, Austria. Associ… find
129. Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. 2024. Long context alignment with short instructions and synthesized positions. arXiv preprint arXiv:2405.03939. — [arXiv:2405.03939](https://arxiv.org/abs/2405.03939) find
130. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Juanzi Li, and Roy Ka-Wei Lee. 2025b. Superwriter: Reflection-driven long-form generation with large language models. arXiv preprint arXiv:2506.04180.
131. Yao Xiao, Lei Wang, Yue Deng, Guanzheng Chen, Ziqi Jin, Jung-jae Kim, Xiaoli Li, Roy Ka-wei Lee, and Lidong Bing. 2026. Document reconstruction unlocks scalable long-context rlvr. arXiv preprint arXiv:2602.08237. — [arXiv:2602.08237](https://arxiv.org/abs/2602.08237) find
132. Zikai Xiao, Fei Huang, Jianhong Tu, Jianhui Wei, Wen Ma, Yuxuan Zhou, Jian Wu, Bowen Yu, Zuozhu Liu, and Junyang Lin. 2025. Long-Weave: A long-form generation benchmark bridging real-world relevance and verifiability. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10… find
133. In The Thirteenth International Conference on Learning Representations. find
134. Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. 2024. Stress-testing long-context language models with lifelong icl and task haystack. Advances in Neural Information Processing Systems, 37:15801–15840. find
135. Kai Yan, Zhan Ling, Kang Liu, Yifan Yang, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025a. Mir-bench: Benchmarking llm’s long-context intelligence via many-shot in-context inductive reasoning. arXiv preprint arXiv:2502.09933. find
136. Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. 2025b. Inftythink: Breaking the length limits of long-context reasoning in large language models. arXiv preprint arXiv:2503.06692.
137. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
138. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
139. Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. 2025b. LongFaith: Enhancing long-context reasoning in LLMs with faithful synthetic data. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3236–3256, Vienna, Austria. Assoc… find
140. Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. 2025. From artificial needles to real haystacks: Improving retrieval capabilities in llms by finetuning on synthetic data. In The Thirteenth International Conference on Learning Representations. find
141. Joonho Yang, Seunghyun Yoon, Hwan Chang, Byeongjeong Kim, and Hwanhee Lee. 2026. Hallucinate at the last in long response generation: A case study on long document summarization. find
142. Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities. find
143. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Sen Yang, Nigel Collier, Dong Yu, and Deqing Yang. 2024b. Logu: Long-form generation with uncertainty expressions.
144. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Dong Yu, Nigel Collier, and Deqing Yang. 2025c. Uncle: Uncertainty expressions in long-form generation. arXiv preprint arXiv:2505.16922. — [arXiv:2505.16922](https://arxiv.org/abs/2505.16922) find
145. Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen.… find
146. Wang Yang, Zirui Liu, Hongye Jin, Qingyu Yin, Vipin Chaudhary, and Xiaotian Han. 2025d. Longer context, deeper thinking: Uncovering the role of long-context ability in reasoning. find
147. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2025a. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. find
148. Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. 2025b. Longproc: Benchmarking long-context language models on long procedural generation. find
149. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2025. Helmet: How to evaluate long-context language models effectively and thoroughly. International Conference on Learning Representations (ICLR). find
150. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shen-gen Yan, and Yu Wang. 2024. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k.
151. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
152. Weihao Zeng, Yuzhen Huang, and Junxian He. 2026. Loca-bench: Benchmarking language agents under controllable and extreme context growth. arXiv preprint arXiv:2602.07962. — [arXiv:2602.07962](https://arxiv.org/abs/2602.07962) find
153. Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Haozhen Zhang, Tao Feng, Pengrui H… find
154. Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2025b. LongReward: Improving long-context large language models with AI feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1…
155. Junhao Zhang, Richong Zhang, Fanshuang Kong, Ziyang Miao, Yanhan Ye, and Yaowei Zheng. 2025c. Lost-in-the-middle in long-text generation: Synthetic dataset, evaluation framework, and mitigation. arXiv preprint arXiv:2503.06868. — [arXiv:2503.06868](https://arxiv.org/abs/2503.06868) find
156. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024b. ∞Bench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
157. Yikai Zhang, Junlong Li, and Pengfei Liu. 2024c. Extending llms’ context window with 100 samples. arXiv preprint arXiv:2401.07004. — [arXiv:2401.07004](https://arxiv.org/abs/2401.07004) find
158. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).
159. Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024. Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 18–37.
160. Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. 2025. Generalizing from short to long: Effective data synthesis for long-context instruction tuning. arXiv preprint arXiv:2502.15592. — [arXiv:2502.15592](https://arxiv.org/abs/2502.15592) find
161. Yonghao Zhuang, Lanxiang Hu, Longfei Yun, Souvik Kundu, Zhengzhong Liu, Eric P Xing, and Hao Zhang. 2025. Scaling long context training data by long-distance referrals. In 13th International Conference on Learning Representations, ICLR 2025, pages 81355–81372. International Conference on Learning Re… find

Updated 1 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-data-foundations-long-context-language/