Elamur Research⌕
☰
home / papers / 2026 / RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

Paper · 2026 · Hanbing Liu, Lang Cao and Yang Li

Summary

LLMs learn facts at pretraining time, so they fall out of date as the world changes. They may also mix up older and newer states of the same fact, which the authors call continuous knowledge drift. Continual finetuning, knowledge editing and RAG have rarely been tested on realistic, chronologically ordered streams of change. To fill this gap, the authors build a benchmark of real-world events from 2024–2025. Changes to dynamic entities were tracked through Wikipedia and stored as time-stamped knowledge quadruples, with five question types that probe:

  • recall of older facts,
  • up-to-date facts,
  • multi-step temporal reasoning,
  • retained commonsense.

Experiments show that parametric updates suffer catastrophic forgetting. Vanilla and ReAct-style RAG help on recent facts but lose accuracy on historical ones and struggle with multi-fact temporal aggregation. The authors propose Chronos, a training-free baseline. It retrieves evidence with temporal signals and assembles it into an Event Evolution Graph, presented to the model in temporal and entity views. Chronos yields more stable, temporally consistent answers, which matters for keeping deployed models reliable without retraining.

Key points

  • The benchmark covers January 1, 2024 to October 31, 2025 across ten domains, such as corporate leadership, sports transfers and economic policy. Facts are encoded as (subject, relation, object, timestamp) quadruples extracted with LLMs from Wikipedia.
  • There are five QA tasks:
    • Historical (pre-2024 facts)
    • Contemporary C1 (one timestamp)
    • Contemporary C2 (several timestamps, one entity)
    • Contemporary C3 (several entities and timestamps)
    • Commonsense
  • Chronos accuracy: 63.96 Historical, 96.30 C1, 66.51 C2, 71.19 C3, 71.36 Commonsense.
  • Ablations show every component contributes. Without history reconstruction, Historical accuracy collapses to 6.31. Dropping the entity view costs 17.0 points on C2 and 20.3 points on C3.
  • Code is released at github.com/hbing-l/chronos. A stated limitation is that the benchmark assumes models with a knowledge cutoff before 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hanbing Liu, Lang Cao and Yang Li. 2026. RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.546

@inproceedings{liu2026rag,
  title = {{RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World}},
  author = {Hanbing Liu, Lang Cao and Yang Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.546},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.546}
}

References

45 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abdallah, Bhawna Piryani, Jonas Wallat, Avishek Anand, and Adam Jatowt. 2025. Extending dense passage retrieval with temporal information. arXiv e-prints, pages arXiv–2502. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. 2024. How do large language models acquire factual knowledge during pretraining? Advances in neural information processing systems, 37:60626–60668. find
  4. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  5. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164. find
  6. Guanting Dong, Jiajie Jin, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. 2025. Rag-critic: Leveraging automated critic-guided agentic workflow for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)…
  7. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491– 6501.
  8. Jiacheng Guo, Yue Wu, Jiahao Qiu, Kaixuan Huang, Xinzhe Juan, Ling Yang, and Mengdi Wang. 2025. Temporal consistency for llm reasoning process error identification. arXiv preprint arXiv:2503.14495. find
  9. Jinwen He, Yujia Gong, Zijin Lin, Cheng’an Wei, Yue Zhao, and Kai Chen. 2024. Llm factoscope: Uncovering llms’ factual discernment through measuring inner states. In Findings of the Association for Computational Linguistics ACL 2024, pages 10218– 10230. find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  11. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. arXiv preprint arXiv:2204.14211. find
  12. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403.14403.
  13. Pengcheng Jiang, Lang Cao, Ruike Zhu, Minhao Jiang, Yunyi Zhang, Jimeng Sun, and Jiawei Han. 2025a. Ras: Retrieval-and-structuring for knowledge-intensive llm generation. arXiv preprint arXiv:2502.10996. — [arXiv:2502.10996](https://arxiv.org/abs/2502.10996) find
  14. Pengcheng Jiang, Jiacheng Lin, Lang Cao, R. Tian, S. Kang, Z. Wang, Jimeng Sun, and Jiawei Han. 2025b. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv: 2503.00223. — [arXiv:2503.00223](https://arxiv.org/abs/2503.00223) find
  15. Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, and 1 others. 2024. Learning to edit: Aligning llms with knowledge editing. arXiv preprint arXiv:2402.11905.
  16. Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532–59569.
  17. Hanbing Liu, Jingge Wang, Xuan Zhang, Ye Guo, and Yang Li. 2024. Enhancing continuous domain adaptation with multi-path transfer curriculum. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 286–298. Springer. find
  18. Nattiya Kanhabua and Avishek Anand. 2016. Temporal information retrieval. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 1235–1238. find
  19. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing. find
  20. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. arXiv preprint arXiv:2302.03241. find
  21. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  22. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115.
  23. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  24. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  25. Chen-An Li and Hung-Yi Lee. 2024. Examining forgetting in continual pre-training of aligned large language models. arXiv preprint arXiv:2401.03129. — [arXiv:2401.03129](https://arxiv.org/abs/2401.03129) find
  26. Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. 2024a. Revisiting catastrophic forgetting in large language model tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4297–4308. find
  27. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.11309. find
  28. Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, and Qi Liu. 2025. Hoh: A dynamic benchmark for evaluating the impact of outdated information on retrieval-augmented generation. arXiv preprint arXiv:2503.04800. find
  29. Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. 2024. Memorag: Moving towards next-gen rag via memory-inspired knowledge discovery. arXiv preprint arXiv:2409.05591, 1. — [arXiv:2409.05591](https://arxiv.org/abs/2409.05591) find
  30. Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024b. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18564–18572. find
  31. Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay B Cohen. 2024. Are large language model temporally grounded? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pap… find
  32. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252.
  33. Kartik Sharma, Peeyush Kumar, and Yunqing Li. 2025. Og-rag: Ontology-grounded retrieval-augmented generation for large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32950–32969.
  34. Hanbing Liu, Lang Cao, Yuanyi Ren, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. 2025a. Bingo: Boosting efficient reasoning of llms via dynamic and significance-based reinforcement learning. arXiv preprint arXiv:2506.08125. — [arXiv:2506.08125](https://arxiv.org/abs/2506.08125) find
  35. Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2025. Continual learning of large language models: A comprehensive survey. ACM Computing Surveys, 58(5):1–42. find
  36. Hanbing Liu, Huaze Tang, Yanru Wu, Yang Li, and Xiao-Ping Zhang. 2025b. Reinforced domain selection for continuous domain adaptation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  37. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, and 1 others. 2025. A survey of reasoning with foundation models: Concepts, methodologies, and outlook. ACM Computing Surveys, 57(11):1–43. find
  38. Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Thomas Hartvigsen. 2025. Wikibigedit: Understanding the limits of lifelong knowledge editing in llms. In Forty-second International Conference on Machine Learning. find
  39. Mingyang Wang, Alisa Stoll, Lukas Lange, Heike Adel, Hinrich Schütze, and Jannik Strötgen. 2025. Bring your own knowledge: A survey of methods for llm knowledge expansion. arXiv preprint arXiv:2502.12598. find
  40. Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2024. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Systems, 37:53764–53797. find
  41. Yuqing Wang and Yun Zhao. 2024. Tram: Benchmarking temporal reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6389–6415. find
  42. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  43. Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. Advances in Neural Information Processing Systems, 37:121156– 121184. find
  44. Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. 2025a. Towards lifelong learning of large language models: A survey. ACM Computing Surveys, 57(8):1–35.
  45. Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. 2025b. Lifelong learning of large language model based agents: A roadmap. arXiv preprint arXiv:2501.07278. — [arXiv:2501.07278](https://arxiv.org/abs/2501.07278) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-rag-or-learning-understanding-limits/