Elamur Research⌕
☰
home / papers / 2026 / Test of Time: Rethinking Temporal Signal of Benchmark Contamination

Test of Time: Rethinking Temporal Signal of Benchmark Contamination

Paper · 2026 · Terry Jingchen Zhang, Gopal Dev, Ning Wang et al.

Summary

Researchers often treat a drop in LLM accuracy on questions published after a model's training cutoff as evidence that older questions leaked into its training data. This paper argues that this temporal signal is fragile, because it depends strongly on how the questions are built. The authors collected 20,277 mathematics and physics arXiv papers covering 26 months and tested 8 models from four families. Questions that an LLM synthesized from these papers showed no post-cutoff decline. Fill-in-the-blank (cloze) items taken from the abstracts of the same papers did show it.

The effect also held outside arXiv. Having an LLM rewrite LiveCodeBench problems and FreshQA-style Wikipedia questions, with the solutions unchanged, removed the decay that had been observed before. Influence function analysis on deliberately contaminated documents explains why. Cloze prompts led models to rely on the source paper, while generated questions rarely did.

The authors warn that missing decay does not prove a benchmark is clean. They call for more robust contamination probes and for clear documentation of benchmark construction.

Key points

  • Data: 20,277 arXiv papers (May 2023 – June 2025, math and physics) yielded 1,643 LLM-generated questions plus cloze questions built from the abstracts of the same papers.
  • Models: DeepSeek-R1-0528, DeepSeek-R1, OpenAI-o4-mini, OpenAI-o3-mini, Gemini-2.5-Flash, Gemini-2.0-Flash, Llama-4-Scout and Llama-3.3-70B, with knowledge cutoffs ranging from 2023.10 to 2025.01.
  • An LLM-driven rewrite of LiveCodeBench, a benchmark previously reported to show post-cutoff decay, removed that temporal pattern without changing the solutions. A FreshQA-style Wikipedia set behaved the same way.
  • Influence function analysis was run on 40 papers. For cloze questions, the source paper was the most influential training document 77.5% of the time (Top-1) and always appeared in the Top-3. For LLM-generated questions, the Top-1 rate was 17.5% and the Top-3 rate was 25.0%.
  • Code is published at github.com/AI4Collaboration/Test-of-Time. The authors note two limitations: budget constrained the sample size, and question difficulty may act as a confounder.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓ETH ZurichZurich · Switzerland72 works · 5,176 citations🔬Vector InstituteToronto · Canada33 works · 4,535 citations🔬Max Planck SocietyMunich · Germany16 works · 42,253 citations🔬Max Planck Institute for Intelligent SystemsStuttgart · Germany8 works · 854 citations🔬ELLIS Institute TübingenTübingen · Germany2 works · 3 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Terry Jingchen Zhang, Gopal Dev, Ning Wang, Max Obreiter, Punya Syon Pandey, Keenan Samway, Wenyuan Jiang, Yinya Huang, Bernhard Schölkopf, Mrinmaya Sachan and Zhijing Jin. 2026. Test of Time: Rethinking Temporal Signal of Benchmark Contamination. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1693

@inproceedings{zhang2026test,
  title = {{Test of Time: Rethinking Temporal Signal of Benchmark Contamination}},
  author = {Terry Jingchen Zhang, Gopal Dev, Ning Wang, Max Obreiter, Punya Syon Pandey, Keenan Samway, Wenyuan Jiang, Yinya Huang, Bernhard Schölkopf, Mrinmaya Sachan and Zhijing Jin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1693},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1693}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meta AI. 2025. Introducing LLaMA 4: Advancing multimodal intelligence. Accessed: April 18, 2026. find
  2. Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. Preprint, arXiv:2502.14425. — [arXiv:2502.14425](https://arxiv.org/abs/2502.14425) find
  3. Deepmind. 2025. Gemini Flash System Card. find
  4. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  5. Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. 2024. Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and v… find
  6. Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  7. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying Large Language Model Genera… find
  8. Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, and Mengdi Wang. 2025. Math-perturb: Benchmarking llms’ math reasoning abilities against hard pe… find
  9. Albérick Euraste Djiré, Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, and Tegawendé F. Bissyandé. 2025. Memorization or interpolation ? detecting llm memorization through input perturbation analysis. Preprint, arXiv:2505.03019. — [arXiv:2505.03019](https://arxiv.org/abs/2505.03019) find
  10. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live-CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representati… find
  11. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting,… find
  12. Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. 2024. LLM Internal States Reveal Hallucination Risk Faced With a Query. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 88–104, Miami, Florida…
  13. Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. 2018. Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS… find
  14. Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana. 2025. ALPACA AGAINST VICUNA: Using LLMs to Uncover Memorization of LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Comp… find
  15. Shahriar Golchin and Mihai Surdeanu. 2024. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Review.net.
  16. Shahriar Golchin and Mihai Surdeanu. 2025. Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. Trans. Assoc. Comput. Linguistics, 13:809–830. find
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Pang Wei Koh and Pe… — Understanding Black-box Predictions via Influence Functions
  18. Seongmin Lee, Hsiang Hsu, and Chun-Fu Chen. 2024. LLM Hallucination Reasoning with Zero-shot Knowledge Test. In Workshop on Socially Responsible Language Modelling Research. find
  19. Changmao Li and Jeffrey Flanigan. 2024. Task Contamination: Language Models May Not Be Few-Shot Anymore. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Education… find
  20. Renzhao Liang, Jingru Chen, Bo Jia, Bo Deng, Chenggang Xie, Yidong Wang, Ke Jin, Xin Wang, Linfeng Zhang, and Cunxiang Wang. 2026. Dvd: A robust method for detecting variant contamination in large language model evaluation. Preprint, arXiv:2601.04895. — [arXiv:2601.04895](https://arxiv.org/abs/2601.04895) find
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  22. Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner, Xuan-qiang Angelo Huang, Vilém Zouhar, Dominik Glandorf, Isabel Dahlgren, Rishit Dagli, Yuen Chen, Felix Leeb, Van Q. Truong, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Mrinmaya Sachan, Bernhard… — PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers in Overleaf
  23. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 24 others. 2025. 2 OL… find
  24. OpenAI. 2025a. Introducing GPT-4.1 in the API. Accessed: 2026-04-18. find
  25. OpenAI. 2025b. OpenAI o3 and o4-mini System Card. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  27. Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2025. A comprehensive survey of contamination detection methods in large language models. Preprint, arXiv:2404.00699. find
  28. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2024. To the cutoff... and beyond? a longitudinal perspective on LLM data contamination. In The Twelfth International Conference on Learning Representations. find
  29. Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. 2025. Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models. Preprint, arXiv:2411.12580. find
  30. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. Fresh-LLMs: Refreshing Large Language Models with Search Engine Augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 136… find
  31. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha V. Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBe… find
  32. Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. 2025. AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge. In Proceedings of the 63rd Annual Meeting of…
  33. Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024. Benchmark data contamination of large language models: A survey. Preprint, arXiv:2406.04244.
  34. Xinnuo Xu, Rachel Lawrence, Kshitij Dubey, Atharva Pandey, Risa Ueno, Fabian Falck, Aditya V. Nori, Rahul Sharma, Amit Sharma, and Javier Gonzalez. 2025. RE-IMAGINE: Symbolic benchmark synthesis for reasoning evaluation. In Forty-second International Conference on Machine Learning. find
  35. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph Gonzalez, and Ion Stoica. 2023. Rethinking benchmark and contamination for language models with rephrased samples.
  36. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024. A Careful Examination of Large Language Model Performance on Grade School Arithmetic. In… find
  37. Jie Zhang, Cezara Petrui, Kristina Nikolić, and Florian Tramèr. 2025. Realmath: A continuous benchmark for evaluating language models on research-level mathematics. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-test-time-rethinking-temporal-signal/