Elamur Research⌕
☰
home / papers / 2025 / ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events

ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events

Paper · 2025 · Duygu Sezen Islakoglu and Jan-Christoph Kalo

Summary

Large language models still struggle with numerical and temporal reasoning, and no earlier benchmark covers all 13 interval relations of Allen's interval algebra. The authors address this gap with ChronoSense, a diagnostic benchmark of balanced True/False questions at year granularity. It contains 16 tasks. Thirteen ask whether a specific Allen relation (such as before, overlaps or finishes) holds between two real events taken from Wikidata, with their start and end years given. The other three test synthetic single-event arithmetic: end timepoint, next occurrence and intermediate timepoint. Every Allen question also has an anonymized "Event A/Event B" variant, which lets the authors check for memorization. Seven LLMs were evaluated with 0-shot, 1-shot, 3-shot and chain-of-thought prompting. Many models perform close to chance, and some often fail to reply with True/False at all. Only GPT-4o is consistently strong on Allen relations. CoT greatly improves the arithmetic tasks. Accuracy usually drops when event names are hidden, which suggests models lean on pretraining knowledge. These findings call into question how far LLMs can be trusted with timeline reasoning in historical, legal or medical settings.

Key points

  • ChronoSense includes 13 Allen-relation tasks and 3 temporal arithmetic tasks. Each task has 4,000 train, 500 validation and 500 test samples, with labels split 50/50 between True and False. The data is released under CC BY 4.0.
  • The evaluated models are Gemma2-9B-it, GPT-4o, GPT-4o-mini, Llama3.1-8B, Mistral-7B, Mixtral-8x7B and Phi-3-mini. Gemma2, Llama3.1 and Mistral frequently produced unclear (non-True/False) outputs.
  • GPT-4o averages 0.87 on Allen questions at 0-shot and 0.95 at 3-shot. With CoT, arithmetic accuracy reaches 0.92–0.99 for most models, compared with roughly 0.35–0.76 at 0-shot.
  • "Before" and "After" are generally the easiest relations. "Equals" (0-shot/CoT) and "Finishes" (few-shot/abstract) are the hardest, and paired symmetric relations such as Meets and Met-by yield unequal accuracy.
  • Common errors include mixing up start and end years, miscalculations, and reaching a wrong conclusion despite a correct explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Duygu Sezen Islakoglu and Jan-Christoph Kalo. 2025. ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-short.46

@inproceedings{islakoglu2025chronosense,
  title = {{ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events}},
  author = {Duygu Sezen Islakoglu and Jan-Christoph Kalo},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.46},
  url = {https://doi.org/10.18653/v1/2025.acl-short.46}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F. Allen. 1989. Maintaining Knowledge about Temporal Intervals, page 361–372. find
  2. BIG-bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
  3. Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. Preprint, arXiv:2108.06314. find
  4. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2023. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. ArXiv, abs/2311.17667. find
  5. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257– 273.
  6. Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. Preprint, arXiv:2406.09170. find
  7. Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023. Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in… find
  8. Tomi Janhunen and Michael Sioutis. 2019. Allen’s interval algebra makes the difference. Preprint, arXiv:1909.01128. — [arXiv:1909.01128](https://arxiv.org/abs/1909.01128) find
  9. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018a. Tempquestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1057–1062. find
  10. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b. Tequila: Temporal question answering over knowledge bases. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, page 1807–1810. find
  11. Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 792–802. find
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  14. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  15. Aakanksha Naik, Luke Breitfeller, and Carolyn Rose. 2019. TDDiscourse: A dataset for discourse-level temporal ordering of events. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 239–249. find
  16. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158–1172.
  17. Suresh Kumar Sanampudi and G Vijaya Kumari. 2010. Temporal reasoning in natural language processing: A survey. International Journal of Computer Applications, 1(4):68–72. find
  18. Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Paper… find
  19. Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023a. Towards benchmarking and improving the temporal reasoning capability of large language models. In Annual Meeting of the Association for Computational Linguistics. find
  20. Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023b. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820–14835. find
  21. Weizhi Tang and Vaishak Belle. 2024. Ltlbench: Towards benchmarks for evaluating temporal logic reasoning in large language models. ArXiv, abs/2407.05434. — [arXiv:2407.05434](https://arxiv.org/abs/2407.05434) find
  22. Paolo Terenziani. 2009. Qualitative Temporal Reasoning, pages 2225–2229. find
  23. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of… find
  24. Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal reasoning in natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4070–4078.
  25. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  26. Yuqing Wang and Yun Zhao. 2023. Tram: Benchmarking temporal reasoning for large language models. ArXiv, abs/2310.00835. find
  27. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  28. Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. Preprint, arXiv:2401.06853. find
  29. Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023. Logical reasoning over natural language as knowledge representation: A survey. Preprint, arXiv:2303.12023.
  30. Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2023. Back to the future: Towards explainable temporal reasoning with large language models. Preprint, arXiv:2310.01074. find
  31. Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang, and Hai Jin. 2022. Can language models serve as temporal knowledge bases? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2024–2037. find
  32. Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/islakoglu-2025-chronosense-exploring-temporal-understanding-large/