Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset

Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset

Paper · 2026 · Vittorio Mazzia, Sandro Pollastrini, Davide Bernardi et al.

Summary

LLMs used as assistants must handle dates, times and schedules correctly. Their practical temporal skills have seen little systematic testing, especially outside English. The authors, from Amazon, built the Temporal Reasoning Dataset (TRD), a synthetic benchmark in ten languages from three families. Templates written with native speakers are filled with randomized dates, times and offsets. This means new test items can be generated on demand, which limits contamination and allows fine control over difficulty.

TRD has five tasks, each at five difficulty levels. Three count as reasoning: date/time arithmetic, duration and recurrence. Two count as memorization: interval boundaries and day-of-week lookup. The experiments vary three things: task complexity, how far into the future the dates lie, and whether distracting sentences are added to the prompt.

Reasoning accuracy drops roughly linearly as difficulty rises but stays steady for distant years. Memorization tasks show the opposite pattern: flat across difficulty, but weaker on far-future dates. Distractors lower scores on average, and their effect does not follow human-like patterns. The benchmark helps diagnose how conversational agents fail at time handling.

Key points

  • Languages: ar-SA, en-US, de-DE, nl-NL, es-ES, fr-FR, it-IT, pt-BR, hi-IN and ja-JP. They cover the Afro-Asiatic, Indo-European and Japonic families.
  • Script effect: Models score better on Indo-European, Latin-script languages than on non-Latin scripts, particularly in the memorization tasks.
  • Size and availability: Each model was evaluated on 104,000 generated samples. Data and the code for generation and evaluation are released via the amazon-science GitHub.
  • Future-date test: Dates ranged from 2025 to 2095. Memorization accuracy falls off noticeably around 2055, which the authors link to sparse coverage of such dates in training data. Reasoning accuracy stays stable across the whole span.
  • Distractor sentences: Day-of-week was the most sensitive task, losing up to 10–22 percentage points. Stronger models such as Claude 3.5 Sonnet v2 coped better with distractors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Vittorio Mazzia, Sandro Pollastrini, Davide Bernardi, Chiara Rubagotti and Daniele Amberti. 2026. Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mazzia2026benchmarking,
  title = {{Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset}},
  author = {Vittorio Mazzia, Sandro Pollastrini, Davide Bernardi, Chiara Rubagotti and Daniele Amberti},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.19/}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The multilingual aspect of this study highlights Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli-how linguistic diversity shapes model performance. cent Ochieng, Krithika Ramesh, Prachi Jain, Ak-Models perform more consistently on languages shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed 175 find
  2. Axmed, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528. language models as tax attorneys: a case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A, 382(2270):20230159.
  3. Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305.14857.
  4. Ching Chang, Yidan Shi, Defu Cao, Wei Yang, Jeehyun Hwang, Haixin Wang, Jiacheng Pang, Wei Wang, Yan Liu, Wen-Chih Peng, and Tien-Fu Chen. 2025. A survey of reasoning and agentic systems in time series with large language models. arXiv preprint arXiv:2509.11575. — [arXiv:2509.11575](https://arxiv.org/abs/2509.11575) find
  5. Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. arXiv preprint arXiv:2108.06314. find
  6. Jeremy R Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar. 2023. Salient span masking for temporal understanding. arXiv preprint arXiv:2303.12860. find
  7. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257– 273.
  8. Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Tucker Balch, Manuela Veloso, and Svitlana Vyetrenko. 2024. Evaluating large language models on time series feature understanding: A comprehensive taxonomy and benchmark. arXiv preprint arXiv:2404.16563. Accepted to EMNLP 2024. find
  9. Taishi Hosokawa, Adam Jatowt, and Kazunari Sugiyama. 2023. Temporal natural language inference: Evidence-based evaluation of temporal text validity. In European Conference on Information Retrieval, pages 441–458. Springer. find
  10. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. arXiv preprint arXiv:2204.14211. find
  11. Mayuko Kimura, Lis Kanashiro Pereira, and Ichiro Kobayashi. 2021. Towards a language model for temporal commonsense reasoning. In Proceedings of the Student Research Workshop Associated with RANLP 2021, pages 78–84. find
  12. Zijia Liu, Peixuan Han, Haofei Yu, Haoru Li, and Jiaxuan You. 2025. Time-r1: Towards comprehensive temporal reasoning in llms. arXiv preprint arXiv:2505.13508. find
  13. Willa Potosnak, Cristian Challu, Mononito Goswami, Michal Wilinski, Nina Zukowska, and Artur Dubrawski. 2024. Implicit reasoning in deep time series forecasting. arXiv preprint arXiv:2409.10840. — [arXiv:2409.10840](https://arxiv.org/abs/2409.10840) find
  14. NeurIPS 2024 Workshop: Time Series in the Age of Large Models. find
  15. Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. Timedial: Temporal commonsense reasoning in dialog. arXiv preprint arXiv:2106.04571. find
  16. Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. arXiv preprint arXiv:2106.01515. find
  17. Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. Probing language models for understanding of temporal expressions. arXiv preprint arXiv:2110.01113. find
  18. Anushka Tonapi. 2023. Zeller’s congruence. At Right Angles, pages 68–71. find
  19. Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023. Revisiting relation extraction in the era of large language models. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2023, page 15566. NIH Public Access. find
  20. Jiexin Wang, Adam Jatowt, Masatoshi Yoshikawa, and Yi Cai. 2023. Bitimebert: Extending pre-trained language representations with bi-temporal information. In Proceedings of the 46th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 812–821. find
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  22. Georg Wenzel and Adam Jatowt. 2023. An overview of temporal commonsense reasoning and acquisition. arXiv preprint arXiv:2308.00002. find
  23. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  24. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. " going on a vacation" takes longer than" going for a walk": A study of temporal commonsense understanding. arXiv preprint arXiv:1909.03065. find
  25. John J Nay, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, and Jungo Kasai. 2024. Large find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mazzia-2026-benchmarking-multilingual-temporal-reasoning-llms/