Elamur Research⌕
☰
home / papers / 2026 / ODL-TempLLM: Ontology-Guided and Description Logic-Reasoned Temporal Reasoning with LLMs

ODL-TempLLM: Ontology-Guided and Description Logic-Reasoned Temporal Reasoning with LLMs

Paper · 2026 · Jinshuo Liu, Cheng Bi, Meng Wang et al.

Summary

LLMs often fail at temporal question answering because they must work out event orderings implicitly from raw context. They miss relations and are confused by irrelevant ones. ODL-TempLLM is a neuro-symbolic pipeline that moves this work outside the model. It has three stages:

  1. Ontology learning. An LLM uses function calling to build an event–time ontology. The ontology extends OWL-Time with Event classes.
  2. Symbolic reasoning. The Pellet reasoner applies SWRL rules based on Allen's interval algebra to infer implicit temporal relations deductively.
  3. Filtered retrieval. Logic-constrained retrieval uses SPARQL to keep only relations relevant to the question before the LLM answers.

The authors evaluated the method on CoTempQA, MentaQA and TOT with several LLM backbones. It beats state-of-the-art baselines by 2.07–31.83 F1 and 1.00–30.73 EM. The work shows that explicit, traceable temporal structure can make LLM reasoning more accurate and easier to interpret. The gains are especially large on long inputs, where plain LLMs overlook key facts.

Key points

  • Ontology construction: the event–temporal ontology uses four object properties (eventTime, hasTime, hasBeginning, hasEnd) and one data property (inXSDDateTimeStamp). The LLM builds it through three API functions: add_individual, add_data_pro and add_obj_pro.
  • Relation extraction accuracy: on TOT, whose inputs average about 10,000 characters, ODL-TempLLM's temporal relation extraction exceeds 80% accuracy, while competing methods fall below 40%. This was measured with GPT-3.5-Turbo on 100 expert-annotated samples per dataset.
  • Ablation: with GPT-3.5-Turbo, disabling both the reasoner and LCRAG causes the largest drops. F1/EM falls by 24.57/28.24 on CoTempQA and 36.12/31.62 on TOT.
  • Filtering: without filtering, the reasoner produces on average about 373 relations per example (CoTempQA), 87 (MentaQA) and 22K (TOT). The DRC filtering strategy keeps under 2% of these yet scores highest, for example 84.17 F1 on CoTempQA versus 69.88 with no filtering.
  • Error analysis: most errors are model prediction errors made despite correct inputs (41%). The rest are event boundary ambiguity (26%), vague temporal expressions (18%) and property assignment errors (15%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓Wuhan UniversityWuhan · China35 works · 1,127 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Jinshuo Liu, Cheng Bi, Meng Wang, Juan Deng, Donghong Ji and Jeff Z. Pan. 2026. ODL-TempLLM: Ontology-Guided and Description Logic-Reasoned Temporal Reasoning with LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.873

@inproceedings{liu2026odl,
  title = {{ODL-TempLLM: Ontology-Guided and Description Logic-Reasoned Temporal Reasoning with LLMs}},
  author = {Jinshuo Liu, Cheng Bi, Meng Wang, Juan Deng, Donghong Ji and Jeff Z. Pan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.873},
  url = {https://doi.org/10.18653/v1/2026.acl-long.873}
}

References

39 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. James F Allen and Patrick J Hayes. 1989. Moments and points in an interval-based temporal logic. Computational Intelligence, 5(3):225–238. find
  3. Adrián Bazaga, Rexhina Blloshmi, Bill Byrne, and Adrià de Gispert. 2025. Learning to reason over time: Timeline self-reflection for improved temporal reasoning in language models. arXiv preprint arXiv:2504.05258. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Ziyang Chen, Jinzhi Liao, and Xiang Zhao. 2023. Multi-granularity temporal question answering over knowledge graphs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11378–11392. find
  6. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  8. Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. Test of time: A benchmark for evaluating llms on temporal reasoning. arXiv preprint arXiv:2406.09170. find
  9. Michael Grüninger and Zhuojun Li. 2017. The time ontology of allen’s interval algebra. In 24th International Symposium on Temporal Representation and Reasoning (TIME 2017), pages 16–1. Schloss Dagstuhl–Leibniz-Zentrum für Informatik. find
  10. Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint event and temporal relation extraction with shared representations and structured prediction. arXiv preprint arXiv:1909.05360. find
  11. Steven Harris and Andy Seaborne. 2013. SPARQL 1.1 query language. W3C recommendation, W3C. https://www.w3.org/TR/2013/ REC-sparql11-query-20130321/. ↗
  12. Ian Horrocks, Peter F Patel-Schneider, Harold Boley, Said Tabet, Benjamin Grosof, Mike Dean, and 1 others. 2004. Swrl: A semantic web rule language combining owl and ruleml. W3C Member submission, 21(79):1–31. find
  13. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 874–880.
  14. Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023. Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in… find
  15. Mayuko Kimura, Lis Kanashiro Pereira, and Ichiro Kobayashi. 2021. Towards a language model for temporal commonsense reasoning. In Proceedings of the Student Research Workshop Associated with RANLP 2021, pages 78–84. find
  16. Mayuko Kimura, Lis Kanashiro Pereira, and Ichiro Kobayashi. 2022. Toward building a language model for understanding temporal commonsense. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on… find
  17. Feng Pan and Jerry R Hobbs. 2006. Time ontology in owl. W3C working draft, W3C, 1(1):1. find
  18. Guy D Rosin, Ido Guy, and Kira Radinsky. 2022. Time masking for temporal language models. In Proceedings of the fifteenth ACM international conference on Web search and data mining, pages 833–841. find
  19. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  20. Evren Sirin, Bijan Parsia, Bernardo Cuenca Grau, Aditya Kalyanpur, and Yarden Katz. 2007. Pellet: A practical owl-dl reasoner. Journal of Web Semantics, 5(2):51–53. find
  21. Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, and Min Zhang. 2024a. Living in the moment: Can large language models grasp co-temporal reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)… find
  22. Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, and Yu Cheng. 2024b. Timo: Towards better temporal reasoning for language models. arXiv preprint arXiv:2406.14192.
  23. Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820–14835. find
  24. Weizhi Tang and Vaishak Belle. 2024. Ltlbench: Towards benchmarks for evaluating temporal logic reasoning in large language models. arXiv preprint arXiv:2407.05434. — [arXiv:2407.05434](https://arxiv.org/abs/2407.05434) find
  25. Chen Lin, Dmitriy Dligach, Timothy A Miller, Steven Bethard, and Guergana K Savova. 2016. Multilayered temporal modeling for the clinical domain. Journal of the American Medical Informatics Association, 23(2):387–395. find
  26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  27. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  28. Jonas Wallat, Abdelrahman Abdallah, Adam Jatowt, and Avishek Anand. 2025. A study into investigating temporal robustness of llms. arXiv preprint arXiv:2503.17073. find
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  30. Jiexin Wang, Adam Jatowt, Masatoshi Yoshikawa, and Yi Cai. 2023. Bitimebert: Extending pre-trained language representations with bi-temporal information. In Proceedings of the 46th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 812–821. find
  31. Yuqing Wang and Yun Zhao. 2024. Tram: Benchmarking temporal reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6389–6415. find
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  33. Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. Menatqa: A new dataset for testing the temporal comprehension and reasoning abilities of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1434… find
  34. Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853. find
  35. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  36. Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. Zero-shot temporal relation extraction with chatgpt. arXiv preprint arXiv:2304.05454. find
  37. Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Back to the future: Towards explainable temporal reasoning with large language models. In Proceedings of the ACM Web Conference 2024, pages 1963–1974. find
  38. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297.
  39. Xinliang Frederick Zhang, Nicholas Beauchamp, and Lu Wang. 2024. Narrative-of-thought: Improving temporal reasoning of large language models via recounted narratives. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16507–16530. *Figure 7: Allen’s Interval Algebra: Tem… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-odl-templlm-ontology-guided-description/