Elamur Research⌕
☰
home / papers / 2023 / An Overview Of Temporal Commonsense Reasoning and Acquisition

An Overview Of Temporal Commonsense Reasoning and Acquisition

Paper · 2023 · Georg Wenzel and Adam Jatowt

Summary

This survey covers temporal commonsense (TCS) reasoning: a model's ability to infer unstated temporal properties of events, such as their typical order, duration, or time of occurrence. For example, a broken leg usually comes before a hospital visit. The authors place TCS between commonsense and temporal reasoning. They exclude temporal factual knowledge and commonsense causality from scope. They gathered transformer-era literature from 2018 to 2023 and reviewed TCS benchmark datasets, evaluation metrics, and the ways researchers have tried to strengthen language models on these tasks.

Their main finding is that transformers beat earlier methods by a wide margin, yet they remain far below human performance when stricter metrics are used. Models are vulnerable to mispriming, reporting bias, small input changes, and overlooked negations. Much of their apparent success seems to come from pattern matching rather than real reasoning.

The survey calls for carefully built datasets and robust metrics so that results are not overstated. It also points to neuro-symbolic augmentation as a promising direction. Progress here would benefit downstream tasks including temporal NLI, timeline summarization, and temporal question answering.

Key points

  • Literature search: The authors searched Google Scholar, Semantic Scholar, and dblp for "temporal commonsense", limited to 2018–2023. They examined all 21 dblp hits and the top 50 results from each of the other two sources. They kept only English papers that mention TCS in the abstract.
  • Related models and resources: Time-aware models such as TempoBERT and BiTimeBERT temporalize embeddings using document creation time or explicit time expressions. Knowledge graphs used in this area include ConceptNet, ATOMIC 2020 (with the COMET model), and ASER.
  • Model improvements: Most work focuses on engineering the inputs and outputs around the transformer, not changing its architecture. Approaches include:
    • adding TCS training data from crowdsourcing, weakly supervised web extraction, or knowledge graphs;
    • ensembling several models;
    • adding symbolic logic, for example probabilistic soft logic.
  • Explicit vs. implicit knowledge: Injecting explicit symbolic knowledge about duration, frequency, and distance often appears to work better than teaching it implicitly through auxiliary training objectives. Pre-training choices also matter, as the accuracy gap between BERT and RoBERTa shows.
  • GPT-4: Early results for GPT-4 are strong, but its TCS ability may still be limited. Its size and closed weights make local use impractical, so research on smaller, portable models remains relevant.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität InnsbruckInnsbruck · Austria23 works · 1,475 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Georg Wenzel and Adam Jatowt. 2023. An Overview Of Temporal Commonsense Reasoning and Acquisition. arXiv:2308.00002.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.00002 · arXiv 2308.00002

@misc{wenzel2023overview,
  title = {{An Overview Of Temporal Commonsense Reasoning and Acquisition}},
  author = {Georg Wenzel and Adam Jatowt},
  year = {2023},
  doi = {10.48550/arxiv.2308.00002},
  eprint = {2308.00002},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.00002}
}

References

90 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abend O, Rappoport A (2013) Universal conceptual cognitive annotation (ucca). Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
  2. Agrawal H, Chandrasekaran A, Batra D, et al (2016) Sort story: Sorting jumbled images and captions into stories. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 925–931 find
  3. Allen JF (1983) Maintaining knowledge about temporal intervals. Communications of the ACM 26(11):832–843. https://doi.org/10.1016/b978-1-4832-1447-4.50033-x ↗
  4. Almquist A, Jatowt A (2019) Towards content expiry date determination: predicting validity periods of sentences. Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part I 41, 86–101 find
  5. Banarescu L, Bonial C, Cai S, et al (2013) Abstract meaning representation for sembanking. Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, 178–186
  6. Bethard S, Savova G, Chen WT, et al (2016) Semeval-2016 task 12: Clinical tempeval. Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), 1052–1062
  7. Bhargava P, Ng V (2022) Commonsense knowledge reasoning and generation with pre-trained language models: A survey. Proceedings of the AAAI Conference on Artificial Intelligence, 12317–12325 find
  8. Bian N, Han X, Sun L, et al (2023) Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. Preprint at https://arxiv.org/abs/2303.16421 ↗
  9. Brown T, Mann B, Ryder N, et al (2020) Language models are few-shot learners. Advances in neural information processing systems 33:1877–1901
  10. Bulian J, Buck C, Gajewski W, et al (2022) Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 291–305
  11. Cai B, Ding X, Chen B, et al (2022) Mitigating reporting bias in semi-supervised temporal commonsense inference with probabilistic soft logic. Proceedings of the AAAI Conference on Artificial Intelligence, 10454–10462 find
  12. Cai B, Ding X, Sun Z, et al (2023) Self-supervised logic induction for explainable fuzzy temporal commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 12580–12588 find
  13. Campos R, Dias G, Jorge AM, et al (2014) Survey of temporal information retrieval and related applications. ACM Computing Surveys (CSUR) 47(2):1–41. https://doi.org/10.1145/2619088 — [doi:10.1145/2619088](https://doi.org/10.1145/2619088) ↗
  14. Cao S, Wang L (2022) Time-aware prompting for text generation. Findings of the Association for Computational Linguistics: EMNLP 2022, 7231–7246 find
  15. Chen W, Wang X, Wang WY (2021) A dataset for answering time-sensitive questions. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) find
  16. Davis E (2023) Benchmarks for automated commonsense reasoning: A survey. Preprint at https://arxiv.org/abs/2302.04752 ↗
  17. DeJong G (1982) An overview of the frump system. Strategies for natural language processing 113:149–176 find
  18. Dhingra B, Cole JR, Eisenschlos JM, et al (2022) Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics 10:257–273. https://doi.org/10.1162/tacl a 00459 ↗
  19. Ferro L, Mani I, Sundheim B, et al (2001) Tides temporal annotation guidelines version 1.0. 2. find
  20. The MITRE Corporation, McLean-VG-USA Fries J (2016) Recurrent neural networks vs. joint inference for clinical temporal information extraction. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 1274–9 find
  21. Gardner M, Artzi Y, Basmov V, et al (2020) Evaluating models’ local decision boundaries via contrast sets. Findings of the Association for Computational Linguistics: EMNLP 2020, 1307–1323
  22. Ghosal D, Hong P, Shen S, et al (2021a) Cider: Commonsense inference for dialogue explanation and reasoning. Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, 301–313 find
  23. Ghosal D, Majumder N, Mihalcea R, et al (2021b) Stack: Sentence ordering with temporal commonsense knowledge. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8676–8686 find
  24. Gordon J, Van Durme B (2013) Reporting bias and knowledge extraction. Preprint at https://openreview.net/pdf?id=AzxEzvpdE3Wcy ↗
  25. Grover C, Tobin R, Alex B, et al (2010) Edinburgh-ltg: Tempeval-2 system description. Proceedings of the 5th International Workshop on Semantic Evaluation, 333–336 find
  26. Guan J, Huang F, Zhao Z, et al (2020) A knowledge-enhanced pretraining model for commonsense story generation. Transactions of the Association for Computational Linguistics 8:93–108. https://doi.org/10.1162/tacl a 00302 ↗
  27. Han R, Liang M, Alhafni B, et al (2019) Contextualized word embeddings enhanced event temporal relation extraction for story understanding. ArXiv preprint arXiv:1904.11942 — [arXiv:1904.11942](https://arxiv.org/abs/1904.11942) find
  28. Helwe C, Clavel C, Suchanek FM (2021) Reasoning with transformer-based models: Deep learning, but shallow reasoning. 3rd conference on automated knowledge base construction Hosokawa T, Jatowt A, Sugiyama K (2023) Temporal natural language inference: Evidence-based evaluation of temporal text validit… find
  29. Huang L, Le Bras R, Bhagavatula C, et al (2019) Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN…
  30. Hwang JD, Bhagavatula C, Le Bras R, et al (2021) (comet-) atomic 2020: on symbolic and neural commonsense knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 6384–6392
  31. Jain R, Sojitra D, Acharya A, et al (2023) Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)
  32. Ji S, Pan S, Cambria E, et al (2021) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems 33(2):494–514. https://doi.org/10.1109/tnnls.2021.3070843 ↗
  33. Kenton JDMWC, Toutanova LK (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, 4171–4186
  34. Kimura M, Pereira LK, Kobayashi I (2021) Towards a language model for temporal commonsense reasoning. Proceedings of the Student Research Workshop Associated with RANLP 2021, 78–84 find
  35. Levesque H, Davis E, Morgenstern L (2012) The winograd schema challenge. Thirteenth international conference on the principles of knowledge representation and reasoning Liu Y, Ott M, Goyal N, et al (2019) Roberta: A robustly optimized bert pretraining approach. Preprint at https://arxiv.org/abs/1907… ↗
  36. Llorens H, Saquete E, Navarro B (2010) Tipsem (english and spanish): Evaluating crfs and semantic roles in tempeval-2. Proceedings of the 5th International Workshop on Semantic Evaluation, 284–291 find
  37. Lymperaiou M, Stamou G (2022) A survey on knowledge-enhanced multimodal learning. Preprint at https://arxiv.org/abs/2211.12328 — [arXiv:2211.12328](https://arxiv.org/abs/2211.12328) ↗
  38. Lynden S, Heilemariam M, Kim KS, et al (2023) Commonsense temporal action knowledge (cotak) dataset. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023) find
  39. Mikolov T, Chen K, Corrado G, et al (2013) Efficient estimation of word representations in vector space. Preprint at https://arxiv.org/abs/1301.3781 ↗
  40. Mostafazadeh N, Chambers N, He X, et al (2016a) A corpus and cloze evaluation for deeper understanding of commonsense stories. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 839–849 find
  41. Mostafazadeh N, Grealish A, Chambers N, et al (2016b) Caters: Causal and temporal relation scheme for semantic annotation of event structures. Proceedings of the Fourth Workshop on Events, 51–61 find
  42. Mostafazadeh N, Kalyanpur A, Moon L, et al (2020) Glucose: Generalized and contextualized story explanations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4569–4586
  43. Ning Q, Wu H, Peng H, et al (2018a) Improving temporal relation extraction with a globally acquired statistical resource. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 841–851 find
  44. Ning Q, Wu H, Roth D (2018b) A multi-axis annotation scheme for event temporal relations. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1318–1328 find
  45. Ning Q, Wu H, Dasigi P, et al (2020a) Easy, reproducible and quality-controlled data collection with crowdaq. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 127–134 find
  46. Ning Q, Wu H, Han R, et al (2020b) Torque: A reading comprehension dataset of temporal ordering questions. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1158–1172
  47. O’Gorman T, Wright-Bettner K, Palmer M (2016) Richer event description: Integrating event coreference with temporal, causal and bridging annotation. Proceedings of the 2nd workshop on computing news storylines (CNS 2016), 47–56
  48. Pan F, Mulkar-Mehta R, Hobbs JR (2006) Extending timeml with typical durations of events. Proceedings of the Workshop on Annotating and Reasoning about Time and Events, 38–45 find
  49. Pasquali A, Campos R, Ribeiro A, et al (2021) Tls-covid19: A new annotated corpus for timeline summarization. European Conference on Information Retrieval, 497–512 find
  50. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1532–1543
  51. Pereira L, Liu X, Cheng F, et al (2020) Adversarial training for commonsense inference. Proceedings of the 5th Workshop on Representation Learning for NLP (RepL4NLP-2020), 55–60 find
  52. Pereira L, Cheng F, Asahara M, et al (2021) Alice++: Adversarial training for robust and effective temporal reasoning. Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, 373–382 find
  53. Pustejovsky J (2003) The timebank corpus. Proceedings of Corpus Linguistics 2003, 647–656 find
  54. Pustejovsky J, Castano JM, Ingria R, et al (2003) Timeml: Robust specification of event and temporal expressions in text. New directions in question answering 3:28– 34 find
  55. Pustejovsky J, Littman J, Saurı́ R, et al (2006) Timebank 1.2 documentation. Event London, no. April, 6–11 find
  56. Qin C, Zhang A, Zhang Z, et al (2023) Is chatgpt a general-purpose natural language processing task solver? Preprint at https://arxiv.org/abs/2302.06476 ↗
  57. Qin L, Gupta A, Upadhyay S, et al (2021) Timedial: Temporal commonsense reasoning in dialog. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7066–7076 find
  58. Radford A, Narasimhan K, Salimans T, et al (2018) Improving language understanding by generative pre-training Radford A, Wu J, Child R, et al (2019) Language models are unsupervised multitask learners. OpenAI blog Raffel C, Shazeer N, Roberts A, et al (2020) Exploring the limits of transfer learning…
  59. Richardson M, Burges CJ, Renshaw E (2013) Mctest: A challenge dataset for the open-domain machine comprehension of text. Proceedings of the 2013 conference on empirical methods in natural language processing, 193–203
  60. Rosin G, Radinsky K (2022) Temporal attention for language models. Findings of the Association for Computational Linguistics: NAACL 2022, 1498–1508 find
  61. Rosin GD, Guy I, Radinsky K (2022) Time masking for temporal language models. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, 833–841 find
  62. Sakaguchi K, Bras RL, Bhagavatula C, et al (2021) Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM 64(9):99–106. https://doi.org/10.1145/3474381 ↗
  63. Sauréi R, Littman J, Knippen B, et al (2005) Timeml annotation guidelines. https://timeml.github.io/site/publications/timeMLdocs/annguide 1.2.pdf Shwartz V, Choi Y (2020) Do neural language models overcome reporting bias? Proceedings of the 28th International Conference on Computational Linguistic… ↗
  64. Singh S, Wen N, Hou Y, et al (2021) Com2sense: A commonsense reasoning benchmark with complementary sentences. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 883–898 find
  65. Sousa H, Jorge A, Campos R (2023) tieval: An evaluation framework for temporal information extraction systems. Preprint at https://arxivorg/abs/230104643 find
  66. Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI conference on artificial intelligence Storks S, Gao Q, Chai JY (2019) Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches.…
  67. Strötgen J, Gertz M (2010) Heideltime: High quality rule-based extraction and normalization of temporal expressions. Proceedings of the 5th international workshop on semantic evaluation, 321–324 find
  68. Sun Z, Fan C, Han Q, et al (2020) Self-explaining structures improve nlp models. Preprint at https://arxiv.org/abs/2012.01786 — [arXiv:2012.01786](https://arxiv.org/abs/2012.01786) ↗
  69. Tan Q, Ng HT, Bing L (2023) Towards benchmarking and improving the temporal reasoning capability of large language models Toosi A, Bottino AG, Saboury B, et al (2021) A brief history of ai: how to prevent another winter (a critical review). PET clinics 16(4):449–469. https://doi.org/10.1016/j.cpet.2… ↗
  70. UzZaman N, Llorens H, Derczynski L, et al (2013) Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (S… find
  71. Vashishtha S, Poliak A, Lal YK, et al (2020) Temporal reasoning in natural language inference. Findings of the Association for Computational Linguistics: EMNLP 2020, 4070–4078
  72. Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. Advances in neural information processing systems 30
  73. Verhagen M, Gaizauskas R, Schilder F, et al (2007) Semeval-2007 task 15: Tempeval temporal relation identification. Proceedings of the fourth international workshop on semantic evaluations (SemEval-2007), 75–80 find
  74. Verhagen M, Sauri R, Caselli T, et al (2010) Semeval-2010 task 13: Tempeval-2. Proceedings of the 5th international workshop on semantic evaluation, 57–62 find
  75. Vila L (1994) A survey on temporal reasoning in artificial intelligence. Ai Communications 7(1):4–28. https://doi.org/10.3233/aic-1994-7102 — [doi:10.3233/aic-1994-7102](https://doi.org/10.3233/aic-1994-7102) ↗
  76. Wang J, Jatowt A, Färber M, et al (2020) Answering event-related questions over long-term news article archives. Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part I 42, 774–789 find
  77. Wang J, Jatowt A, Yoshikawa M (2022) Bitimebert: Extending pre-trained language representations with bi-temporal information. Preprint at https://arxiv.org/abs/ 2204.13032 ↗
  78. Wang X, Jiang M (2020) Precise temporal slot filling via truth finding with data-driven commonsense. Knowledge and Information Systems 62(10):4113–4139. https://doi.org/10.1007/s10115-020-01493-w Wolff P, Song G (2003) Models of causation and the semantics of causal verbs. Cognitive psychology 47(3… — [doi:10.1007/s10115-020-01493-w](https://doi.org/10.1007/s10115-020-01493-w) ↗
  79. Yang Z, Du X, Rush AM, et al (2020) Improving event duration prediction via time-aware pre-training. Findings of the Association for Computational Linguistics: EMNLP 2020 find
  80. Yin D, Dong L, Cheng H, et al (2022) A survey of knowledge-intensive nlp with pre-trained language models. Preprint at https://arxiv.org/abs/2202.08772 — [arXiv:2202.08772](https://arxiv.org/abs/2202.08772) ↗
  81. Yu C, Zhang H, Song Y, et al (2022) Cocolm: Complex commonsense enhanced language model with discourse relations. Findings of the Association for Computational Linguistics: ACL 2022, 1175–1187 find
  82. Zhang H, Liu X, Pan H, et al (2020a) Aser: A large-scale eventuality knowledge graph. Proceedings of the web conference 2020, 201–211 find
  83. Zhang J, Zhang H, Su W, et al (2022) Rock: Causal inference principles for reasoning about commonsense causality. International Conference on Machine Learning, 26750–26771 find
  84. Zhang L, Lyu Q, Callison-Burch C (2020b) Reasoning about goals, steps, and temporal ordering with wikihow. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4630–4639 find
  85. Zhang Z, Zhou Z, Tang H, et al (2021) Enriching ontology with temporal commonsense for low-resource audio tagging. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 3652–3656 find
  86. Zhou B, Khashabi D, Ning Q, et al (2019) “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… find
  87. Zhou B, Ning Q, Khashabi D, et al (2020a) Temporal common sense acquisition with minimal supervision. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7579–7589 find
  88. Zhou B, Richardson K, Ning Q, et al (2021) Temporal reasoning on implicit events from distant supervision. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1361–1371 find
  89. Zhou B, Chen Y, Liu K, et al (2022) Generating temporally-ordered event sequences via event optimal transport. Proceedings of the 29th International Conference on Computational Linguistics, 1875–1884 find
  90. Zhou Y, Zhao T, Jiang M (2020b) A probabilistic model with commonsense constraints for pattern-based temporal fact extraction. Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), 18–25 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wenzel-2023-overview-temporal-commonsense-reasoning-acquisition/