Elamur Research⌕
☰
home / papers / 2022 / Time-Aware Language Models as Temporal Knowledge Bases

Time-Aware Language Models as Temporal Knowledge Bases

Summary

Language models are often viewed as repositories of facts. However, many facts become outdated over time: someone changes club, position, or employer. Models, meanwhile, are trained on a snapshot of data collected at a particular moment. The authors from Google Research investigate how this affects the knowledge of models. They identify three problems: conflicting facts from different years get "averaged", knowledge about less represented years is forgotten, and predictions about the future are poorly calibrated. For diagnosis, the authors created the TempLAMA dataset: cloze queries built from Wikidata whose correct answer depends on the year. As a solution, they propose a simple modification of T5 pretraining: a year prefix is added to each example, and documents are sampled uniformly from the entire time span of the CustomNews news corpus. Three models are compared: Uniform (no time), Yearly (a separate model for each year), and Temporal. The temporal model better memorizes facts from the training period and predicts the future more cautiously. In addition, it can be cheaply updated with new data without full retraining. This matters for any system that relies on up-to-date facts.

Key points

  • TempLAMA: 50,310 queries over 11 years based on a Wikidata snapshot from November 2020. It covers 9 relations (e.g., "plays for", "works for", "member of political party") and the 1000 most frequent subjects for each relation. The dataset is published on GitHub (google-research/language).
  • CustomNews: 1 million news articles per year for 2010–2020. The models are trained with salient span masking (SSM): named entities recognized by a tagger based on BERT trained on CoNLL-2003, as well as dates.
  • Method: the model learns the distribution P(y | x, t), i.e., it takes into account not only the text but also the time. The year is added to the T5 input as a text prefix such as "year: 2018". All models are further trained from public T5 checkpoints for 300K steps.
  • Updating: the temporal model can be further trained on new data only. It reaches the level of retraining from scratch in roughly 30 times fewer steps and does not lose old knowledge.
  • Calibration: a separate set of queries is divided into facts that change frequently, rarely, or never. It shows that model uncertainty grows fastest for facts that change frequently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Language models & pretraining Knowledge graphs & ontologies

Cite

Cite as Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein and William W. Cohen. 2022. Time-Aware Language Models as Temporal Knowledge Bases.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00459 · arXiv 2106.15110

@misc{dhingra2022time,
  title = {{Time-Aware Language Models as Temporal Knowledge Bases}},
  author = {Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein and William W. Cohen},
  year = {2022},
  doi = {10.1162/tacl\_a\_00459},
  eprint = {2106.15110},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00459}
}

References

49 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Applied Sciences, 10(12). https://doi.org/10.3390/app10124180 — [doi:10.3390/app10124180](https://doi.org/10.3390/app10124180) ↗
  2. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977. our card
  3. Lea Frermann and Mirella Lapata. 2016. A Bayesian model of diachronic meaning change. Transactions of the Association for Computational Linguistics, 4:31–45. https://doi.org/10.1162/tacl_a_00081 ↗
  4. Robert Bamler and Stephan Mandt. 2017. Dynamic word embeddings. In International Conference on Machine Learning, pages 380–389. PMLR. find
  5. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https://do… find
  6. Edouard Delasalles, Sylvain Lamprier, and Ludovic Denoyer. 2019. Learning dynamic author representations with temporal language models. In 2019 IEEE International Conference on Data Mining (ICDM), pages 120–129. https://doi.org/10.1109/ICDM.2019 .00022 — [doi:10.1109/icdm.2019](https://doi.org/10.1109/icdm.2019) ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  8. Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960–3973, Online. Association for Computational Linguistic… our card
  9. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR. find
  10. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. Association for Com… our card
  11. Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Mike Bendersky, and Marc Najork. 2021. Dynamic language models for continuously evolving content. In Knowledge Discovery and Data Mining (KDD). https://doi.org/10.1145/3447548.3467162 — [doi:10.1145/3447548.3467162](https://doi.org/10.1145/3447548.3467162) ↗
  12. Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for… find
  13. Xiaolei Huang and Michael J. Paul. 2018. Examining temporality in document classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694–699, Melbourne, Australia. Association for Computational Linguistics. https://doi.… ↗
  14. Xiaolei Huang and Michael J. Paul. 2019. Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models. In Proceedings of Komal Florio, Valerio Basile, Marco Polignano, Pierpaolo Basile, and Viviana Patti. 2020. Time of your hate: The challenge of… find
  15. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021 .eacl-main.284 the 57th Annual Meeting of the Association for Computational Linguistics, pages 4113–4123, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1403 ↗
  16. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. 2018. Dynamic evaluation of neural sequence models. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2766–2775. PMLR. find
  17. Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. CoRR, abs/2106 .13353. our card
  18. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… our card
  19. Kokil Jaidka, Niyati Chhaya, and Lyle Ungar. 2018. Diachronic degradation of language models: Insights from social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 195–200, Melbourne, Australia. Association for Computat… find
  20. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. Tempquestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, pages 1057–1062. https://doi.org/10 .1145/3184558.3191536 find
  21. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Ass… find
  22. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Sy… find
  23. Konstantina Lazaridou, Alexander Löser, Maria Mestre, and Felix Naumann. 2020. Discovering biased news articles leveraging multiple human annotations. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1268–1277, Marseille, France. European Language Resources Association… find
  24. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. https://doi.org/10.1162/tacl_a_00324 ↗ our card
  25. Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 36–41, Online. Association for Computational Linguistics. find
  26. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. our card
  27. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Associ… find
  28. Jialu Liu, Tianqi Liu, and Cong Yu. 2021. Newsembed: Modeling news through pre-trained document representations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, pages 1076–1086, New York, NY, USA. Association for Computing Machinery. https://doi.org/10… — [doi:10.1145/3447548](https://doi.org/10.1145/3447548) find
  29. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. our card
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  32. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling… our card
  33. Jan Lukes and Anders Søgaard. 2018. Sentiment analysis under temporal shift. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 65–71, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W1… find
  34. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158–1172, Online. Association for Computatio… our card
  35. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  36. Michael Ringgaard, Rahul Gupta, and Fernando C. N. Pereira. 2017. SLING: A framework for frame semantic parsing. CoRR, abs/1710.07032. — [arXiv:1710.07032](https://arxiv.org/abs/1710.07032) find
  37. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. https://… find
  38. Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400–2412, Punta Cana, Dominican Republic. Association for Compu… find
  39. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. In Proceedin… our card
  40. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In International Conference on Learning Representations. find
  41. Commun. ACM, 57(10):78–85. https://doi.org/10.1145/2629489 — [doi:10.1145/2629489](https://doi.org/10.1145/2629489) ↗
  42. Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural DiversiTy on the Social Web, DETECT ’11, pages 35–40, New York, NY, USA. Association for Computing Machinery. h… — [doi:10.1145/2064448](https://doi.org/10.1145/2064448) find
  43. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. ‘‘going on a vacation’’ takes longer than ‘‘going for a walk’’: A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen… find
  44. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. https://doi.or… ↗ our card
  45. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. https://doi.org/10.3115/1119176.1119195 ↗ our card
  46. Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. our card
  48. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. CoRR, abs/2012.00363. find
  49. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dhingra-2022-time-aware-language-models-as/