Elamur Research⌕
☰
home / papers / 2022 / Entity Cloze By Date: What LMs Know About Unseen Entities

Entity Cloze By Date: What LMs Know About Unseen Entities

Paper · 2022 · Yasumasa Onoe, Michael Zhang, Eunsol Choi et al.

Summary

Pretrained language models stay frozen for years, while new entities such as products, events and organizations keep appearing. This paper asks what LMs can infer about entities that came into existence after pretraining. The authors build Entity Cloze By Date (ECBD), an automatically collected benchmark. Entities are taken from Wikidata and tagged with their earliest recorded date (inception, start time, discovery and similar). Cloze sentences come from the entities' English Wikipedia articles (September 2021 dump). The masked spans sit shortly after an entity mention, so filling them should depend on knowing that entity. T5-large, BART-large and GPT-Neo are scored zero-shot by perplexity on the masked spans. Inputs are tested in four forms: unchanged, with the mention swapped for "the entity," with a random entity's definition prepended, and with the entity's own definition prepended. All three models do worse on entities from 2020–2021 than on older or popular ones. Supplying the definition helps most for these new entities. Because the pipeline can be rerun on fresh dumps, ECBD can serve as a time-indexed testbed for knowledge editing and continual learning.

Key points

  • Dataset size: ECBD has about 35k masked sentences covering roughly 2.1K Origination Date Indexed Entities (ODIE) from 2017–2021, plus 8k sentences about POPULAR entities, for about 43k in total. Data is split into dev and test sets by entity, and people are excluded.
  • Span types: spans are either noun phrases extracted with spaCy or random 1–5 word sequences. They start no more than ten words after the mention. The span vocabulary is about 19K, compared with about 2K in LAMA TREx.
  • Sanity check: on POPULAR entities, replacing the mention with "the entity" raises median span perplexity by 32.2% for T5. Without the distance criterion, the increase is only 25.9%.
  • Effect of definitions: prepending the definition lowers T5 perplexity from 19.43 to 13.60 on 2020–2021 entities, but only from 13.02 to 11.04 on POPULAR entities.
  • Availability: code and data are released at github.com/yasumasaonoe/ecbd.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Yasumasa Onoe, Michael Zhang, Eunsol Choi and Greg Durrett. 2022. Entity Cloze By Date: What LMs Know About Unseen Entities. Findings of the Association for Computational Linguistics: NAACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-naacl.52

@inproceedings{onoe2022entity,
  title = {{Entity Cloze By Date: What LMs Know About Unseen Entities}},
  author = {Yasumasa Onoe, Michael Zhang, Eunsol Choi and Greg Durrett},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.52},
  url = {https://doi.org/10.18653/v1/2022.findings-naacl.52}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Robert Bamler and Stephan Mandt. 2017. Dynamic Word Embeddings. In Proceedings of the International Conference on Machine Learning (ICML). find
  2. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  3. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  4. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. Time-Aware Language Models as Temporal Knowledge Bases.
  5. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  6. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  7. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models. abs/2204.14211. find
  8. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards Continual Knowledge Learning of Language Models. find
  9. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science.
  10. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the Gap: Assessing Temporal Generalization in Neural Langu… find
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Asso…
  12. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262. find
  13. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast Model Editing at Scale. abs/2110.11309. find
  14. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  15. Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. find
  16. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
  17. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. ArXiv, abs/1606.06031.
  18. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. In International Conference on Learning Representations (ICLR).
  19. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of th…
  20. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67.
  22. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable Neural Networks. In International Conference on Learning Representations (ICLR). find
  23. Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In International Workshop on Detecting and Exploiting Cultural Diversity on the Social Web, DETECT. find
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  25. Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  26. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/onoe-2022-entity-cloze-date-what-lms/