Language Models as Knowledge Bases?

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

1211 papers of our library cite it (of the 140673 whose bibliographies we read); 85 of them have a page here, 1126 not yet.

• Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
• PIQA: Reasoning about Physical Commonsense in Natural Language
• On the Opportunities and Risks of Foundation Models
• Extracting Training Data from Large Language Models
• RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
• Chain-of-Verification Reduces Hallucination in Large Language Models
• English Machine Reading Comprehension Datasets: A Survey
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• Multilingual Sequence-to-Sequence Models for Hebrew NLP
• Making Pre-trained Language Models Better Few-shot Learners
• RARR: Researching and Revising What Language Models Say, Using Language Models
• Injecting Numerical Reasoning Skills into Language Models
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
• Evaluating Large Language Models: A Comprehensive Survey
• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
• ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
• Measuring Massive Multitask Language Understanding
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Survey of Hallucination in Natural Language Generation
• How Can We Know What Language Models Know?
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• Deduplicating Training Data Mitigates Privacy Risks in Language Models
• Large Language Models Struggle to Learn Long-Tail Knowledge
• Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
• Hurdles to Progress in Long-form Question Answering
• Large Language Models with Controllable Working Memory
• Symbolic Chain-of-Thought Distillation: Small Models Can Also ``Think'' Step-by-Step
• Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models
• Few-shot Learning with Multilingual Language Models
• GPT Understands, Too
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Entity-Based Knowledge Conflicts in Question Answering
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• True Few-Shot Learning with Language Models
• How Context Affects Language Models' Factual Predictions
• KILT: a Benchmark for Knowledge Intensive Language Tasks
• Reasoning with Language Model Prompting: A Survey
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Pre-trained Models for Natural Language Processing: A Survey
• Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
• Leveraging Large Language Models for Multiple Choice Question Answering
• Learning To Retrieve Prompts for In-Context Learning
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
• Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• mGPT: Few-Shot Learners Go Multilingual
• Retrieval Augmentation Reduces Hallucination in Conversation
• Prompting GPT-3 To Be Reliable
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
• oLMpics -- On what Language Model Pre-training Captures
• Transformer Memory as a Differentiable Search Index
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
• Multimodal Few-Shot Learning with Frozen Language Models
• Steering Language Models With Activation Engineering
• Information-Theoretic Probing with Minimum Description Length
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
• Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
• Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation
• Generate rather than Retrieve: Large Language Models are Strong Context Generators
• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English
• Explainability for Large Language Models: A Survey
• A Survey of Large Language Models
• Factual Probing Is [MASK]: Learning vs. Learning to Recall

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1250 · arXiv 1909.01066

@inproceedings{petroni2019language,
  title = {{Language Models as Knowledge Bases?}},
  author = {Petroni, Fabio and Rocktäschel, Tim and Riedel, Sebastian and Lewis, Patrick and Bakhtin, Anton and Wu, Yuxiang and Miller, Alexander},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1250},
  eprint = {1909.01066},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1250}
}

References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by https://github.com/allenai/allennlp https://github.com/pytorch/fairseq 9 https://github.com/huggingface/ pytorch-transformers modelling reward. In Inte… ↗
2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD… find our card
3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
4. Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of… find
5. S. R. K. Branavan, David Silver, and Regina Barzilay. 2011. Learning to win by reading manuals in a monte-carlo framework. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA,… find
6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. CoRR, abs/1704.00051. find our card
7. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2018. Babyai: First steps towards grounded language learning with a human in the loop. CoRR, abs/1810.08272. find
8. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860. find
9. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 933–941. find our card
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. find
12. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluati… find
13. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287. find our card
14. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. 2019. Natural questions: a benchmark for question answering research. find our card
17. Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. 2019. A Survey of Reinforcement Learning Informed by Natural Language. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intel… find
18. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1192–1202. find
19. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. find our card
20. Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. CoRR, abs/1707.05589. find
21. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843. find our card
22. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), Miami, FL, USA, December 2-5, 2012, pages 234–239. find
23. Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11–33. find
24. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
25. Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1499–1… find our card
26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
29. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. CoRR, abs/1808.07042. find our card
30. Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1784–1789. find
31. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lear… find our card
32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… find
33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels,… find our card
34. Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. arXiv preprint arXiv:1902.02192. — [arXiv:1902.02192](https://arxiv.org/abs/1902.02192) find
35. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329. find our card
36. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018. From recognition to cognition: Visual commonsense reasoning. CoRR, abs/1811.10830. find
37. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santia… find our card
38. Robert Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679–3686. find
39. Mihai Surdeanu and Heng Ji. 2014. Overview of the English Slot Filling Track at the TAC2014 Knowledge Base Population Evaluation. page 15. find
40. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find