Elamur Research⌕
☰
home / papers / 2019 / Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Paper · 2019 · Wenhan Xiong, Jingfei Du, William Yang Wang et al.

Summary

Pretrained language models such as BERT seem to absorb some real-world knowledge, but their token-level objectives do not target facts about entities directly. The authors first measure how much entity knowledge these models hold. They turn Wikidata triples for 10 common relations into cloze queries and rank candidate entities, which lets them score multi-token names automatically. The test shows that BERT and GPT-2 capture such facts only partially. To improve this, they introduce WKLM, a weakly supervised pretraining scheme. Entity mentions in English Wikipedia are found through anchor links and Wikidata aliases. Some mentions are then swapped for other entities of the same Wikidata type, and the model learns to detect which mentions were replaced. This task is trained together with a reduced masked-LM objective. WKLM ranks facts much better than the baselines on most relations. It also beats BERT on four open-domain QA datasets and sets a new state of the art on FIGER entity typing. Because the knowledge comes from raw text alone, fine-tuning needs no external knowledge base, extra memory or architecture changes.

Key points

  • Downstream gains: WKLM improves on BERT by 2.7 F1 on average across WebQuestions, TriviaQA, SearchQA and Quasar-T, and by 5.7 accuracy points on FIGER.
  • Fact completion benchmark: 1000 cloze queries for each of 10 Wikidata relations. Negative candidates are drawn from the objects of the same relation, and results are reported as Hits@10. The baselines are BERT base/large and GPT-2.
  • Replacement setup:
    • Text is split into 512-token chunks.
    • Each chunk is copied 10 times, each copy with different negative entities.
    • Adjacent entities are never both replaced.
    • The replacement classifier reads the representations of the words on either side of the mention.
  • Model and training:
    • The architecture is BERT base (12 layers, hidden size 768).
    • It starts from an in-house Fairseq BERT trained for 2M updates on BooksCorpus and Wikipedia.
    • Pretraining then runs for about 1M updates at batch size 128, with 5% masking that skips entity spans.
    • This took 32 V100 GPUs for 3 days.
  • Robustness: BERT's fact-completion scores vary strongly with the size of the candidate set and the number of correct answers, while WKLM is less sensitive to both factors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Meta (Israel)Tel Aviv · Israel46 works · 42,810 citations🎓University of California, Santa BarbaraSanta Barbara · United States67 works · 10,550 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Wenhan Xiong, Jingfei Du, William Yang Wang and Veselin Stoyanov. 2019. Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model. arXiv:1912.09637.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.09637 · arXiv 1912.09637

@misc{xiong2019pretrained,
  title = {{Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model}},
  author = {Wenhan Xiong, Jingfei Du, William Yang Wang and Veselin Stoyanov},
  year = {2019},
  doi = {10.48550/arxiv.1912.09637},
  eprint = {1912.09637},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.09637}
}

References

49 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language model. arXiv preprint arXiv:1608.00318, 2016.
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, pp. 1533–1544. ACL, 2013.
  3. Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pp. 2787–2795, 2013.
  4. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. COMET: commonsense transformers for automatic knowledge graph construction. In ACL (1), pp. 4762–4779. Association for Computational Linguistics, 2019.
  5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In ACL (1), pp. 1870–1879. find
  6. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pp. 4171–4186. ACL, 2019.
  8. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904, 2017. find
  9. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017.
  10. Charles J Fillmore et al. Frame semantics and the nature of language. In Annals of the New York Academy of Sciences: Conference on the origin and development of language and speech, volume 280, pp. 20–32, 1976. find
  11. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pp. 1693– 1701, 2015.
  12. Kentaro Inui, Sebastian Riedel, Pontus Stenetorp, and Sonse Shimaoka. Neural architectures for fine-grained entity type classification. In EACL (1), pp. 1271–1280. Association for Computational Linguistics, 2017. find
  13. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL (1), pp. 1601–1611. Association for Computational Linguistics, 2017.
  14. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In ACL (1), pp. 6086–6096. Association for Computational Linguistics, 2019.
  16. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. Denoising distantly supervised open-domain question answering. In ACL (1), pp. 1736–1745. Association for Computational Linguistics, 2018.
  17. Xiao Ling and Daniel S. Weld. Fine-grained entity recognition. In AAAI. AAAI Press, 2012. find
  18. Angli Liu, Jingfei Du, and Veselin Stoyanov. Knowledge-augmented language model and its application to unsupervised named-entity recognition. In NAACL-HLT (1), pp. 1142–1150. Association for Computational Linguistics, 2019a. find
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019b.
  20. Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In ACL, pp. 5962–5971. ACL, 2019. find
  21. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In NIPS, pp. 6294–6305, 2017.
  22. Todor Mihaylov and Anette Frank. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 821–832, Melbourne, Australia, July 2018. As…
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.
  24. Marvin Minsky. Society of mind. Simon and Schuster, 1988. find
  25. Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In ICML, volume 11, pp. 809–816, 2011. find
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  27. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532–1543. ACL, 2014.
  28. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018a.
  29. Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In EMNLP, pp. 1499–1509. Association for Computational Linguistics, 2018b.
  30. Matthew E Peters, Mark Neumann, IV Logan, L Robert, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. EMNLP, 2019. find
  31. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? EMNLP, 2019.
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, pp. 2383–2392. The Association for Computational Linguistics, 2016. — SQuAD: 100,000+ Questions for Machine Comprehension of Text
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998–6008, 2017.
  35. Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledge base. 2014. find
  36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019a.
  37. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. R3 : Reinforced ranker-reader for open-domain question answering. In AAAI, pp. 5981–5988. AAAI Press, 2018a. find
  38. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR, 2018b. find
  39. Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv:1908.08167, 2019b.
  40. Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 564–573, Copenhagen, Denmark, September 2017. Association for Computational Lingui… find
  41. Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Improving question answering over incomplete KBs with knowledge-aware reader. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational L… find
  42. Bishan Yang and Tom M. Mitchell. Leveraging knowledge bases in lstms for improving machine reading. In ACL (1), pp. 1436–1446. ACL, 2017. find
  43. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718, 2019a.
  44. Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. Data augmentation for bert fine-tuning in open-domain question answering. arXiv preprint arXiv:1904.06652, 2019b. — [arXiv:1904.06652](https://arxiv.org/abs/1904.06652)
  45. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019c.
  46. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP, pp. 93–104. ACL, 2018. find
  47. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018.
  48. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: enhanced language representation with informative entities. In ACL (1), pp. 1441–1451. ACL, 2019. find
  49. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19–27. IEEE Computer Society, 2015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2019-pretrained-encyclopedia-weakly-supervised-knowledge/