Elamur Research⌕
☰
home / papers / 2019 / Inducing Relational Knowledge from BERT

Inducing Relational Knowledge from BERT

Paper · 2019 · Zied Bouraoui, Jose Camacho-Collados and Steven Schockaert

Summary

The paper investigates whether pre-trained language models like BERT store relational knowledge beyond what static word embeddings (Skip-gram, GloVe) already encode. The authors frame this as relation induction: given a few example pairs, decide whether a new pair belongs to the same relation. BERT's answers depend heavily on how a query sentence is worded, so the authors automate the search for good prompts. They collect corpus sentences that mention the seed pairs and treat them as templates. They keep only the templates from which a pre-trained model can recover the masked head or tail word of the seed examples. BERT is then fine-tuned to judge candidate pairs inserted into the retained templates. No corpus sentences about the candidate pair are consulted, so predictions depend solely on what the model has internalized. The method was tested on relations from the Google analogy dataset, BATS and DiffVec. It clearly beats embedding-based methods on encyclopedic and commonsense relations, roughly matches them on lexical ones, and performs poorly on morphological ones. This shows that factual knowledge can be pulled out of BERT without hand-written patterns.

Key points

  • The pipeline has three steps: collect sentences containing seed pairs, filter templates by checking whether BERT can predict the masked head or tail words, and fine-tune BERT as a pair classifier over the instantiated templates.
  • Probing BERT-Large-Uncased with short hand-made sentences gives mixed results. It handles capital-of well but fails on colours, and adding a single word such as "often" can flip an answer from sensible to meaningless.
  • Automatic templates far outperform short manual ones (F1, hand vs. auto): capital-world 51.4 vs. 89.5 and currency 46.2 vs. 93.6 (Google), item:location 28.6 vs. 54.2 and concealment 25.7 vs. 52.7 (DiffVec).
  • Selected templates tend to be long and context-rich. Some give only indirect evidence, for example sentences about embassies located in capitals.
  • The method gives strong gains on encyclopedic and commonsense relations, performs on par with word vectors for hypernymy and meronymy, and is unsuitable for morphological relations in BATS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cardiff UniversityCardiff · United Kingdom23 works · 2,549 citations🔬Centre de Recherche en Informatique de LensLens · France3 works · 105 citations🎓Université d'ArtoisArras · France3 works · 105 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Zied Bouraoui, Jose Camacho-Collados and Steven Schockaert. 2019. Inducing Relational Knowledge from BERT. arXiv:1911.12753.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.12753 · arXiv 1911.12753

@misc{bouraoui2019inducing,
  title = {{Inducing Relational Knowledge from BERT}},
  author = {Zied Bouraoui, Jose Camacho-Collados and Steven Schockaert},
  year = {2019},
  doi = {10.48550/arxiv.1911.12753},
  eprint = {1911.12753},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.12753}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, C., and Hospedales, T. M. 2019. Analogies explained: Towards understanding word embeddings. In Proc. ICML. find
  2. Amrami, A., and Goldberg, Y. 2018. Word sense induction with neural biLM and symmetric patterns. In Proc. EMNLP, 4860–4867. find
  3. Arora, S.; Li, Y.; Liang, Y.; Ma, T.; and Risteski, A. 2016. A latent variable model approach to pmi-based word embeddings. TACL 4:385–399.
  4. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proc. ACL, 4762–4779.
  5. Bouraoui, Z.; Jameel, S.; and Schockaert, S. 2018. Relation induction in word embeddings revisited. In Proc. find
  6. COLING. Camacho-Collados, J.; Espinosa-Anke, L.; Jameel, S.; and Schockaert, S. 2019. A latent variable model for learning distributional relation vectors. In Proc. find
  7. IJCAI. Camacho-Collados, J.; Espinosa-Anke, L.; and Schockaert, S. 2019. Relational word embeddings. In Proc. ACL. find
  8. Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT.
  9. Drozd, A.; Gladkova, A.; and Matsuoka, S. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proc. COLING, 3519–3530. find
  10. Espinosa Anke, L., and Schockaert, S. 2018. SeVeN: Augmenting word embeddings with unsupervised relation vectors. In Proc. COLING, 2653–2665. find
  11. Forbes, M.; Holtzman, A.; and Choi, Y. 2019. Do neural language representations learn physical commonsense? Proc. of the Conference of the Cognitive Science Society.
  12. Gladkova, A.; Drozd, A.; and Matsuoka, S. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proc. of the Student Research Workshop at NAACL, 8–15.
  13. Goldberg, Y. 2019. Assessing bert’s syntactic abilities. CoRR abs/1901.05287.
  14. Gordon, J., and Durme, B. V. 2013. Reporting bias and knowledge acquisition. In Proc. AKBC, 25–30. Gupta, A.; Boleda, G.; Baroni, M.; and Padó, S. 2015. Distributional vectors encode referential attributes. In Proc. EMNLP, 12–21. find
  15. Hewitt, J., and Manning, C. D. 2019. A structural probe for finding syntax in word representations. In Proc. NAACL, 4129–4138.
  16. Jameel, S.; Bouraoui, Z.; and Schockaert, S. 2018. Unsupervised learning of distributional relation vectors. In Proc. ACL, 23–33. find
  17. Jawahar, G.; Sagot, B.; and Seddah, D. 2019. What does BERT learn about the structure of language? In Proc. ACL, 3651–3657.
  18. Joshi, M.; Choi, E.; Levy, O.; Weld, D. S.; and Zettlemoyer, L. 2019. pair2vec: Compositional word-pair embeddings for cross-sentence inference. In Proc. NAACL. find
  19. Levy, O.; Goldberg, Y.; and Ramat-Gan, I. 2014. Linguistic regularities in sparse and explicit word representations. In Proc. CoNLL, 171–180.
  20. Linzen, T. 2016. Issues in evaluating semantic spaces using word analogies. In Proc. of the Workshop on Evaluating Vector-Space Representations for NLP, 13–18. find
  21. Logan IV, R. L.; Liu, N. F.; Peters, M. E.; Gardner, M.; and Singh, S. 2019. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In Proc. ACL, 5962–5971. find
  22. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. In Proc. ICLR.
  23. Mikolov, T.; Yih, W.-t.; and Zweig, G. 2013. Linguistic regularities in continuous space word representations. In Proc. NAACL, 746–751.
  24. Nissim, M.; van Noord, R.; and van der Goot, R. 2019. Fair is better than sensational:man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866.
  25. Pennington, J.; Socher, R.; and Manning, C. D. 2014. GloVe: Global vectors for word representation. In Proc. EMNLP, 1532–1543. Petroni, F.; Rocktäschel, T.; Riedel, S.; Lewis, P.; Bakhtin, A.; Wu, Y.; and Miller, A. 2019. Language models as knowledge bases? In Proc. EMNLP, 2463–2473.
  26. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. Technical Report. find
  27. Rogers, A.; Drozd, A.; and Li, B. 2017. The (too many) problems of analogical reasoning with word vectors. In Proc. *SEM, 135–148. find
  28. Rubinstein, D.; Levi, E.; Schwartz, R.; and Rappoport, A. 2015. How well do distributional models capture different types of semantic knowledge? In Proc. ACL, 726–730. find
  29. Tenney, I.; Das, D.; and Pavlick, E. 2019. BERT rediscovers the classical NLP pipeline. In Proc. COLING, 4593–4601.
  30. Vu, T., and Shwartz, V. 2018. Integrating multiplicative features into supervised distributional methods for lexical entailment. In Proc. *SEM, 160–166. Vulić, I., and Mrkšić, N. 2018. Specialising word vectors for lexical entailment. In Proc. NAACL, 1134–1145.
  31. Vylomova, E.; Rimell, L.; Cohn, T.; and Baldwin, T. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proc. ACL. find
  32. Washio, K., and Kato, T. 2018a. Filling missing paths: Modeling co-occurrences of word pairs and dependency paths for recognizing lexical semantic relations. In Proc. NAACL. find
  33. Washio, K., and Kato, T. 2018b. Neural latent relational analysis to capture lexical semantic relations in a vector space. In Proc. EMNLP, 594–600. find
  34. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bouraoui-2019-inducing-relational-knowledge-bert/