Elamur Research
☰
home / papers / 2019 / K-BERT: Enabling Language Representation with Knowledge Graph

K-BERT: Enabling Language Representation with Knowledge Graph

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1909.07606

Summary

BERT-style models learn general language from open-domain text but lack the specialist knowledge needed for fields like medicine, finance, or law. Pre-training a domain-specific model is costly. K-BERT addresses this by inserting relevant knowledge graph triples directly into input sentences at fine-tuning time. A knowledge layer looks up entities in a KG and attaches their triples, producing a sentence tree with depth 1. Too much injected knowledge can distort a sentence's meaning, a problem the authors call knowledge noise. To prevent this, K-BERT uses two mechanisms. Soft-position embeddings preserve the original word order. A visible matrix inside a mask-transformer limits which tokens can attend to the added triples. Because K-BERT keeps BERT's parameters unchanged, any pre-trained BERT checkpoint can be loaded without extra pre-training. On twelve Chinese NLP tasks, using HowNet, CN-DBpedia and a MedicalKG, gains were small on open-domain tasks but consistent on domain-specific ones. This makes K-BERT a cheap way to add expert knowledge to language models.

Key points

  • Architecture: K-BERT has four components: a knowledge layer, an embedding layer (token, soft-position and segment embeddings), a seeing layer that builds the visible matrix, and a mask-transformer encoder.
  • Domain benchmarks: Finance Q&A and Law Q&A were built from data crawled from Baidu Zhidao, roughly 770k and 36k samples. Finance NER covers 3,000 news articles. Medicine NER is the CCKS 2017 CNER task.
  • F1 gains over Google BERT:
    • Medicine NER: 92.5 → 94.2 (MedicalKG)
    • Law Q&A: 86.4 → 87.5 (CN-DBpedia)
    • Finance NER: 86.1 → 87.6 (HowNet)
  • Task-dependent effects: KGs gave no clear benefit on sentiment tasks. HowNet outperformed the encyclopedic KG on semantic similarity.
  • Ablation: Without the visible matrix, K-BERT scored below plain BERT on Law Q&A, which confirms the knowledge noise problem. K-BERT also peaked earlier (epoch 2 vs. 4). Code and KGs are released at github.com/autoliuweijie/K-BERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 193 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.07606 · arXiv 1909.07606

@misc{liu2019k,
  title = {{K-BERT: Enabling Language Representation with Knowledge Graph}},
  author = {Liu, Weijie and Zhou, Peng and Zhao, Zhe and Wang, Zhiruo and Ju, Qi and Deng, Haotang and Wang, Ping},
  year = {2019},
  doi = {10.48550/arxiv.1909.07606},
  eprint = {1909.07606},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.07606}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bodenreider, O. 2008. Biomedical ontologies in action: role in knowledge management, data integration and decision support. Yearbook of medical informatics 17(01):67– 79. find
  2. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, 2787–2795. find
  3. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317. find our card
  4. Cao, Y.; Hou, L.; Li, J.; Liu, Z.; Li, C.; Chen, X.; and Dong, T. 2018. Joint representation learning of cross-lingual words and entities via attentive distant supervision. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  5. Conneau, A.; Lample, G.; Rinott, R.; Williams, A.; Bowman, S. R.; Schwenk, H.; and Stoyanov, V. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  6. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Yang, Z.; Wang, S.; and Hu, G. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101. find our card
  7. Dai, Z.; Yang, Z.; Yang, Y.; Cohen, W. W.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. find
  8. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  9. Dong, Z.; Dong, Q.; and Hao, C. 2006. Hownet and the computation of meaning. Citeseer. find
  10. Han, X.; Liu, Z.; and Sun, M. 2016. Joint representation learning of text and knowledge for knowledge graph completion. arXiv preprint arXiv:1611.04125. — [arXiv:1611.04125](https://arxiv.org/abs/1611.04125) find
  11. Joshi, M.; Chen, D.; Liu, Y.; Weld, D. S.; Zettlemoyer, L.; and Levy, O. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. find our card
  12. Levow, G. A. 2006. The 3rd international chinese language processing bakeoff. In Sighan Workshop on Chinese Language Processing. find
  13. Liu, X.; Chen, Q.; Deng, C.; Zeng, H.; Chen, J.; Li, D.; and Tang, B. 2018. LCQMC:a large-scale chinese question matching corpus. International conference on computational linguistics 1952–1962. find
  14. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. find
  15. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  16. Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  17. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  18. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; and Wu, H. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. find our card
  19. Toutanova, K.; Chen, D.; Pantel, P.; Poon, H.; Choudhury, P.; and Gamon, M. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1499–1509. find
  20. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008. find
  21. Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014. Knowledge graph and text jointly embedding. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1591–1601. find
  22. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  23. Xu, B.; Xu, Y.; Liang, J.; Xie, C.; Liang, B.; Cui, W.; and Xiao, Y. 2017. Cn-dbpedia: A never-ending chinese knowledge extraction system. International conference industrial, engineering and other applications applied intelligent systems 428–438. find
  24. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  25. Zhang, Z.; Han, X.; Liu, Z.; Jiang, X.; Sun, M.; and Liu, Q. 2019. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129. find