Elamur Research
☰
home / papers / 2020 / K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2002.01808

Summary

Large pre-trained language models, BERT and RoBERTa among them, capture factual and linguistic knowledge only weakly. Earlier knowledge-injection methods retrain all model weights, so each newly added knowledge type tends to erase earlier ones (catastrophic forgetting), and the resulting representations are entangled. The authors propose K-Adapter, which keeps RoBERTa frozen and attaches a separate compact network outside the backbone for each knowledge type. Each adapter reads hidden states from intermediate RoBERTa layers and is pre-trained on its own, so adapters can be trained independently and in parallel. The authors build two adapters. The factual adapter learns relation classification on Wikipedia sentences automatically aligned with Wikidata triplets. The linguistic adapter learns to predict dependency relations that an off-the-shelf parser produced on web text. For downstream tasks, adapter outputs are concatenated and fed to task-specific layers. On relation classification, entity typing and question answering, each adapter outperforms RoBERTa, and combining both helps further. The design enables modular, continual knowledge infusion without touching the base model.

Key points

  • Backbone: RoBERTa LARGE (24 layers, hidden size 1024, 16 heads, 355M parameters; Huggingface implementation) stays fixed. Adapters have far fewer trainable parameters, which makes pre-training memory-efficient.
  • Adapter layer (KIA): Down- and up-projection layers wrap N=2 transformer layers, with a skip connection between them. The layer's input is the backbone layer output concatenated with the previous adapter layer's output. This differs from Houlsby et al. (2019), who insert adapters inside every transformer layer.
  • Evaluation: Six benchmarks across three tasks. K-Adapter consistently beats RoBERTa and a RoBERTa+multitask baseline and reaches state of the art on five datasets. On TACRED it outperforms all compared baselines.
  • Probing: Zero-shot LAMA cloze probing shows more accurate factual predictions than RoBERTa, for example correctly ranking "Malayalam" first as Mammootty's native language.
  • Code: Released at github.com/microsoft/k-adapter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

206 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 200 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2002.01808 · arXiv 2002.01808

@misc{wang2020k,
  title = {{K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters}},
  author = {Wang, Ruize and Tang, Duyu and Duan, Nan and Wei, Zhongyu and Huang, Xuanjing and ji, Jianshu and Cao, Guihong and Jiang, Daxin and Zhou, Ming},
  year = {2020},
  doi = {10.48550/arxiv.2002.01808},
  eprint = {2002.01808},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.01808}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Relation Learning. In ACL, pages 2895–2905. find
  2. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787–2795. find
  3. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2018. Ask the Right Questions: Active Question Reformulation with Reinforcement Learning. In ICLR. find
  4. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In ACL, pages 87–96. find
  5. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, pages 2978–2988. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186. find
  7. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. In arXiv preprint arXiv:1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
  8. Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. In ArXiv preprint arXiv:1704.05179. find our card
  9. Hady ElSahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Simperl. 2018. T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples. In LREC. find
  10. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  11. Bin He, Di Zhou, Jinghui Xiao, Qun Liu, Nicholas Jing Yuan, Tong Xu, et al. 2019. Integrating Graph Contextualized Knowledge into Pre-trained Language Models. In arXiv preprint arXiv:1912.00147. — [arXiv:1912.00147](https://arxiv.org/abs/1912.00147) find
  12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In ICML, pages 2790–2799. find
  13. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In EMNLP, pages 2391–2401. find our card
  14. Nora Kassner and Hinrich Schütze. 2019. Negated LAMA: Birds cannot fly. arXiv preprint arXiv:1911.03343. find our card
  15. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13)… find
  16. Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2019. Informing unsupervised pretraining with external linguistic knowledge. arXiv preprint arXiv:1909.02339. — [arXiv:1909.02339](https://arxiv.org/abs/1909.02339) find
  17. Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019. Sensebert: Driving some sense into bert. arXiv preprint arXiv:1908.05646. find
  18. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In ACL, pages 1736–1745. find
  19. Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design challenges for entity linking. TACL, 3:315– 328. find
  20. Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada. AAAI Press. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL, pages 2227–2237. find
  23. Matthew E Peters, Mark Neumann, IV Logan, L Robert, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP, pages 43– 54. find
  24. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language Models as Knowledge Bases? In EMNLP, pages 2463–2473. find our card
  25. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA. arXiv preprint arXiv:1911.03681. find
  26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Blog. find
  27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  29. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional Attention Flow for Machine Comprehension. ArXiv, abs/1611.01603. find our card
  30. Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2016. An attentive neural architecture for fine-grained entity type classification. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction(AKBC), pages 69– 74. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008. find
  32. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2017a. Reinforced reader-ranker for open-domain question answering. arXiv preprint arXiv:1709.00023. — [arXiv:1709.00023](https://arxiv.org/abs/1709.00023) find
  33. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR. find
  34. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017b. Gated self-matching networks for reading comprehension and question answering. In ACL, pages 189–198. find
  35. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. arXiv preprint arXiv:1911.06136. find
  36. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model. In ICLR. find
  37. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv preprint arXiv:1906.08237. find our card
  38. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. In EMNLP, pages 2205–2215. find our card
  39. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware Attention and Supervised Data Improve Slot Filling. In EMNLP, pages 35–45. find
  40. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In ACL, pages 1441–1451. find
  41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In ICCV, pages 19–27. find our card