Elamur Research
☰
home / papers / 2019 / Transfer Learning in Natural Language Processing

Transfer Learning in Natural Language Processing

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-5004.pdf

Summary

This document is a NAACL 2019 tutorial abstract, not an empirical study. It outlines a three-hour course on transfer learning for NLP. The motivation is that standard supervised learning trains one model on one dataset in isolation. That approach needs many labeled examples and suits mainly narrow, well-defined tasks. Transfer learning instead draws on additional tasks or domains to improve generalization. The authors observe that within about two years, pretrained approaches such as language-model fine-tuning (Howard and Ruder, 2018) and BERT raised state-of-the-art results on many tasks. They compare this shift to the earlier impact of word embeddings and of ImageNet pretraining in vision.

The tutorial centers on sequential transfer. A pretraining stage uses unsupervised, supervised, distantly supervised or multi-task objectives. An adaptation stage then applies feature extraction or fine-tuning, with attention to learning-rate schedules, architecture changes and regularization. Further sessions probe what the learned representations encode, work through hands-on downstream examples and discuss open problems. No new data, models or scores are reported. The contribution is a structured map of the field plus practical guidance.

Key points

  • Published in Proceedings of NAACL HLT 2019: Tutorial Abstracts, pp. 15–18 (Minneapolis, June 2, 2019). Presenters: Ruder, Peters, Swayamdipta and Wolf, affiliated with Insight Centre/Aylien, AI2, CMU LTI and Huggingface.
  • The 3-hour schedule has seven parts:
    • introduction (15 min)
    • pretraining (35 min)
    • representation analysis (20 min)
    • break (20 min)
    • adaptation (30 min)
    • downstream applications (40 min)
    • open problems (20 min)
  • Pretraining coverage runs from non-neural methods (LSA, Brown clusters, topic models) to skip-thoughts/auto-encoders and language-model approaches (BERT). It also includes supervised sentence encoders (e.g., NLI-based, Universal Sentence Encoder) and distant supervision such as emoji labels (Felbo et al., 2017).
  • Adaptation topics include fine-tuning varying portions of the model, specialized regularization and sequential fine-tuning on several datasets. Architecture changes range from extra embeddings to layers inserted inside the pretrained network.
  • Prerequisites are familiarity with RNN, CNN and Transformers and with standard NLP tasks. The expected audience was about 200 attendees.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 105 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-5004

@inproceedings{ruder2019transfer,
  title = {{Transfer Learning in Natural Language Processing}},
  author = {Ruder, Sebastian and Peters, Matthew E. and Swayamdipta, Swabha and Wolf, Thomas},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials},
  year = {2019},
  doi = {10.18653/v1/n19-5004},
  url = {https://doi.org/10.18653/v1/n19-5004}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4-5):993–1022. find our card
  2. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks.… find
  3. Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  4. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-Thought Vectors. In Proceedings of NIPS 2015. find
  5. Anonymous. 2019. Looking for ELMo’s friends: Sentence-Level Pretraining Beyond Language Modeling. find
  6. Georg Wiese, Dirk Weissenborn, and Mariana Neves. 2017. Neural Domain Adaptation for Biomedical Question Answering. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). find
  7. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of ICLR 2018. find our card
  8. Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. 2015. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of ICML, volume 37, Lille, France. find
  9. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards Universal Paraphrastic Sentence Embeddings. In Proceedings of ICLR. find our card
  10. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient Episodic Memory for Continuum Learning. In Proceedings of NIPS 2017. find
  11. Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural Word Segmentation with Rich Pretraining. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017). find
  12. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in Translation: Contextualized Word Vectors. In Advances in Neural Information Processing Systems. find our card
  13. Kelly W Zhang and Samuel R Bowman. 2018. Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis. arXiv preprint arXiv:1809.10040. find
  14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems. find
  15. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer Learning for Low-Resource Neural Machine Translation. In Proceedings of EMNLP 2016. find
  16. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 1003–1011. find
  17. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. find
  18. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of NAACL-HLT 2018. find
  19. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, Wen-tau Yih, Paul G Allen, and Computer Science. 2018b. Dissecting Contextual Word Embeddings: Architecture and Representation. In Proceedings of EMNLP 2018. find our card
  20. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. find
  21. Aliaksei Severyn and Alessandro Moschitti. 2015. UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment Classification. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 464–469. find
  22. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. In Proceedings of ICLR 2018. find
  23. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic Scaffolds for Semantic Structures. In Proceedings of EMNLP 2018. find