Elamur Research⌕
☰
home / papers / 2019 / To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

Paper · 2019 · Matthew E. Peters, Sebastian Ruder and Noah A. Smith

Summary

Research on transfer learning in NLP has mostly compared pretraining objectives. This paper instead examines the adaptation step: should the pretrained weights stay frozen (feature extraction) or be updated on the target task (fine-tuning)? The authors test ELMo and BERT-base under both regimes on seven tasks. These include NER (CoNLL 2003), sentiment (SST-2), NLI (MultiNLI, SICK-E), paraphrase detection (MRPC) and similarity (STS-B, SICK-R). Hyper-parameters are searched carefully for each task, and Skip-thoughts serves as a sentence-embedding baseline.

The two strategies usually perform about the same. Large gaps appear when the pretraining and target tasks are either very close or very far apart. Fine-tuning BERT helps clearly on similarity and paraphrase tasks, which resemble its next-sentence objective. Fine-tuning ELMo hurts on sentence-pair tasks. Further analyses cover several factors:

  • how pairwise interactions are modelled
  • added task-specific parameters
  • domain distance
  • layer-wise information, measured with diagnostic classifiers and mutual information (EDGE)

The result is a set of practical guidelines for choosing an adaptation strategy.

Key points

  • ELMo: feature extraction beats fine-tuning on sentence-pair tasks.
    • MNLI: 79.6 vs 76.4
    • SICK-E: 86.3 vs 83.3
    • SICK-R: 86.1 vs 83.3
  • BERT-base: fine-tuning wins on the similarity-style tasks.
    • MRPC: 84.8 vs 78.1
    • STS-B: 87.1 vs 82.9
    • SICK-R: 88.7 vs 86.4
    • NER, SST-2 and MNLI differ by at most 0.5.
  • Cross-sentence bi-attention is essential for fine-tuning ELMo on pair tasks. Without it, STS-B dev drops from 80.2 to 38.5. Encoding the two sentences separately also lowers BERT's scores.
  • Extra parameters on NER (CoNLL 2003 dev F1):
    • Feature extraction benefits from a BiLSTM+CRF head (95.5 vs 91.9 with CRF only).
    • Fine-tuning works best with a minimal CRF head plus gradual unfreezing (95.5).
  • Domain distance does not explain the results: Jensen-Shannon divergence between BERT's pretraining text and the MNLI genres showed no significant correlation with the fine-tuning gain.
  • Layer-wise analysis:
    • Pretrained BERT layers carry almost no mutual information with task labels.
    • Fine-tuning raises it: sharply in the top layers for single-sentence tasks, gradually across layers for MRPC.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,763 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,137 citations🎓Harvard UniversityCambridge · United States126 works · 45,130 citations🎓Ollscoil na Gaillimhe – University of GalwayGalway · Ireland25 works · 2,751 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Matthew E. Peters, Sebastian Ruder and Noah A. Smith. 2019. To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. arXiv:1903.05987.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1903.05987 · arXiv 1903.05987

@misc{peters2019tune,
  title = {{To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks}},
  author = {Matthew E. Peters, Sebastian Ruder and Noah A. Smith},
  year = {2019},
  doi = {10.48550/arxiv.1903.05987},
  eprint = {1903.05987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1903.05987}
}

References

40 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In Proceedings of ICLR 2017.
  2. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval@ACL.
  3. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In Proceedings of ACL 2017. find
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  5. Andrew M. Dai and Quoc V. Le. 2015. supervised sequence learning. In NIPS. find
  6. Semi-Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  7. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  8. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  9. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
  10. Kaiming He, Ross Girshick, and Piotr Dollár. 2018. Rethinking ImageNet Pre-training. arXiv preprint arXiv:1811.08883. find
  11. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018. find
  12. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1746–1751.
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  14. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-Thought Vectors. In Proceedings of NIPS 2015. find
  15. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… find
  16. Simon Kornblith, Jonathon Shlens, Quoc V Le, and Google Brain. 2018. Do Better ImageNet Models Transfer Better? arXiv preprint arXiv:1805.08974. find
  17. John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. find
  18. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of NAACL-HLT 2016.
  19. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of ICLR 2018.
  20. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. find
  21. Fixing CoRR, Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In LREC, pages 216–223.
  22. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. find
  23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in Translation: Contextualized Word Vectors. In Advances in Neural Information Processing Systems.
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
  25. Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How Transferable are Neural Networks in NLP Applications? Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing.
  26. Morteza Noshad, Yu Zeng, and Alfred O. Hero III. 2018. Scalable Mutual Information Estimation using Dependence Graphs. arXiv preprint arXiv:1801.09125. — [arXiv:1801.09125](https://arxiv.org/abs/1801.09125) find
  27. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. find
  28. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
  29. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT 2018.
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. find
  31. Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian Optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  32. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL.
  33. Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. 2018. On the Information Bottleneck Theory of Deep Learning. In Proceedings of ICLR 2018. find
  34. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
  35. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. In Proceedings of ICLR 2018. find
  36. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL.
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
  38. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
  39. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
  40. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In NIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peters-2019-tune-tune-adapting-pretrained-representations/