Elamur Research⌕
☰
home / papers / 2016 / How Transferable are Neural Networks in NLP Applications?

How Transferable are Neural Networks in NLP Applications?

Paper · 2016 · Lili Mou, Zhao Meng, Rui Yan et al.

Summary

Neural NLP models overfit easily when the dataset of interest is small, so reusing knowledge from a larger source dataset is attractive. Earlier neural NLP work applied transfer in an ad hoc way and reported contradictory outcomes: tiny gains in one study, large jumps in another. The authors run a controlled study on six public datasets in two groups. The first group is single-sentence classification with an LSTM-RNN, using IMDB as the source and MR and QC as targets. The second is sentence-pair classification with a Siamese CNN, using SNLI as the source and SICK and MSRP as targets. In each group, one target has the same task meaning as the source and the other does not, while the network architecture stays identical. The authors compare three strategies: initializing the target model with source parameters (INIT), training both tasks jointly (MULT), and combining the two. They also examine which layers carry over. Transfer success depends mainly on how semantically close the tasks are, which differs from image processing. This gives practitioners realistic expectations about when neural transfer in NLP pays off.

Key points

  • Dataset sizes (train/val/test):
    • IMDB: 550,000 / 50,000 / 2,000
    • MR: 8,500 / 1,100 / 1,100
    • QC: 4,800 / 600 / 500
    • SNLI: 550,152 / 10,000 / 10,000
    • SICK: 4,439 / 495 / 4,906
    • MSRP: 3,575 / 501 / 1,725
  • Non-transfer baselines were averaged over 5 random initializations and are close to comparable published models. Examples include MR at 75.1 ± 0.6, SICK at 70.9 ± 1.3 and MSRP at 69.0 ± 0.5.
  • Transfer helped the semantically equivalent pairs (IMDB→MR, SNLI→SICK) but not the semantically different targets (QC, MSRP). This held for both INIT and MULT.
  • Layer-level findings:
    • The output layer is largely tied to its dataset and does not transfer.
    • Word embeddings (pretrained with word2vec, 100-dimensional) may still help semantically different tasks.
  • MULT and INIT reached similar peak accuracy. MULT was slightly lower than INIT in Experiment I and slightly higher in Experiment II. MULT+INIT was stable across values of λ but did not beat either method alone.
  • In SNLI→SICK, parameters taken from epoch 2 of source training gave the best SICK accuracy (78.98%). At that point SNLI accuracy was only 72.65%, versus 76.26% at epoch 23. Transfer accuracy later fell by about 2.7%, while IMDB→MR showed no such mismatch.

Interesting ideas

  • I1. Paired targets per source. Each source gets one same-meaning target and one different-meaning target, all under a fixed architecture. This isolates task semantics as the variable and would suit testing transfer between, say, Ukrainian corpora or tasks.
  • I2. Layer-by-layer transfer notation. Each layer (embedding, hidden, output) is marked as not transferred, transferred and frozen, or transferred and fine-tuned. This systematically maps which components are reusable and is cheap to replicate for any layered model.
  • I3. Transferring from intermediate source epochs. Copying source parameters at several training checkpoints showed that a well-trained source model is not always the best donor. This could save compute when the source dataset is large.
  • I4. Tuning the mixing weight λ. In multi-task training, λ balances source and target losses. The authors first searched it coarsely, then finely near the small values that worked well. This is a simple, transferable recipe for weighting auxiliary data.
  • I5. Fair baselines. Baselines average several random seeds, while the source model is trained only once, since a specific trained model is what gets transferred. This keeps the comparison honest without inflating cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China100 works · 35,618 citations🔬Institute of SoftwareBeijing · China10 works · 4,807 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang and Zhi Jin. 2016. How Transferable are Neural Networks in NLP Applications?. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1046

@inproceedings{mou2016how,
  title = {{How Transferable are Neural Networks in NLP Applications?}},
  author = {Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang and Zhi Jin},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1046},
  url = {https://doi.org/10.18653/v1/d16-1046}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2006. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, pages 153–160. find
  2. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 120–128.
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  4. Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a “Siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669–688. find
  5. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. find
  6. Hal Daumé III, Abhishek Kumar, and Avishek Saha. 2010. Frustratingly easy semi-supervised domain adaptation. In Proceedings of the Workshop on Domain Adaptation for Natural Language Processing, pages 53–59. find
  7. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263.
  8. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on Machine Learning, pages 647–655. find
  9. Li Dong, Furu Wei, Shujie Liu, Ming Zhou, and Ke Xu. 2015. A statistical parsing framework for sentiment classification. Computational Linguistics, 41(2):293–336. find
  10. George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 451–459.
  11. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th International Conference on Machine Learning, pages 513–520. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  13. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems, pages 2042–2050. find
  14. Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264– 271.
  15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105. find
  16. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2750–2756.
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119.
  18. Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015. A comparative study on regularization strategies for embedding-based neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2106–2111. find
  19. Barbara Plank and Alessandro Moschitti. 2013. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1498–1507. find
  20. Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural correspondence learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1118–1127.
  21. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161.
  22. Dong Wang and Thomas Fang Zheng. 2015. Transfer learning for speech and language processing. In Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pages 1225–1237. find
  23. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, pages 3320–3328. find
  24. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of 13th European Conference on Computer Vision, pages 818–833.
  25. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In International Joint Conference on Artificial Intelligence, pages 4069–4076. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mou-2016-how-transferable-are-neural-networks/