Elamur Research⌕
☰
home / papers / 2017 / Regularization techniques for fine-tuning in neural machine translation

Regularization techniques for fine-tuning in neural machine translation

Paper · 2017 · Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann et al.

Summary

The paper looks at supervised domain adaptation for neural machine translation (NMT). A model is first trained on a large general-domain corpus and then fine-tuned on a small in-domain set. In that second stage, overfitting is a central risk. The usual safeguard, early stopping, requires holding back some of the scarce in-domain data for validation. The authors compare three regularizers for fine-tuning: Bayesian (recurrent) dropout, MAP-L2, and tuneout. MAP-L2 penalizes the L2 distance of the weights from the out-of-domain weights. Tuneout is a new method that resets randomly dropped weight columns to their out-of-domain values instead of zero. Experiments use Nematus models trained on WMT data and adapted to IWSLT TED-talk data for English→German and English→Russian. Using dropout and MAP-L2 together yields small but significant gains over plain fine-tuning, about 0.3–0.4 BLEU, and makes training resistant to overfitting. Tuneout gives mixed results. Experiments with smaller in-domain samples show that BLEU rises roughly logarithmically with data size. This matters for low-resource and online adaptation settings where a validation set is impractical.

Key points

  • Data:
    • Out-of-domain training used WMT corpora plus back-translations, with about 7.92M sentence pairs for En→De and 4.06M for En→Ru.
    • In-domain training used IWSLT data, with about 206k pairs for En→De (IWSLT 2015) and 181k for En→Ru (IWSLT 2014).
    • All text was segmented with BPE.
  • Main result: Combining dropout and MAP-L2 raised average test BLEU over the fine-tuning baseline in both directions. Both improvements are significant at 5%.
    • En→De went from 31.20 to 31.48.
    • En→Ru went from 18.67 to 19.10.
  • Tuneout: It was significantly worse for En→De, with an average of 30.78. It gave only a slight, non-significant gain for En→Ru.
  • Training stability: Unregularized fine-tuning began overfitting after roughly 2–3 epochs. The regularized runs converged without visible overfitting, so they can be trained for a fixed number of epochs without a held-out set.
  • Data size and code:
    • Fine-tuning was tested on random samples ranging from 10 to 206k sentence pairs, and the BLEU gain grew roughly logarithmically with sample size.
    • Dropout and MAP-L2 are included in the Nematus master branch. Tuneout is available in a separate branch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine translation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1156

@inproceedings{barone2017regularization,
  title = {{Regularization techniques for fine-tuning in neural machine translation}},
  author = {Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann and Rico Sennrich},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1156},
  url = {https://doi.org/10.18653/v1/d17-1156}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” Proceedings of the International Conference on Learning Representations (ICLR). our card
  2. Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi… our card
  3. Cettolo, Mauro, Christian Girardi, and Marcello Federico. 2012. “WIT 3 : Web Inventory of Transcribed and Translated Talks.” Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), 261–268. Trento, Italy. find
  4. Cettolo, Mauro, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2016. “Report on the 13th IWSLT Evaluation Campaign.” IWSLT 2016. Seattle, USA. find
  5. Chelba, Ciprian and Alex Acero. 2006. “Adaptation of maximum entropy capitalizer: Little data can help a lot.” Computer Speech & Language, 20(4):382– 399. find
  6. Chu, Chenhui, Raj Dabre, and Sadao Kurohashi. 2017. “An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada. find
  7. Daume III, Hal. 2007. “Frustratingly Easy Domain Adaptation.” Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 256–263. Prague, Czech Republic. our card
  8. Finkel, Jenny Rose and Christopher D. Manning. 2009. “Hierarchical Bayesian Domain Adaptation.” Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL ’09, 602–610. Stroudsburg, PA, USA. find
  9. Gal, Yarin and Zoubin Ghahramani. 2016. “A Theoretically Grounded Application of Dropout in Recurrent Neural Networks.” Advances in Neural Information Processing Systems 29 (NIPS). our card
  10. Hinton, Geoffrey E and Ruslan R Salakhutdinov. 2006. “Reducing the dimensionality of data with neural networks.” Science, 313(5786):504–507. find
  11. Kingma, Diederik P. and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” The International Conference on Learning Representations. San Diego, California, USA. find
  12. Kirkpatrick, James, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. “Overcoming catastrophic forgetting in neural net… find
  13. Koehn, Philipp. 2004. “Statistical Significance Tests for Machine Translation Evaluation.” Proceedings of EMNLP 2004, 388–395. Barcelona, Spain. our card
  14. Luong, Minh-Thang and Christopher D. Manning. 2015. “Stanford Neural Machine Translation Systems for Spoken Language Domains.” Proceedings of the International Workshop on Spoken Language Translation 2015. Da Nang, Vietnam. 54th Annual Meeting of the Association for Computational Linguistics (Volume… find
  15. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016c. “Neural Machine Translation of Rare Words with Subword Units.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725. Berlin, Germany. our card
  16. Srivastava, Nitish, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. “Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research, 15(1):1929–1958. find
  17. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. “Sequence to Sequence Learning with Neural Networks.” Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, 3104–3112. Montreal, Quebec, Canada. find
  18. Wuebker, Joern, Spence Green, and John DeNero. 2015. “Hierarchical Incremental Adaptation for Statistical Machine Translation.” Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1059–1065. Lisbon, Portugal. find
  19. Yosinski, Jason, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. “How transferable are features in deep neural networks?” Advances in neural information processing systems, 3320–3328. find
  20. Mesnil, Grégoire, Yann Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, Ian J Goodfellow, Erick Lavoie, Xavier Muller, Guillaume Desjardins, David Warde-Farley, et al. 2012. “Unsupervised and Transfer Learning Challenge: a Deep Learning Approach.” ICML Unsupervised and Transfer Learning, 27:97–11… find
  21. Sennrich, Rico, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. “Nematus: a Toolkit for Neural Machine Translation.” Proceedings of the Software Demonstrations of… our card
  22. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016a. “Edinburgh Neural Machine Translation Systems for WMT 16.” Proceedings of the First Conference on Machine Translation, 371–376. Berlin, Germany. our card
  23. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016b. “Improving Neural Machine Translation Models with Monolingual Data.” Proceedings of the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barone-2017-regularization-techniques-fine-tuning-neural/