Elamur Research⌕
☰
home / papers / 2019 / Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Paper · 2019 · Brian Thompson, Jeremy Gwinnup, Huda Khayrallah et al.

Summary

When a general-domain neural machine translation model is fine-tuned on a small in-domain corpus (continued training), it improves on the target domain but translates generic text much worse. The authors treat this as catastrophic forgetting. They adapt Elastic Weight Consolidation (EWC) to NMT and give a Bayesian derivation that does not require the two translation tasks to be independent. The diagonal of the empirical Fisher matrix, computed from back-propagation gradients, measures how important each parameter is for general-domain translation. A penalty then keeps those important parameters near their general-domain values, while less important ones adapt freely to the new domain.

The experiments use six directions (En↔De, En↔Ru, En↔Zh). General-domain data come from WMT17 and OpenSubtitles18, and in-domain patent data come from WIPO COPPA-V2. Across the full trade-off between in-domain and general-domain BLEU, EWC beats the earlier MCL distillation baseline in five of six directions. At equal in-domain quality, it recovers most of the general-domain BLEU lost by plain fine-tuning. Translation still uses a single model, with no added memory or compute.

Key points

  • Setup: Systems are two-layer LSTMs (512 hidden units) built in Sockeye, with separate source/target BPE vocabularies of about 30k. Code is released at github.com/thompsonb/sockeye_ewc.
  • Data: The general-domain corpora are large (323M, 730M and 584M English words for De, Ru and Zh). The in-domain WIPO sets are small (3.2M, 0.81M and 0.80M words), and WIPO De↔En was cut to 100k lines.
  • Fisher estimation: The Fisher diagonal is estimated on 500k training sentences in under an hour on a GTX 1080 Ti. Using more data or held-out data made no meaningful difference.
  • Results without in-domain loss: EWC raises general-domain BLEU over continued training by roughly 8–19 points. For example, Zh→En goes from 9.9 to 28.7 BLEU, against 43.5 for the original model. MCL gives no gain for En→Zh or Zh→En.
  • Exception and sensitivity: The only case where MCL wins is Ru→En, where it gives a small in-domain edge at lower general-domain quality. Both learning rate and regularization strength strongly shape the outcome for either method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations

Also on the byline: United States Air Force Research Laboratory.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh and Philipp Koehn. 2019. Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1209

@inproceedings{thompson2019overcoming,
  title = {{Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation}},
  author = {Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh and Philipp Koehn},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1209},
  url = {https://doi.org/10.18653/v1/n19-1209}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine transla…
  2. Patrick Cadwell, Sharon OBrien, and Carlos SC Teixeira. 2018. Resistance and accommodation: factors for the (non-) adoption of machine translation among professional translators. Perspectives, 26(3):301–321. find
  3. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–391. Association for Computational Li…
  4. Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 36–44. Association for Computational… find
  5. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327. Association for Computational Linguistics.
  6. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  7. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, page 2016118… find
  8. Markus Freitag and Yaser Al-Onaizan. 2016. Fast Domain Adaptation for Neural Machine Translation. CoRR, abs/1612.06897. find
  9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc… find
  10. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. find
  11. Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In Proceedings of the 1st Workshop on Neural Machine Translation (and Generation) at ACL. Association for Computational Linguistics. find
  12. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690. find
  13. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Res…
  14. Praveen Dakwale and Christof Monz. 2017. Fine-tuning for neural machine translation with limited degradation across in-and out-of-domain data. Proceedings of the XVI Machine Translation Summit, page 117. find
  15. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NeurIPS Deep Learning and Representation Learning Workshop.
  16. Marcin Junczys-Dowmunt, Bruno Pouliquen, and Christophe Mazenc. 2016. COPPA V2.0: Corpus of Parallel Patent Applications building large parallel corpora with GNU make. In 4th Workshop on Challenges in the Management of Large Corpora Workshop Programme. find
  17. Opinions, interpretations, conclusions and recommendations are those of the authors and are not necessarily endorsed by the United States Government. Cleared for public release on 14 Jan 2019. Originator reference number RH-19-119318. Case number 88ABW-2019-0136. find
  18. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domain. In International Workshop on Spoken Language Translation, Da Nang, Vietnam. find
  19. David J. C. MacKay. 1992. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472. find
  20. James Martens. 2014. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193. — [arXiv:1412.1193](https://arxiv.org/abs/1412.1193) find
  21. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1489–1494, Copenhagen, Denmark. Association…
  22. Razvan Pascanu and Yoshua Bengio. 2013. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) find
  23. Victoria Porro Rodriguez, Lucia Morado Vazquez, and Pierrette Bouillon. 2017. Study on the use of machine translation and post-editing in swiss-based language service providers. Parallèles, (29 (2)). find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. Association for Computational Linguistics.
  26. Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinnup, Tim Anderson, and Philipp Koehn. 2018. Freezing subnetworks to analyze domain adaptation in neural machine translation. In Proceedings of the Third Conference on Machi… find
  27. David Vilar. 2018. Learning hidden unit contribution for adapting neural machine translation models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 500–505. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thompson-2019-overcoming-catastrophic-forgetting-during-domain/