Elamur Research⌕
☰
home / papers / 2019 / Curriculum Learning for Domain Adaptation in Neural Machine Translation

Curriculum Learning for Domain Adaptation in Neural Machine Translation

Paper · 2019 · Xuan Zhang, Pamela Shapiro, Gaurav Kumar et al.

Summary

Neural MT systems trained on broad, mixed-domain data tend to underperform on specialised domains where parallel text is scarce. A standard remedy is continued training: a generic model is fine-tuned on in-domain bitext, often enlarged with web-crawled pairs chosen by data selection. The difficulty is that any similarity cut-off is arbitrary, and the selected pool still mixes close and distant sentences. The authors instead use selection scores to set a training schedule. Paracrawl sentences are ranked with Moore-Lewis or cynical data selection, grouped into shards, and added to continued training step by step, starting with the in-domain data and the most similar shards. Within each phase, ordering stays random through shuffling, bucketing and mini-batching. For both German-English and Russian-English, on patent abstracts as well as TED talks, this curriculum consistently beats standard continued training on the same data, by up to 3.22 BLEU, and makes better use of distant, noisy webcrawl. Because it works with any NMT architecture and is easy to add, it suits low-resource domain adaptation.

Key points

  • Data: Generic models were trained on OpenSubtitles2018 + WMT 2017 (about 28M de-en and 51M ru-en pairs after length filtering). In-domain data was 15k sampled pairs each from TED (Duh, 2018) and WIPO COPPA-V2 patents. Paracrawl was cleaned with Zipporah (13.6M de, 3.7M ru), and up to 4096k/2048k pairs were selected.
  • Setup: The models were built in Sockeye as a 2-layer LSTM attention seq2seq with 512 units and 30k BPE. Similarity scores came from 5-gram source-side KenLM models. The curriculum used 40 shards (the first holds all in-domain data) and 1000 batches per phase.
  • Results: On TED(de), CL_CDS reached 38.88 BLEU vs 35.83 for std_CDS and 34.59 for GEN. On patent(de), CL_CDS reached 55.51 vs 52.58 for std_CDS.
  • Selection method and errors: CDS worked better for the distant patent domain, while ML was slightly ahead on TED. S4 analysis showed fewer SENSE and SCORE lexical errors.
  • Code: The code is released in sockeye-recipes (egs/curriculum).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat and Kevin Duh. 2019. Curriculum Learning for Domain Adaptation in Neural Machine Translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1189

@inproceedings{zhang2019curriculum,
  title = {{Curriculum Learning for Domain Adaptation in Neural Machine Translation}},
  author = {Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat and Kevin Duh},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1189},
  url = {https://doi.org/10.18653/v1/n19-1189}
}

References

45 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod. 2017. Cynical selection of language model training data. arXiv preprint arXiv: 1709.02279. — [arXiv:1709.02279](https://arxiv.org/abs/1709.02279) find
  2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the conference on empirical methods in natural language processing. our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. our card
  4. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. our card
  5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation. our card
  6. Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation. find
  7. Boxing Chen and Fei Huang. 2016. Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. find
  8. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. our card
  9. Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. arXiv preprint arXiv:1806.00258. our card
  10. Kevin Duh. 2018. The multitarget ted talks task. http://www.cs.jhu.edu/˜kevinduh/a/ multitarget-tedtalks/. ↗
  11. Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
  12. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 find
  13. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. George Foster, Cyril Goutte, and Roland Kuhn. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the 2010 conference on e… our card
  14. Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. arXiv preprint arXiv:1612.06897. find
  15. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2014. An empirical investigation of catastrophic forgetting in gradient-based neural networks. In Proceedings of the 2nd International Conference on Learning Representations. find
  16. Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning. find
  17. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. our card
  18. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690. find
  19. Ann Irvine, John Morgan, Marine Carpuat, Hal Daumé III, and Dragos Munteanu. 2013. Measuring machine translation errors in new domains. Transactions of the Association for Computational Linguistics. our card
  20. Marcin. Junczys-Dowmunt, Bruno. Pouliquen, and Christophe. Mazenc. 2016. Coppa v 2 . 0 : Corpus of parallel patent applications building large parallel corpora with gnu make. find
  21. Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation. find
  22. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations. find
  23. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. find
  24. Tom Kocmi and Ondřej Bojar. 2017. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing. find
  25. Philipp Koehn, Kevin Duh, and Brian Thompson. 2018. The jhu machine translation systems for wmt 2018. In Proceedings of the Third Conference on Machine Translation. find
  26. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  27. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. find
  28. Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019. Reinforcement learning based curriculum optimization for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V… find
  29. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. our card
  30. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. our card
  31. Saab Mansour and Hermann Ney. 2012. A simple and effective weighted phrase extraction for machine translation adaptation. In International Workshop on Spoken Language Translation. find
  32. Spyros Matsoukas, Antti-Veikko I Rosti, and Bing Zhang. 2009. Discriminative corpus weight estimation for machine translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. our card
  33. Robert C Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers. our card
  34. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M. Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  35. Lucı́a Santamarı́a and Amittai Axelrod. 2017. Data selection with cluster-based language difference models and cynical selection. In International Workshop on Spoken Language Translation. find
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. our card
  37. Kashif Shah, Loı̈c Barrault, and Holger Schwenk. 2010. Translation model adaptation by resampling. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. find
  38. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… our card
  39. Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. our card
  40. Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference on Machine Translation: Research Papers. our card
  41. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. our card
  42. Daphna Weinshall and Gad Cohen. 2018. Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of the 35th International Conference on Machine Learning. find
  43. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find
  44. Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An empirical exploration of curriculum learning for neural machine translation. arXiv preprint arXiv:1811.00739. — [arXiv:1811.00739](https://arxiv.org/abs/1811.00739) find
  45. Xinpeng Zhou, Hailong Cao, and Tiejun Zhao. 2015. Domain adaptation for smt using sentence weight. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-curriculum-learning-domain-adaptation-neural/