Elamur Research⌕
☰
home / papers / 2017 / An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation

An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation

Summary

The paper addresses neural machine translation (NMT) for domains with little parallel data. In these settings plain NMT often loses to phrase-based SMT, and standard fine tuning on a small in-domain corpus quickly overfits. The authors propose "mixed fine tuning", which combines fine tuning with tag-based multi domain NMT. A model is first trained on out-of-domain data. Training then continues on a blend of out-of-domain data and oversampled in-domain data, with every source sentence marked by a domain tag.

They compare this method against fine tuning, multi domain NMT, and stacked variants in two settings:

  • Chinese→English: NTCIR patents adapted to IWSLT TED talks, where the in-domain corpus is clean and manually built.
  • Chinese→Japanese: ASPEC scientific text adapted to WIKI-CJ, where the in-domain corpus is noisy and automatically extracted.

Mixed fine tuning gives the best in-domain BLEU on IWSLT and ties with fine tuning on WIKI-CJ. Unlike fine tuning, it preserves out-of-domain quality. The work matters because it is a simple, architecture-free recipe, and it is an early systematic comparison of NMT domain adaptation methods.

Key points

  • Data:
    • NTCIR-CE: about 1M training pairs.
    • IWSLT-CE: 209,491 pairs; tested on the 2010–2013 test sets.
    • ASPEC-CJ: 672,315 pairs.
    • WIKI-CJ: 136,013 pairs, automatically mined from Wikipedia using ASPEC-CJ as a seed.
  • Systems: KyotoNMT with 2-layer LSTMs and a 32k vocabulary. BPE used 30k merges, with a joint BPE model for Chinese–Japanese. The baseline was Moses PBSMT.
  • IWSLT-CE results (average BLEU):
    • Mixed fine tuning scored 18.01.
    • Fine tuning scored 16.41 and multi domain 16.34.
    • Plain NMT scored 7.87, below SMT at 14.31.
  • WIKI-CJ results:
    • Fine tuning (37.66), mixed fine tuning (37.57), and mixed fine tuning + fine tuning (37.77) were close.
    • Multi domain trailed at 35.79.
  • Out-of-domain retention:
    • Fine tuning collapsed out-of-domain quality, with NTCIR-CE falling from 37.11 to 17.37 and ASPEC-CJ from 42.92 to 22.10.
    • Mixed fine tuning kept out-of-domain quality at 37.01 and 42.56 respectively.
    • Adding a final in-domain fine-tuning stage gave no in-domain gain.

Interesting ideas

  • I1. Keep the source data during adaptation. Continuing training on in-domain data mixed with the original corpus acts as a regularizer against overfitting and forgetting. This transfers to any low-resource adaptation, such as adapting a general Ukrainian model to a niche domain.
  • I2. Use domain tags as cheap conditioning. A pseudo-token like "<2domain>" on the source side lets one model serve several domains without architecture changes. Tags helped both multi domain and mixed fine tuning in these experiments.
  • I3. Oversample the small corpus. Oversampling the minority domain forces training to give each domain comparable weight. The authors also note a trade-off: without tags, the larger corpus can benefit more from the shared data.
  • I4. Measure forgetting explicitly. The authors evaluate on the out-of-domain test set as well as the target domain. This makes catastrophic forgetting visible, and it is a useful default check for any adaptation experiment.
  • I5. Treat in-domain data quality as a variable. Using one clean and one noisy, automatically extracted in-domain corpus showed that the benefit of mixing depends on data quality. This is relevant whenever parallel data is scraped or mined rather than curated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 99 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2061

@inproceedings{chu2017empirical,
  title = {{An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation}},
  author = {Chu, Chenhui and Dabre, Raj and Kurohashi, Sadao},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2061},
  url = {https://doi.org/10.18653/v1/p17-2061}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M Cettolo, J Niehues, S Stüker, L Bentivogli, R Cattoni, and M Federico. 2015. The iwslt 2015 evaluation campaign. In Proceedings of the Twelfth International Workshop on Spoken Language Translation (IWSLT). find
  2. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP 2004. Association for Computational Linguistics, Barcelona, Spain, pages 388–395. find
  3. Kyunghyun Cho, Bart van Merriënboer, Çalar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan… find
  4. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  5. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2016a. Parallel sentence extraction from comparable corpora with neural network features. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), Paris, France. find
  6. Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements of Japanese morphological analyzer JUMAN. In Proceedings of the International Workshop on Sharable Natural Language. pages 22–28. find
  7. Chenhui Chu, Toshiaki Nakazawa, Daisuke Kawahara, and Sadao Kurohashi. 2016b. SCTB: A Chinese treebank in scientific domain. In Proceedings of the 12th Workshop on Asian Language Resources (ALR12). find
  8. The COLING 2016 Organizing Committee, Osaka, Japan, pages 59–67. http://aclweb.org/anthology/W16-5407. ↗
  9. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the 12th International Workshop on Spoken Language Translation. Da Nang, Vietnam, pages 76–79. find
  10. Fabien Cromieres, Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2016. Kyoto university participation to wat 2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016). find
  11. The COLING 2016 Organizing Committee, Osaka, Japan, pages 166–174. http://aclweb.org/anthology/W16-4616. ↗
  12. Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How transferable are neural networks in nlp applications? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 479–489. http… find
  13. Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. arXiv preprint arXiv:1612.06897 . — [arXiv:1612.06897](https://arxiv.org/abs/1612.06897) find
  14. Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita. 2015. Overview of the 2nd Workshop on Asian Translation. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015). Kyoto, Japan, pages 1–28. find
  15. Isao Goto, Ka-Po Chow, Bin Lu, Eiichiro Sumita, and Benjamin K. Tsou. 2013. Overview of the patent machine translation task at the ntcir-10 workshop. In Proceedings of the 10th NTCIR Conference. National Institute of Informatics (NII), Tokyo, Japan, pages 260–286. find
  16. Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. Aspec: Asian scientific paper excerpt corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Res… find our card
  17. Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loı̈c Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. CoRR abs/1503.03535. http://arxiv.org/abs/1503.03535. ↗ our card
  18. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. CoRR abs/1611.04558… find
  19. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sapporo, Japan, pages 160– 167. http://www.aclweb.org/anthology/P03-1021. ↗
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the Catherine Kobus, Josep Crego, and Jean Senellart. 2016. Domain control for neural machine translation. arXiv preprint arXiv… find our card
  21. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 35–40. http://www.aclweb.org/anthology/N16-1005. ↗ our card
  22. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany,… find
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages… find
  24. Christophe Servan, Josep Crego, and Jean Senellart. 2016. Domain specialization: a post-training domain adaptation for neural machine translation. arXiv preprint arXiv:1612.06141 . — [arXiv:1612.06141](https://arxiv.org/abs/1612.06141) find
  25. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems. MIT Press, Cambridge, MA, USA, NIPS’14, pages 3104–3112. http://dl.acm.org/citation.cfm?id=2969033.29… ↗
  26. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 1568–1575. http://aclweb… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chu-2017-empirical-comparison-domain-adaptation-methods/