Elamur Research⌕
☰
home / papers / 2017 / Effective Domain Mixing for Neural Machine Translation

Effective Domain Mixing for Neural Machine Translation

Summary

NMT systems are often trained on pooled data from very different sources, but simply concatenating such corpora can lower quality on each individual domain. The authors (Pryzant and Britz, equal contribution, with Le) ask how one model trained on multi-domain data can outperform domain-specific models on every component domain without knowing a sentence's domain at test time. They extend an attentional LSTM encoder-decoder in three ways. The first is a jointly trained discriminator that predicts the domain from the encoder's attention context. The second is an adversarial variant that reverses the discriminator's gradients, which pushes the encoder toward domain-invariant features. The third is a domain token that the decoder emits as its first output and that is dropped at inference. Experiments pair contrasting corpora for EN-JA, EN-ZH and EN-FR. Naive mixing hurt distant domain pairs. All three methods beat domain-specific baselines, with the discriminator the most reliable. They gave an average gain of 1.1 BLEU and state-of-the-art EN-JA results, which is useful for systems that receive input of unknown origin.

Key points

  • EN-JA data: ASPEC (about 3M scientific sentence pairs) and SubCrawl (about 3.2M subtitle pairs crawled from the web), with a shared 32,000 subword vocabulary. EN-ZH (WMT'17 news commentary vs. TED) and EN-FR (Europarl vs. OpenSubtitles) each have about 200k training pairs.
  • Domain distance was measured with a proxy A-distance computed from a bag-of-words SVM: 1.89 for ASPEC/SubCrawl, 1.73 for News/TED and 1.23 for Europarl/OpenSubs.
  • When domains are close, as in EN-FR, plain merging already helps. The degradation from naive mixing grows as domains become more distant.
  • Model setup: 4-layer bidirectional LSTM encoder with 512 units, 4-layer LSTM decoder, Bahdanau attention, Adam (lr 0.0001) and 8 K40m GPUs. EN-JA was scored with KyTea and the other pairs with Moses tokenization.
  • Unlike source-side domain tags (Kobus et al., 2016), none of the proposed methods needs domain labels at inference. The discriminator can also be removed after training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

55 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 54 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4712

@inproceedings{britz2017effective,
  title = {{Effective Domain Mixing for Neural Machine Translation}},
  author = {Britz, Denny and Le, Quoc and Pryzant, Reid},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4712},
  url = {https://doi.org/10.18653/v1/w17-4712}
}

References

40 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yaroslav Gani, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2015. Domain-adversarial training of neural networks. arxiv preprint. arXiv preprint arXiv:1505.07818 . — [arXiv:1505.07818](https://arxiv.org/abs/1505.07818) find
  2. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 . find our card
  3. Anonymous. 2017. Subcrawl: A colloquial parallel corpus for english-japanese translation. Manuscript submitted for publication. . find
  4. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. 2015. Domain-Adversarial Training of Neural Networks. ArXiv e-prints . find
  5. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 355–362. find our card
  6. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th international conference on machine learning (ICML-11). pages 513–520. find
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  8. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. pages 2672–2680. find
  9. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  10. Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. 2007. Analysis of representations for domain adaptation. Advances in neural information processing systems 19:137. find
  11. Barry Haddow and Philipp Koehn. 2012. Analysing the effect of out-of-domain data on smt systems. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 422–432. find
  12. D. Britz, A. Goldie, T. Luong, and Q. Le. 2017. Massive Exploration of Neural Machine Translation Architectures. ArXiv e-prints . find
  13. Motoko Hori. 1986. A sociolinguistic analysis of the japanese honorifics. Journal of pragmatics 10(3):373–386. find
  14. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. find
  15. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. CoRR abs/1611.04558.… find
  16. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of simple domain adaptation methods for neural machine translation. CoRR abs/1701.03214. http://arxiv.org/abs/1701.03214. — [arXiv:1701.03214](https://arxiv.org/abs/1701.03214) ↗
  17. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  18. Jonathan H Clark, Alon Lavie, and Chris Dyer. 2012. One system, many domains: Open-domain statistical machine translation via feature augmentation . find
  19. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  20. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science 14(2):179–211. find
  21. Catherine Kobus, Josep Crego, and Jean Senellart. 2016. Domain control for neural machine translation. arXiv preprint arXiv:1612.06140 . — [arXiv:1612.06140](https://arxiv.org/abs/1612.06140) find
  22. M Amin Farajian, Marco Turchi, Matteo Negri, Nicola Bertoldi, and Marcello Federico. 2017. Neural vs. phrase-based machine translation in a multi-domain scenario. find
  23. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit. volume 5, pages 79–86. find
  24. Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. CoRR abs/1612.06897. http://arxiv.org/abs/1612.06897. — [arXiv:1612.06897](https://arxiv.org/abs/1612.06897) ↗
  25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  27. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In LREC. volume 2012, pages 2214–2218. find
  28. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation. find our card
  29. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  30. Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. 2015. Learning transferable features with deep adaptation networks. In ICML. pages 97–105. find
  31. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation. find
  32. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In EMNLP. find
  33. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural ma-CoRR abs/1604.02201. chine translation. http://arxiv.org/abs/1604.02201. ↗
  34. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 . find
  35. Arindam Mandal, Dimitra Vergyri, Wen Wang, Jing Zheng, Andreas Stolcke, Gokhan Tur, D Hakkani-Tur, and Necip Fazil Ayan. 2008. Efficient data selection for machine translation. In Spoken Language Technology Workshop, 2008. find
  36. SLT 2008. IEEE. IEEE, pages 261–264. find
  37. Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. Aspec: Asian scientific paper excerpt corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016). pages 2204–2208. find our card
  38. Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computati… find
  39. Pavel Pecina, Antonio Toral, and Josef Van Genabith. 2012. Simple and effective parameter tuning for domain adaptation of statistical machine translation. In COLING. pages 2209–2224. find
  40. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 . find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/britz-2017-effective-domain-mixing-neural-machine/