Elamur Research⌕
☰
home / papers / 2017 / Multi-Domain Neural Machine Translation through Unsupervised Adaptation

Multi-Domain Neural Machine Translation through Unsupervised Adaptation

Summary

The paper addresses a practical deployment setting for neural MT. Sentences arrive from many domains in arbitrary order and carry no domain label, and a single generic model must handle all of them instead of a set of specialised engines. The authors work with English–French and build on the attention-based GRU encoder–decoder of Bahdanau et al. (2014). They adapt this model separately for each input sentence:

  1. Source sentences similar to the input are retrieved from the whole pool of parallel training data.
  2. The generic model is briefly fine-tuned on the retrieved pairs.
  3. The input is translated with the fine-tuned model.
  4. The weights are reset to the generic model before the next sentence.

The main novelty compared with the earlier instance-based approach of Li et al. (2016) is that the learning rate and the number of epochs are not fixed. They are set on the fly according to how close the best retrieved pair is to the input. On multi-domain test data, this approach beats the generic NMT model and a strong phrase-based system. It also beats domain-specific NMT oracles, even though those oracles need domain labels. This suggests that one adaptable model can replace costly multi-engine infrastructures.

Key points

  • On the test set, the adaptive system (Adaptive Dynamic Lrate-Epochs) improves BLEU by +14.5 over generic NMT, +3.9 over PBMT and +2.8 over the pool of domain-specific NMT oracles.
  • On the dev set, choosing the number of epochs dynamically adds +3.9 BLEU on top of a dynamic learning rate alone. The gains are largest in highly similar domains, such as +12.2 for Gnome. Allowing up to 9 epochs helped, while more updates caused overfitting.
  • The performance gain correlates with retrieval similarity at about 0.9. A TER analysis shows lexical errors dropping by 14%, and output lengths move closer to the reference length of 15.3 words.
  • Domains in the data include ECB, KDE4, JRC and Gnome. The baseline comparison uses the authors' reimplementation of Li et al. (2016).
  • In the error analysis, failures come from retrieved pairs that look alike on the surface but differ in meaning (KDE4 "collapse"), and from inconsistent reference translations (JRC).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

61 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 56 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4713

@inproceedings{farajian2017multi,
  title = {{Multi-Domain Neural Machine Translation through Unsupervised Adaptation}},
  author = {Farajian, M. Amin and Turchi, Marco and Negri, Matteo and Federico, Marcello},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4713},
  url = {https://doi.org/10.18653/v1/w17-4713}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078. http://arxiv.org/abs/1406.1078. ↗
  2. ACL on Interactive Poster and Demonstration Sessions. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL ’07, pages 177–180. http://dl.acm.org/citation.cfm?id=1557769.1557821. ↗
  3. Lee Raymond Dice. 1945. Measures of the amount of ecologic association between species. Ecology 26(3):297–302. http://www.jstor.org/pss/1932409. ↗
  4. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 224–227. http://www.aclweb.org/anthology/W/… ↗ our card
  5. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research . find
  6. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. One sentence one model for neural machine translation. CoRR abs/1609.06490. http://arxiv.org/abs/1609.06490. ↗
  7. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In In Proc. NAACL. find
  8. Matthias Eck, Stephan Vogel, and Alex Waibel. 2004. Language model adaptation for statistical machine translation based on information retrieval. In In Proceedings of the 4th International Conference on language resources and evaluation (LREC-2004. pages 327–330. find
  9. Minh-Thang Luong and Christopher D Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of the International Workshop on Spoken Language Translation. find
  10. M. McCandless, E. Hatcher, and O. Gospod-netić. 2010. Lucene in Action. Manning Pubs Co Series. Manning. https://books.google.it/books?id=XrJBPgAACAAJ. ↗
  11. George Foster and Roland Kuhn. 2007. Mixture-model adaptation for SMT. In Proceedings of the Second Workshop on Statistical Machine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 128–135. http://www.aclweb.org/anthology/W/W07/W07-0217. ↗ our card
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages… find our card
  13. Philip Gage. 1994. A New Algorithm for Data Compression. C Users J. 12(2):23–38. http://dl.acm.org/citation.cfm?id=177910.177914. ↗
  14. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org. ↗
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics. find
  16. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Sofi… find our card
  17. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of the 20th International Conference on Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA, USA, COLING ’04.… — [doi:10.3115/1220355.1220414](https://doi.org/10.3115/1220355.1220414) find
  18. Almut Silja Hildebrand, Mattias Eck, Stephan Vogel, and Alex Waibel. 2005. Adaptation of the translation model for statistical machine translation based on information retrieval. In Proceedings of the 10th annual conference of the European association for machine translation. Budapest, Hungary, page… find our card
  19. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/farajian-2017-multi-domain-neural-machine-translation/