Elamur Research⌕
☰
home / papers / 2020 / Domain Robustness in Neural Machine Translation

Domain Robustness in Neural Machine Translation

Summary

The paper asks whether modern neural machine translation still breaks down on text from domains it never saw in training, and why. The authors train Transformer models (Sockeye) and phrase-based Moses systems on a single domain and test them on unseen ones. The setups are German→English, trained on medical OPUS data and evaluated on IT, koran, law and subtitles, and low-resource German→Romansh, trained on law and press releases and evaluated on blogs.

Out-of-domain BLEU collapses for both system types, and their average scores end up nearly equal. The failures differ, though. SMT output stays faithful to the source but reads poorly. NMT output reads well but often has nothing to do with the input. Manual evaluation shows these hallucinations become much more frequent outside the training domain.

The authors then compare reconstruction, subword regularization, noisy channel reranking, defensive distillation and their combinations. These give moderate BLEU gains but improve adequacy only slightly. The work matters because it shows that BLEU alone can conceal very different error profiles. It also releases baselines for studying robustness when no domain adaptation is possible.

Key points

  • Data:
    • DE→EN uses the five OPUS domains from Koehn and Knowles (2017), with EMEA medical data (1.1m pairs) as the training domain.
    • DE→RM trains on about 100k pairs (the Allegra corpus plus in-house press releases) and tests on Convivenza blog posts.
  • DE→EN baselines:
    • In-domain, NMT beats SMT (61.5 vs 58.4 BLEU).
    • Out-of-domain, both average only about 11.7–11.8 BLEU.
    • On IT, SMT is 4.3 BLEU ahead of NMT.
  • OOV words are not the main cause: the koran and subtitles test sets score far lower than IT despite comparable OOV rates.
  • Reconstruction:
    • It lowers the average share of hallucinated out-of-domain outputs from 35% to 29%, at some cost to fluency.
    • Only reconstruction combined with noisy channel reranking improved BLEU consistently across all domains and both language pairs.
  • Overall gains: the best systems improve average out-of-domain BLEU by 1.5 (DE→EN) and 2.7 (DE→RM). Code and data are available at github.com/ZurichNLP/domain-robustness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 61 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{muller2020domain,
  title = {{Domain Robustness in Neural Machine Translation}},
  author = {Müller, Mathias and Rios, Annette and Sennrich, Rico},
  booktitle = {Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)},
  year = {2020},
  url = {https://aclanthology.org/2020.amta-research.14/}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arthur, P., Neubig, G., and Nakamura, S. (2016). Incorporating discrete translation lexicons into neural machine translation. In Proceedings of EMNLP 2016, pages 1557–1567. find
  2. Ba, J. and Caruana, R. (2014). Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654–2662. find
  3. Belinkov, Y. and Bisk, Y. (2018). Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada. find
  4. Goodfellow, I., Shlens, J., and Szegedy, C. (2015). Explaining and harnessing adversarial examples. In International Conference on Learning Representations. find our card
  5. Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572. find our card
  6. Hieber, F., Domhan, T., Denkowski, M., Vilar, D., Sokolov, A., Clifton, A., and Post, M. (2018). The Sockeye Neural Machine Translation Toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (AMTA), pages 200–207. find
  7. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  8. Jang, E., Gu, S., and Poole, B. (2017). Categorical reparametrization with gumbel-softmax. In Proceedings International Conference on Learning Representations 2017. OpenReviews.net. find
  9. Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J. (2017). Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Lingui… find
  10. Kim, Y. and Rush, A. M. (2016). Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. find our card
  11. Kobus, C., Crego, J., and Senellart, J. (2017). Domain control for neural machine translation. In Proceedings of RANLP 2017, pages 372–378. INCOMA Ltd. find
  12. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007 Demo and Poster Sessio… find
  13. Koehn, P. and Knowles, R. (2017). Six Challenges for Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. find
  14. Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of ACL 2018, pages 66–75. find
  15. Läubli, S., Müller, M., Horat, B., and Volk, M. (2018). mtrain: A convenience tool for machine translation. find
  16. Lee, K., Firat, O., Agarwal, A., Fannjiang, C., and Sussillo, D. (2018). Hallucinations in neural machine translation. find
  17. Li, J. and Jurafsky, D. (2016). Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372. — [arXiv:1601.00372](https://arxiv.org/abs/1601.00372) find
  18. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Li, X., Michel, P., Anastasopoulos, A., Belinkov, Y., Durrani, N., Firat, O., Koehn, P., Neubig, G., Pino, J., and Sajjad, H. (2019). Findings of the First S… find
  19. Lison, P. and Tiedemann, J. (2016). OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of LREC 2016, pages 923–929. find our card
  20. Luong, M.-T. and Manning, C. D. (2015). Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of IWSLT 2015, Da Nang, Vietnam. find
  21. Nguyen, T. and Chiang, D. (2018). Improving Lexical Choice in Neural Machine Translation. In Proceedings of NAACL-HLT 2018, pages 334–343. find
  22. Niu, X., Xu, W., and Carpuat, M. (2019). Bi-directional differentiable input reconstruction for low-resource neural machine translation. In Proceedings of NAACL-HLT 2019, pages 442–448. find
  23. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  24. Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016). Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582–597. IEEE. find
  25. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, PA. find our card
  26. Post, M. (2018). A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. find our card
  27. Scherrer, Y. and Cartoni, B. (2012). The Trilingual ALLEGRA Corpus: Presentation and Possible Use for Lexicon Induction. In Proceedings of LREC-2012, pages 2890–2896, Istanbul, Turkey. find
  28. Sennrich, R., Haddow, B., and Birch, A. (2016a). Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of ACL 2016, pages 86–96, Berlin, Germany. find
  29. Sennrich, R., Haddow, B., and Birch, A. (2016b). Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL 2016, pages 1715–1725, Berlin, Germany. find
  30. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. find our card
  31. Tu, Z., Liu, Y., Shang, L., Liu, X., and Li, H. (2017). Neural Machine Translation with Reconstruction. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3097–3103. find
  32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. find
  33. Yee, K., Dauphin, Y., and Auli, M. (2019). Simple and effective noisy channel modeling for neural machine translation. In Proceedings of EMNLP-IJCNLP 2019, pages 5696–5701, Hong Kong, China. find
  34. Yu, L., Blunsom, P., Dyer, C., Grefenstette, E., and Kociský, T. (2017). The neural noisy channel. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. find
  35. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/muller-2020-domain-robustness-neural-machine-translation/