Elamur Research⌕
☰
home / papers / 2018 / Towards Robust Neural Machine Translation

Towards Robust Neural Machine Translation

Paper · 2018 · Yong Cheng, Zhaopeng Tu, Fandong Meng et al.

Summary

Neural machine translation systems are fragile: replacing one source word with a synonym can drastically change the output. The authors tested this by perturbing one word per sentence in a test set and found that most translations shifted. Their remedy is adversarial stability training, an architecture-independent training scheme that pushes both encoder and decoder to treat a sentence and a slightly noised version of it the same way. Noisy inputs are built in two ways. The lexical method swaps sampled words for vocabulary items with similar embeddings. The feature method adds Gaussian noise to word embeddings. The objective combines three losses: the usual translation loss, an adversarial invariance loss in which a discriminator tries to separate encoder states of clean and perturbed inputs, and a loss for translating the perturbed source into the original target. Experiments covered Chinese-English, English-German and English-French. The method beat strong baselines, outperformed the RNN-based GNMT, and roughly matched the CNN-based system of Gehring et al. (2017). This matters for noisy real-world input, such as ASR output in simultaneous translation or text with typos.

Key points

  • Baseline sensitivity: after replacing a single word with a synonym, 69.74% of test translations changed. BLEU between translations of original and perturbed inputs was only 79.01.
  • Two perturbation variants are proposed:
    • AST lexical samples replacements by embedding cosine similarity.
    • AST feature injects Gaussian noise into embeddings.
    • The framework also accepts task-specific noise, such as homonyms for speech-based pipelines.
  • Ablation on Chinese-English (AST lexical):
    • Dropping the adversarial loss cost 0.64 BLEU.
    • Dropping L_noisy cost 1.66 BLEU.
    • Using L_noisy alone still gave a 0.88 BLEU gain over the baseline.
  • L_inv settles near 0.68 after about 100K iterations. This means the discriminator outputs about 0.5 for both clean and perturbed samples, so the encoder treats them almost identically.
  • Training is less stable than MLE. BLEU first drops by up to about 7 points before recovering, and the loss curves oscillate. The method needs no extra monolingual or bilingual corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢TencentShenzhen · China66 works · 17,945 citations🔬State Key Laboratory of Intelligent Technology and SystemsBeijing · China21 works · 9,641 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations

Also on the byline: Beijing Advanced Innovation Center for Language Resources.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai and Yang Liu. 2018. Towards Robust Neural Machine Translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1163

@inproceedings{cheng2018robust,
  title = {{Towards Robust Neural Machine Translation}},
  author = {Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai and Yang Liu},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1163},
  url = {https://doi.org/10.18653/v1/p18-1163}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of ICLR. our card
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. our card
  3. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. our card
  4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR. find
  5. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. In Proceedings of ACL. find
  6. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2017. Neural machine translation in linear time. In Proceedings of ICML. find
  7. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. our card
  8. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  9. Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Proceedings of NIPS. find
  10. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR. our card
  11. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Proceedings of ACL. our card
  12. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of EMNLP. our card
  13. Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, and Wei Xu. 2017. Joint training for pivot-based neural machine translation. In Proceedings of IJCAI. find
  14. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP. our card
  15. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of EMNLP. find
  16. Aleksander Madry, Makelov Aleksandar, Schmidt Ludwig, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of ICLR. our card
  17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of ICML. our card
  18. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of NIPS. find
  19. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of ICLR. our card
  20. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016a. Dual learning for machine translation. In Proceedings of NIPS. find
  21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Deep residual learning for image recognition. In Proceedings of CVPR. find
  22. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  23. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In Proceedings of ICLR. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a methof for automatic evaluation of machine translation. In Proceedings of ACL. find
  25. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434. our card
  26. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Proceedings of NIPS. our card
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving nerual machine translation models with monolingual data. In Proceedings of ACL. find
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of ACL. our card
  29. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of ACL. our card
  30. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceddings of NIPS. find
  31. Christian Szegedy, Wojciech Zaremba, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of ICML. our card
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. our card
  33. Mingxuan Wang, Zhengdong Lu, Jie Zhou, and Qun Liu. 2017. Deep neural machine translation with linear associative unit. In Proceedings of ACL. find
  34. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. arXiv preprint Z. Yang, W. Chen, F. Wang, and B. Xu. 2018. Improvi… find
  35. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of EMNLP. our card
  36. Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. 2018. Asynchronous Bidirectional Decoding for Neural Machine Translation. In Proeedings of AAAI. find
  37. Hao Zheng, Yong Cheng, and Yang Liu. 2017. Maximum expected likelihood estimation for zero-resource neural machine translation. In Proceedings of IJCAI. find
  38. Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. 2016. Improving the robustness of deep neural networks via stability training. In Proceedings of CVPR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2018-robust-neural-machine-translation/