Elamur Research⌕
☰
home / papers / 2019 / Robust Neural Machine Translation with Doubly Adversarial Inputs

Robust Neural Machine Translation with Doubly Adversarial Inputs

Summary

NMT systems can produce much worse output when the input changes only slightly. In one example, swapping "he" for "she" in a Chinese sentence derailed a Transformer translation. The authors address this weakness with a white-box adversarial training scheme for word-level Transformer models.

Their gradient-guided procedure, AdvGen, replaces a sampled fraction of words with substitutes. It favours words whose embedding shift best aligns with the gradient of the translation loss. Candidate substitutes come from a small set proposed by a bidirectional language model, which keeps the search cheap and the replacements plausible.

The method works in two stages:

  • Attack: perturbed source sentences attack the encoder.
  • Defence: adversarially altered decoder inputs, chosen using a mix of language-model and NMT predictions, train the model to tolerate errors caused by the noisy source.

Training uses only parallel data. On Chinese-English and English-German, the approach beats Transformer on clean benchmarks and degrades less on noisy input. This shows that adversarial training can improve both accuracy and robustness.

Key points

  • Reported gains over Transformer are 2.8 BLEU on Chinese-English and 1.6 BLEU on English-German, measured on standard clean test sets.
  • In the Chinese-English ablation, the full model (clean loss, robust loss and language-model loss) reaches 46.95 BLEU, compared with 44.59 for the clean-loss-only baseline. Removing the adversarial target inputs causes the largest drop, 1.87 BLEU.
  • Replacement ratios have only a mild effect on results. The best validation setting replaces 25% of source words (γ_src = 0.25) and 50% of target inputs (γ_trg = 0.50). Any non-zero γ_trg helped, while larger source ratios hurt.
  • Using only parallel corpora, the method outperforms back-translation with extra monolingual data (Sennrich et al., 2016b). Adding monolingual data improves it further.
  • The work differs from earlier robustness efforts that relied on black-box or synthetic noise and from the white-box attacks of Ebrahimi et al. (2018a) on character-level models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1425

@inproceedings{cheng2019robust,
  title = {{Robust Neural Machine Translation with Doubly Adversarial Inputs}},
  author = {Cheng, Yong and Jiang, Lu and Macherey, Wolfgang},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1425},
  url = {https://doi.org/10.18653/v1/p19-1425}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Empirical Methods in Natural Language Processing. find our card
  2. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  3. Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. arXiv preprint arXiv:1808.07561. find
  4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. find
  5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Association for Computational Linguistics. find our card
  6. Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Association for Computational Linguistics. find
  7. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Association for Computational Linguistics. find
  8. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018a. On adversarial examples for character-level neural machine translation. In Proceedings of COLING. find
  9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018b. Hotflip: White-box adversarial examples for text classification. In Association for Computational Linguistics. find our card
  10. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Empirical Methods in Natural Language Processing. find
  11. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Association for Computational Linguistics. find our card
  12. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning. find our card
  13. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations. find our card
  14. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. find
  15. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690. — [arXiv:1712.05690](https://arxiv.org/abs/1712.05690) find
  16. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing. find our card
  17. Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. 2015. Self-paced curriculum learning. In AAAI Conference on Artificial Intelligence. find
  18. Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. find
  19. Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. arXiv preprint arXiv:1902.01509. find
  20. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Empirical Methods in Natural Language Processing. find our card
  21. Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, and Zhongjun He. 2018. Robust neural machine translation with joint textual and phonetic embedding. arXiv preprint arXiv:1810.06729. find
  22. Paul Michel and Graham Neubig. 2018. Mtnt: A testbed for machine translation of noisy text. arXiv preprint arXiv:1809.00388. find our card
  23. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations. find our card
  24. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition. find
  25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Association for Computational Linguistics. find
  26. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation. find
  27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. find
  28. Christian Szegedy, Wojciech Zaremba, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Machine Learning. find our card
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  30. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. In Empirical Methods in Natural Language Processing. find our card
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a methof for automatic evaluation of machine translation. In Association for Computational Linguistics. find
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. find our card
  34. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving nerual machine translation models with monolingual data. In Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/