Elamur Research⌕
☰
home / papers / 2019 / Bridging the Gap between Training and Inference for Neural Machine Translation

Bridging the Gap between Training and Inference for Neural Machine Translation

Summary

Neural MT decoders learn to predict each target word from the reference prefix, but at test time they must condition on their own earlier outputs. This mismatch, known as exposure bias, lets errors compound. Word-level cross-entropy also penalizes valid alternative wordings and pulls the model back toward the exact reference, a problem the authors call overcorrection.

The proposed fix, OR-NMT, sometimes replaces the gold previous word with an "oracle" word chosen from the model's own predictions during training. Oracles come from two sources:

  • Word-level: greedy selection, optionally perturbed with Gumbel-Max noise.
  • Sentence-level: beam-search candidates reranked by BLEU, with force decoding so the candidate length matches the reference.

The chance of using the gold word decays over training, so supervision gradually loosens.

The method was tested with RNNsearch and Transformer on Chinese→English and WMT'14 English→German. It consistently outperformed the baselines and the related methods scheduled sampling and MIXER. Sentence-level oracles worked better than word-level ones. The approach matters because it is model-agnostic and directly targets a basic weakness of autoregressive training.

Key points

  • On WMT'14 En→De (newstest2014), OR-NMT improved RNNsearch from 25.82 to 27.41 BLEU (+1.59) and Transformer (base) from 27.34 to 28.65 (+1.31). In both cases it beat SS-NMT and MIXER.
  • The best Zh→En model used sentence-level oracles with Gumbel noise (τ = 0.5). Without noise, training converged faster but reached lower BLEU, which suggests the noise acts as a regularizer against overfitting.
  • In an exposure-bias check on 1K Zh→En training pairs, the model scored +2.18 BLEU over RNNsearch (24.87). It also gave higher probability to 65.06% of the gold reference words (18,391 of 28,266).
  • On the MT03 test set, gains appeared across all source-length bins and were especially large for very long sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 88 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1426 · arXiv 1906.02448

@inproceedings{zhang2019bridging,
  title = {{Bridging the Gap between Training and Inference for Neural Machine Translation}},
  author = {Zhang, Wen and Feng, Yang and Meng, Fandong and You, Di and Liu, Qun},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1426},
  eprint = {1906.02448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1426}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. find our card
  2. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1171–1179. Curran… find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. find
  4. Learning Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 531–540, Ann Arbor, Michigan. Association for Computational Linguist… find our card
  5. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243–1252, International Convention… find our card
  6. Emil Julius Gumbel. 1954. Statistical theory of extreme valuse and some practical applications. Nat. Bur. Standards Appl. Math. Ser. 33. find
  7. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  8. Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3086–3094. Curran Associates, Inc. find
  9. Fandong Meng and Jinchao Zhang. 2019. Dtmt: A novel deep transition architecture for neural machine translation. Proceedings of AAAI. find
  10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  11. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732. find our card
  12. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  13. Chenze Shao, Xilin Chen, and Yang Feng. 2018. Greedy search with probabilistic n-gram matching for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4778–4784. find
  14. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1683–1692. find our card
  15. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. find
  16. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of ACL. find our card
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  18. Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell. 2015. Improving multi-step prediction of learned time series models. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, pages 3024–3030. AAAI Press. find
  19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  20. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  21. Jinchao Zhang, Mingxuan Wang, Qun Liu, and Jie Zhou. 2017. Incorporating word reordering knowledge into attention-based neural machine translation. In Proceedings of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/