Elamur Research⌕
☰
home / papers / 2018 / A Study of Reinforcement Learning for Neural Machine Translation

A Study of Reinforcement Learning for Neural Machine Translation

Summary

Reinforcement learning can align NMT training with sequence-level metrics such as BLEU. In practice it is unstable, and most earlier evidence came from shallow RNN models. Lijun Wu and colleagues ran a systematic study of RL for deep Transformer models on large datasets: WMT14 English-German, WMT17 English-Chinese and WMT17 Chinese-English. Using REINFORCE with a sentence-level BLEU reward, they compared:

  • beam search vs. multinomial sampling for generating training candidates,
  • reward shaping and baseline rewards,
  • different weightings that mix the MLE and RL objectives.

They also proposed ways to bring source-side and target-side monolingual data into RL training, both sequentially and in a unified setup. This is the first work to explore this combination.

Sampling worked better than beam search. The popular stabilization tricks had little effect, but interpolating MLE with RL was essential. Monolingual data and RL complemented each other, giving a state-of-the-art Zh-En result. The work matters because it offers a practical recipe for RL fine-tuning of strong NMT systems.

Key points

  • Multinomial sampling gave better rewards for training than beam search. Reward shaping and a baseline reward made no significant difference, while combining the MLE and RL losses was important.
  • On WMT17 Zh-En, a Transformer with source and target monolingual data plus RL reached 26.73 BLEU. This is up from 26.13 without RL and beats the WMT17 winner SougouKnowing-ensemble (26.40).
  • The best system improves on a Transformer + back-translation baseline (25.24) by nearly 1.5 BLEU. Plain Transformer scores 24.29 and Transformer + RL scores 25.04.
  • Adding only monolingual data during the RL stage performed roughly on par with MLE training on bilingual plus monolingual data (25.15 vs. 25.24).
  • The code and data are released at github.com/apeterswu/RL4NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1397

@inproceedings{wu2018study,
  title = {{A Study of Reinforcement Learning for Neural Machine Translation}},
  author = {Wu, Lijun and Tian, Fei and Qin, Tao and Lai, Jianhuang and Liu, Tie-Yan},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1397},
  url = {https://doi.org/10.18653/v1/d18-1397}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. Fifth International Conference on Learning Representations. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. Third International Conference on Learning Representations. find our card
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine transla… find
  4. Samidh Chatterjee and Nicola Cancedda. 2010. Minimum error rate training by sampling the translation lattice. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 606–615. Association for Computational Linguistics. find
  5. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. meeting of the association for computational linguistics, pages 1965–1974. find
  6. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2017. Classical struc-Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. Proceedings of the 34th international conference on machine learnin… find our card
  7. Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115–135. find
  8. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567. find our card
  9. Di He, Hanqing Lu, Yingce Xia, Tao Qin, Liwei Wang, and Tieyan Liu. 2017. Decoding with value networks for neural machine translation. In Advances in Neural Information Processing Systems, pages 178– 187. find
  10. Xiaodong He and Li Deng. 2012. Maximum expected bleu training of phrase and lexicon translation models. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 292–301. Association for Computational Linguistics. find
  11. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. Thirthy-Second AAAI Conference On Artificial Intelligence. find
  12. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Third International Conference on Learning Representations. find
  13. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  14. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. Fifth International Conference on Learning Representations. find our card
  15. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. Advances in neural information processing systems, workshop. find
  16. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning, pages 807–814. find
  17. Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278–287. find
  18. Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  20. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. Fourth International Conference on Learning Representations. find our card
  21. FW Scholz. 1985. Maximum likelihood estimation. Encyclopedia of statistical sciences. find
  22. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The university of edinburgh’s neural mt systems for wmt17. In Proceedings of the Second Conference on Machine Translation, pages 389–399. Association… find our card
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find
  25. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. meeting of the association for computational linguistics, pages 1683–1692. find our card
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  28. Yijun Wang, Yingce Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and T Liu. 2018. Dual transfer learning for neural machine translation with marginal distribution regularization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. find
  29. Lex Weaver and Nigel Tao. 2001. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pages 538–545. Morgan Kaufmann Publishers Inc. find
  30. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Reinforcement Learning, pages 5–32. Springer. find
  31. Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018a. Beyond error propagation in neural machine translation: Characteristics of language also matter. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  32. Lijun Wu, Fei Tian, Li Zhao, Jianhuang Lai, and Tie-Yan Liu. 2018b. Word attention for sequence to sequence text understanding. In AAAI. find
  33. Lijun Wu, Yingce Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2017a. Adversarial neural machine translation. arXiv preprint arXiv:1704.06933. — [arXiv:1704.06933](https://arxiv.org/abs/1704.06933) find
  34. Lijun Wu, Li Zhao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2017b. Sequence prediction with unlabeled data by reward function learning. IJCAI-17, pages 3098–3104. find
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  36. Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Weiying Ma. 2016. Dual learning for machine translation. neural information processing systems, pages 820–828. find
  37. Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In Advances in Neural Information Processing Systems, pages 1784–1794. find
  38. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2018-study-reinforcement-learning-neural-machine/