Elamur Research
☰
home / papers / 2016 / An Actor-Critic Algorithm for Sequence Prediction

An Actor-Critic Algorithm for Sequence Prediction

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1607.07086

Summary

RNN sequence generators are usually trained with teacher forcing. They learn to predict each token from the gold prefix, but at test time they must continue from their own, possibly wrong, outputs. They also optimize log-likelihood rather than the evaluation metric, such as BLEU. The authors adapt reinforcement-learning actor-critic training to this supervised setting. The actor is an attention-based GRU encoder-decoder. A separate critic network estimates, for each candidate next token, the expected future task score if the actor keeps sampling. Unlike in classic RL, the critic may read the ground-truth output. The critic learns from temporal-difference targets with deep-RL tricks: delayed target networks, reward shaping and a penalty on value variance. The actor, pre-trained with log-likelihood, is then updated with critic-weighted gradients. On a synthetic spelling-correction task and IWSLT 2014 German-English translation, the method beats maximum likelihood and REINFORCE. This suggests RL-style training can directly target metrics in generation tasks like translation, captioning and dialogue.

Key points

  • The training recipe is: pre-train the actor with log-likelihood, pre-train the critic with the actor frozen, then train both jointly. An optional log-likelihood term with weight λ_LL = 0.1 can be added.
  • The data are IWSLT 2014 De-En and WMT14 En-Fr. IWSLT 2014 has validation/test sets of 6,969/6,750 sentences and vocabularies of 22,822 (English) and 32,009 (German) words. WMT14 uses 30,000-word vocabularies. Maximum sentence length is 50 for both.
  • In the ablation (greedy decoding, first 10 epochs), the full setup reaches 23.16 validation BLEU. Dropping TD learning for the critic lowers this to 16.36, and removing the variance penalty (λ = 0) gives 20.55.
  • A non-delayed target critic (γ_φ = 1) causes training to collapse to 8.14 validation BLEU, confirming that target networks matter.
  • For beam search, a length bonus ρ was tuned: 0.8 for log-likelihood/REINFORCE models and 1.0 for actor-critic models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

148 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 141 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1607.07086 · arXiv 1607.07086

@misc{bahdanau2016actor,
  title = {{An Actor-Critic Algorithm for Sequence Prediction}},
  author = {Bahdanau, Dzmitry and Brakel, Philemon and Xu, Kelvin and Goyal, Anirudh and Lowe, Ryan and Pineau, Joelle and Courville, Aaron and Bengio, Yoshua},
  year = {2016},
  doi = {10.48550/arxiv.1607.07086},
  eprint = {1607.07086},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.07086}
}

References

47 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the ICLR 2015, 2015. find our card
  2. Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. Systems, Man and Cybernetics, IEEE Transactions on, (5):834–846, 1983. find
  3. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. arXiv preprint arXiv:1506.03099, 2015. find
  4. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. In Proc. of IWSLT, 2014. find our card
  5. William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) find
  6. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
  7. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. find
  8. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, KyungHyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. CoRR, abs/1506.07503, 2015. URL http: //arxiv.org/abs/1506.07503. find our card
  9. Hal Daumé III and Daniel Marcu. Learning as search optimization: Approximate large margin methods for structured prediction. In Proceedings of the 22nd international conference on Machine learning, pp. 169–176. ACM, 2005. find
  10. Hal Daumé Iii, John Langford, and Daniel Marcu. Search-based structured prediction. Machine learning, 75(3):297–325, 2009. find
  11. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.… find our card
  12. Vaibhava Goel and William J Byrne. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115–135, 2000. find
  13. Awni Y Hannun, Andrew L Maas, Daniel Jurafsky, and Andrew Y Ng. First-pass large vocabulary continuous speech recognition using bi-directional recurrent dnns. arXiv preprint arXiv:1408.2873, 2014. — [arXiv:1408.2873](https://arxiv.org/abs/1408.2873) find
  14. Tamir Hazan, Joseph Keshet, and David A McAllester. Direct loss minimization for structured prediction. In Advances in Neural Information Processing Systems, pp. 1594–1602, 2010. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  16. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128– 3137, 2015. find our card
  17. Diederik P Kingma and Jimmy Ba. A method for stochastic optimization. In International Conference on Learning Representation, 2015. find
  18. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. find our card
  19. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. find
  20. Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp. 71–78. Association for Computational… find
  21. Francis Maes, Ludovic Denoyer, and Patrick Gallinari. Structured prediction with reinforcement learning. Machine learning, 77(2-3):271–301, 2009. find
  22. W Thomas Miller, Paul J Werbos, and Richard S Sutton. Neural networks for control. MIT press, 1995. find
  23. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. find
  24. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278–287, 1999. find
  25. Franz Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pp. 160–167. Association for Computational Linguistics, 2003. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics, 2002. find our card
  27. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732, 2015. find our card
  28. Stéphane Ross, Geoffrey J Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. arXiv preprint arXiv:1011.0686, 2010. find
  29. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015. find
  30. Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681, 1997. find
  31. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433, 2015. find our card
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104–3112, 2014. find
  33. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9–44, 1988. find
  34. Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, volume 135. MIT Press Cambridge, 1998. find
  35. Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPS, volume 99, pp. 1057– 1063, 1999. find
  36. Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. 1984. find
  37. Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215–219, 1994. find
  38. Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. URL http://arxiv.org/abs/1605.02688. ↗ our card
  39. John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. Automatic Control, IEEE Transactions on, 42(5):674–690, 1997. find
  40. Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning. arXiv:1506.00619 [cs, stat], June 2015. find
  41. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164, 2015. find our card
  42. Andreas Vlachos. An investigation of imitation learning algorithms for structured prediction. In EWRL, pp. 143–154. Citeseer, 2012. find
  43. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  44. Sam Wiseman and Alexander M Rush. Sequence-to-sequence learning as beam-search optimization. arXiv preprint arXiv:1606.02960, 2016. find
  45. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  46. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, Fran… find our card
  47. Wojciech Zaremba, Tomas Mikolov, Armand Joulin, and Rob Fergus. Learning simple algorithms from examples. arXiv preprint arXiv:1511.07275, 2015. — [arXiv:1511.07275](https://arxiv.org/abs/1511.07275) find