Sequence Level Training with Recurrent Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.06732

Summary

Recurrent language models used for generation are usually trained to predict the next word from ground-truth history. At inference, however, they must build whole sequences from their own outputs. This mismatch, called exposure bias, lets errors compound. The training loss also operates per word, while systems are judged with sequence-level, non-differentiable metrics such as BLEU or ROUGE.

The authors propose MIXER (Mixed Incremental Cross-Entropy Reinforce), which builds on the REINFORCE algorithm so the model is trained on its own samples and rewarded directly with the test metric. Applying REINFORCE naively fails because the space of possible word sequences is enormous. MIXER therefore starts from a cross-entropy-trained model and gradually hands more of each sequence over to the REINFORCE objective, using a hybrid loss.

The method was tested on text summarization, machine translation and image captioning. With plain greedy decoding, MIXER beat XENT, DAD and a new End-to-End BackProp (E2E) baseline. It was also competitive with or better than XENT using beam search, while running much faster. This makes metric-driven sequence training practical for neural text generation.

Key points

The paper compares four training regimes along three properties: avoiding exposure bias, end-to-end backpropagation, and sequence-level loss. XENT has none of them, DAD has only the first, E2E has the first two, and MIXER has all three.
Greedy MIXER outperformed XENT with beam search while being at least 10× faster than a beam of size 10. Combining MIXER with beam search can add further gains, depending on the task.
The curriculum schedule found by hyperparameter search was:
- N_XENT = 20 (summarization), 25 (translation) and 20 (captioning);
- N_XE+R = 5 for all tasks;
- Δ = 2, 3 and 2 respectively.
Summarization used an Elman RNN and was evaluated with both BLEU and ROUGE-2, showing the method is metric-agnostic. Translation and captioning used LSTMs. All models used an attentive encoder that averages position-aware embeddings over a 5-word window.
The code is released at github.com/facebookresearch/MIXER.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

466 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 437 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06732 · arXiv 1511.06732

@misc{ranzato2015sequence,
  title = {{Sequence Level Training with Recurrent Neural Networks}},
  author = {Ranzato, Marc'Aurelio and Chopra, Sumit and Auli, Michael and Zaremba, Wojciech},
  year = {2015},
  doi = {10.48550/arxiv.1511.06732},
  eprint = {1511.06732},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06732}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Auli, M. and Gao, J. Decoder integration and expected bleu training for recurrent neural network language models. In Proc. of ACL, June 2014. find
Ba, J.L., Mnih, V., and Kavukcuoglu, K. Multiple object recognition with visual attention. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015. find
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., , and Federico, M. Report on the 11th iwslt evaluation campaign. In Proc. of IWSLT, 2014. find our card
Daume III, H., Langford, J., and Marcu, D. Search-based structured prediction as classification. Machine Learning Journal, 2009. find
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. Imagenet: a large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009. find
Elman, Jeffrey L. Finding structure in time. Cognitive Science, 14(2):179–211, 1990. find
Graff, D., Kong, J., Chen, K., and Maeda, K. English gigaword. Technical report, 2003. find
Graves, A. and Jaitly, N. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014. find
He, X. and Deng, L. Maximum expected bleu training of phrase and lexicon translation models. In ACL, 2012. find
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. find
Kneser, Reinhard and Ney, Hermann. Improved backing-off for M-gram language modeling. In Proc. of the International Conference on Acoustics, Speech, and Signal Processing, pp. 181–184, May 1995. find
Koehn, Philipp, Hoang, Hieu, Birch, Alexandra, Callison-Burch, Chris, Federico, Marcello, Bertoldi, Nicola, Cowan, Brooke, Shen, Wade, Moran, Christine, Zens, Richard, Dyer, Chris, Bojar, Ondrej, Constantin, Alexandra, and Herbst, Evan. Moses: Open source toolkit for statistical machine translation.… find
Liang, Percy, Bouchard-Côté, Alexandre, Taskar, Ben, and Klein, Dan. An end-to-end discriminative approach to machine translation. In acl-coling2006, pp. 761–768, Jul 2006. find
Lin, C.Y. and Hovy, E.H. Automatic evaluation of summaries using n-gram co-occurrence statistics. In HLT-NAACL, 2003. find
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C.L. Microsoft coco: Common objects in context. Technical report, 2014. find our card
McAllester, D., Hazan, T., and Keshet, J. Direct loss minimization for structured prediction. In NIPS, 2010. find
Mikolov, T., Karafit, M., Burget, L., Cernock, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH, 2010. find
Mnih, V., Heess N., Graves, A., and Kavukcuoglu, K. Recurrent models of visual attention. In NIPS, 2014. find
Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In AISTATS, 2005. find
Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. find our card
Reiter, E. and Dale, R. Building natural language generation systems. Cambridge university press, 2000. find
Ross, S., Gordon, G.J., and Bagnell, J.A. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011. find
Rosti, Antti-Veikko I, Zhang, Bing, Matsoukas, Spyros, and Schwartz, Richard. Expected bleu training for graphs: Bbn system description for wmt11 system combination task. In Proc. of WMT, pp. 159–165. Association for Computational Linguistics, July 2011. find
Rumelhart, D.E., Hinton, G.E., and Williams, R.J. Learning internal representations by backpropagating errors. Nature, 323:533–536, 1986. find
Rush, A.M., Chopra, S., and Weston, J. A neural attention model for abstractive sentence summarization. In EMNLP, 2015. find
Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc. Sequence to sequence learning with neural networks. In Proc. of NIPS, 2014. find
Sutton, R.S. and Barto, A.G. Reinforcement learning: An introduction. MIT Press, 1988. find
Venkatraman, A., Hebert, M., and Bagnell, J.A. Improving multi-step prediction of learned time series models. In AAAI, 2015. find
Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. find
Xu, X., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. find our card
Zaremba, W. and Sutskever, I. Reinforcement learning neural turing machines. Technical report, 2015. find