Elamur Research⌕
☰
home / papers / 2018 / Classical Structured Prediction Losses for Sequence to Sequence Learning

Classical Structured Prediction Losses for Sequence to Sequence Learning

Summary

Neural sequence to sequence models are usually trained token by token, yet at test time they must output whole sequences. Recent fixes include reinforcement learning and beam search optimization. This paper asks whether older structured prediction objectives, long used for linear NLP models, still work for neural models. The authors apply a family of such losses to a convolutional encoder-decoder (Gehring et al., 2017b): sequence-level likelihood, expected risk, several margin-based losses, and mixtures of sequence- and token-level terms. Each sequence loss is computed over a candidate set generated by the model itself, and the cost is based on BLEU or ROUGE. They evaluate on IWSLT'14 German-English, Gigaword headline summarization and WMT'14 English-French. The sequence losses end up performing about equally well. They reach new best results on IWSLT'14 and Gigaword, and 41.5 BLEU on WMT'14 English-French, which matches the state of the art. In an LSTM setup built to match Wiseman and Rush (2016), expected risk slightly beats beam search optimization. The takeaway is that simple classical objectives remain strong alternatives to more elaborate sequence-level training methods.

Key points

  • The losses compared are TokNLL, TokLS (label smoothing), SeqNLL, Risk, MaxMargin, MultiMargin and SoftmaxMargin. Two ways of mixing token and sequence losses are also tested: Weighted and Constrained. The code is released as part of fairseq.
  • In the like-for-like LSTM comparison, Risk adds +2.75 BLEU over the reimplemented baseline. Beam search optimization reported +2.33 BLEU.
  • On WMT'14 English-French, sequence-level training lifts an already strong TokLS model by +0.37 BLEU. With decoder self-attention added, the gain shrinks to +0.2 BLEU. Running Risk on only the news-commentary data gives 41.5 BLEU.
  • On Gigaword, the TokLS baseline already beats prior systems on ROUGE-2 and ROUGE-L. Optimizing each ROUGE metric with Risk improves it further.
  • On Gigaword, combining Risk with the token loss (Weighted) beats Risk alone. For example, RG-2 is 18.05 with Weighted versus 17.34 with Risk alone.

Interesting ideas

  • I1. Train toward a pseudo reference. Instead of the gold target, the loss uses the candidate with the highest BLEU or ROUGE in the model's own candidate set. Using the gold target can lead to degenerate solutions, so this trick is worth trying wherever the exact reference is hard for a model to reproduce.
  • I2. Keep a token-level loss in the mix. The best results came from combining sequence-level and token-level losses, either by weighting them or by choosing one per example. This helps keep basic token accuracy while tuning for the final metric, which is useful for any metric-driven fine-tuning.
  • I3. Rescale costs within each candidate set. BLEU scores are normalized by the max–min spread among each sentence's candidates. Short sentences, whose candidates vary widely in score, then no longer dominate the updates. This simple fix suits any reward-based training over variable-length outputs.
  • I4. Generate candidates with the current model. Decoding the candidate set from the model being trained, rather than from a fixed earlier model, was needed for the best results. The cost is slower training. The authors propose reusing older candidates alongside fresh ones, MERT-style, as a way to cut that cost.
  • I5. Make sentence-level BLEU usable as a training signal. Higher-order n-gram counts are smoothed by starting them at one, so zero matches do not collapse the geometric mean. Sequence probabilities are also length-normalized to avoid favouring short outputs. Both are cheap, general fixes for training against sentence-level metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1033

@inproceedings{edunov2018classical,
  title = {{Classical Structured Prediction Losses for Sequence to Sequence Learning}},
  author = {Edunov, Sergey and Ott, Myle and Auli, Michael and Grangier, David and Ranzato, Marc'Aurelio},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1033},
  url = {https://doi.org/10.18653/v1/n18-1033}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. Proc. of ICLR . find
  2. Ayana, Shiqi Shen, Yu Zhao, Zhiyuan Liu, Maosong Sun, et al. 2016. Neural headline generation with sentence-wise optimization. arXiv preprint arXiv:1604.01904 . — [arXiv:1604.01904](https://arxiv.org/abs/1604.01904) find
  3. Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017. Deep recurrent generative decoder for abstractive text summarization. arXiv . find
  4. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An Actor-Critic Algorithm for Sequence Prediction. In arXiv preprint arXiv:1607.07086. find our card
  5. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. find
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  7. Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating automatic evaluation metrics for machine translation. In Proc. of COLING. find our card
  8. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In Proc. of IWSLT. find our card
  9. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP. find
  10. Samidh Chatterjee and Nicola Cancedda. 2010. Minimum error rate training by sampling the translation lattice. find
  11. Franz Josef Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. Sapporo, Japan, pages 160–167. find
  12. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language Modeling with Gated Convolutional Networks. In Proc. of ICML. find our card
  13. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of The 30th International Conference on Machine Learning. pages 1310–1318. find
  14. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, and Yasemin Altun. 2005. Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research 6:1453—-1… find
  15. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 . find our card
  16. Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proc. of ICLR Workshop. find our card
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv . find
  18. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level Training with Recurrent Neural Networks. In Proc. of ICLR. find our card
  19. R. J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8:229—-256. find
  20. Antti-Veikko I Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2010. BBN System Description for WMT10 System Combination Task. In Proc. of WMT. Association for Computational Linguistics, pages 321–326. find
  21. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proc. of ACL. find
  22. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 . find
  23. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proc. of EMNLP. find
  24. Tim Salimans and Diederik P Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868 . — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) find
  25. Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In Proc. of NIPS. find
  26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proc. of ACL. find
  27. Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. arXiv . find
  28. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In Proc. of ACL. find our card
  29. David A. Smith and Jason Eisner. 2006. Minimum Risk Annealing for Training Log-Linear Models. In Proc. of ACL. find
  30. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent Neural Networks from overfitting. JMLR 15:1929–1958. find
  31. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of initialization and momentum in deep learning. In ICML. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS. pages 3104–3112. find
  33. Jun Suzuki and Masaaki Nagata. 2017. Cutting-off redundant repeating generations for neural abstractive summarization. arXiv preprint arXiv:1701.00138 . find
  34. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. arXiv . find
  35. Ben Taskar, Carlos Guestrin, and Daphne Koller. 2003. Max-margin markov networks. In NIPS. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/edunov-2018-classical-structured-prediction-losses-sequence/