Elamur Research⌕
☰
home / papers / 2018 / Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation

Paper · 2018 · Yilin Yang, Liang Huang and Mingbo Ma

Summary

In neural machine translation, widening the beam beyond about 5 usually makes BLEU worse. This is the "beam search curse." Yang, Huang and Ma trace it to output length. A wider beam reaches </eos> earlier, and the model's scores favor short candidates. Translations therefore shrink, and the brevity penalty lowers BLEU sharply.

To fix this, the authors train a small MLP that predicts the target/source generation ratio from the encoder states. That gives an expected output length. They plug it into three rescoring schemes:

  • Bounded Word-Reward with the predicted length
  • Bounded Adaptive-Reward, which takes its reward from the current beam
  • BP-Norm, which adds a log brevity penalty to the length-normalized score

The last two have no hyperparameters. The authors also derive optimal stopping rules for these scores.

On Chinese-to-English (NIST), the new methods keep length ratios near 1 and improve as the beam grows. The hyperparameter-free variants beat standard length normalization by about +2.0 BLEU. The authors recommend BP-Norm as the simplest choice that performs as well as the others.

Key points

  • Data and model: about 2M Chinese–English sentence pairs with BPE, giving 18k Chinese and 10k English vocabularies. NIST 06 is the dev set and NIST 08 the test set. The model is a 2-layer bidirectional LSTM in OpenNMT-py with 28.5M parameters.
  • Test results with large beams (b≈40): BP-Norm scores 36.22 BLEU and Bounded Adaptive-Reward 36.23. Length normalization scores 34.26, and the default b=5 decoder 32.88.
  • Length ratio: the proposed methods hold it around 0.97–0.99, compared with roughly 0.87–0.89 for the baselines. They help most on long input sentences.
  • Optimal stopping alone: adding it to length normalization raises BLEU by about +0.9.
  • Baselines compared: Moses, GNMT-style length and coverage penalties (α=β=0.3), and tuned Bounded Word-Reward.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China40 works · 14,627 citations🎓Oregon State UniversityCorvallis · United States6 works · 886 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Yilin Yang, Liang Huang and Mingbo Ma. 2018. Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1342

@inproceedings{yang2018breaking,
  title = {{Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation}},
  author = {Yilin Yang, Liang Huang and Mingbo Ma},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1342},
  url = {https://doi.org/10.18653/v1/d18-1342}
}

References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  2. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. of ICML.
  3. Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with smt features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 151–157. AAAI Press. find
  4. Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? optimal beam search for neural text generation (modulo beam size). In EMNLP. find
  5. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, volume 3, page 413.
  6. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints.
  7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  8. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  9. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of ACL.
  10. Xing Shi, Kevin Knight, and Deniz Yuret. 2016. Why neural translations are the right length. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2278–2282. find
  11. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  12. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI, pages 3097–3103. find
  13. Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of EMNLP. find
  14. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  15. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. CoRR, abs/1706.03872. find
  16. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML. find
  17. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of WMT 2018. find
  18. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. arXiv preprint arXiv:1803.00047.
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL, pages 311–318, Philadephia, USA.
  20. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. ICLR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-breaking-beam-search-curse-study/