Elamur Research⌕
☰
home / papers / 2018 / Correcting Length Bias in Neural Machine Translation

Correcting Length Bias in Neural Machine Translation

Paper · 2018 · Kenton Murray and David Chiang

Summary

Neural machine translation (NMT) has two well-known problems. Its output tends to be too short, and widening the search beam often lowers quality instead of raising it. Murray and Chiang argue that both problems come from label bias. NMT models are locally normalized, so an overestimated end-of-sentence probability can let short or even empty hypotheses beat full translations once a wide beam stops pruning them. They treat length corrections as changes to the model rather than search tricks. They compare length normalization with a constant per-word reward γ. They also learn γ after training with a perceptron-style rule, derived from a globally normalized objective, that nudges γ by the gap between reference and output lengths. Experiments cover German–English (WMT 2016), Russian–English (WMT 2017) and low-resource French↔English (IWSLT 2010 TALK). Correcting length nearly removes the beam problem, and the tuned word reward matches or slightly beats length normalization. This matters because the fix is cheap and makes NMT baselines stronger.

Key points

  • Without correction, larger beams sharply reduce BLEU and METEOR. In one setting, a beam of 1000 produced empty output for 37% of sentences. Both corrections keep length ratios concentrated near 1.0.
  • On Russian–English with full data, baseline BLEU falls from 24.9 to 22.5 as the beam grows. With the tuned reward it stays around 26.5. With 10% or 50% of the data, wider tuned beams even improve BLEU, which suggests low-resource systems suffer more from the beam problem.
  • The best γ differs by language pair, data size and beam size (for example, about 0.62–0.89 for Russian–English). The authors therefore recommend tuning it for every trained model.
  • Tuning uses only about 1000 development sentences (892 for French–English). On an Nvidia GTX 1080Ti it takes minutes to hours, a small fraction of training time.
  • On METEOR, the word reward generally outperformed length normalization. On BLEU the results were mixed: normalization won on German–English and French–English, tied on English–French and lost on Russian–English. The word reward was better at beam 1000.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Notre DameNotre Dame · United States59 works · 37,127 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,105 works

Cite

Cite as Kenton Murray and David Chiang. 2018. Correcting Length Bias in Neural Machine Translation. arXiv:1808.10006.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.10006 · arXiv 1808.10006

@misc{murray2018correcting,
  title = {{Correcting Length Bias in Neural Machine Translation}},
  author = {Kenton Murray and David Chiang},
  year = {2018},
  doi = {10.48550/arxiv.1808.10006},
  eprint = {1808.10006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.10006}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proc. ACL, pages 2442–2452.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR.
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proc. Conference on Machine Translation, pages 169– 214.
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 Conference on Machine Translation. In Proc. Conference on Machine Translation, volume 2, pages 13…
  5. Léon Bottou. 1991. Une Approche théorique de l’Apprentissage Connexioniste; Applications à la reconnaissance de la Parole. Ph.D. thesis, Université de Paris Sud. find
  6. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013. Audio chord recognition with recurrent neural networks. In ISMIR, pages 335– 340. Citeseer. find
  7. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proc. Workshop on Statistical Machine Translation.
  8. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 18–27.
  9. Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with SMT features. In Proc. AAAI. find
  10. Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134–2139. find
  11. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for wmt’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140. find
  12. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. find
  13. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. ICML, pages 282–289. find
  14. Graham Neubig. 2015. lamtram: A toolkit for language and translation modeling using neural networks. http://www.github.com/neubig/lamtram. ↗
  15. Graham Neubig. 2016. Lexicons and minimum risk training for neural machine translation: NAIST-CMU at WAT2016. In Proceedings of the 3rd Workshop on Asian Translation, pages 119–125.
  16. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. ACL, pages 160–167. find
  17. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proc. ACL, pages 295–302. Association for Computational Linguistics.
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318.
  19. Michael Paul, Marcello Federico, and Sebastian Stüker. 2010. Overview of the IWSLT 2010 evaluation campaign. In International Workshop on Spoken Language Translation (IWSLT) 2010. find
  20. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. In Proceedings of ICLR.
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL, pages 1715–1725.
  22. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proc. of ACL.
  23. Arun Venkatraman, Martial Hebert, and J Andrew Bagnell. 2015. Improving multi-step prediction of learned time series models. In AAAI, pages 3024– 3030. find
  24. Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of EMNLP. find
  25. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. find
  26. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
  27. Yue Zhang and Joakim Nivre. 2012. Analyzing the effect of global learning and beam-search on transition-based dependency parsing. In Proceedings of COLING 2012: Posters, pages 1391–1400. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/murray-2018-correcting-length-bias-neural-machine/