Elamur Research
☰
home / papers / 2019 / On NMT Search Errors and Model Errors: Cat Got Your Tongue?

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1331.pdf

Summary

Neural machine translation is almost always decoded with beam search. Until this paper, no one had measured how often beam search misses the output the model actually scores highest, because the search space is far too large to enumerate. Stahlberg and Byrne build an exact decoder that combines beam search with depth-first search. NMT log-probabilities can only fall as a hypothesis grows, so any partial translation scoring below the best complete one found so far can be safely discarded, with the beam output serving as the initial bound. They apply it to a Transformer base model on the full WMT15 English-German test set. Beam search misses the true optimum for most sentences, and for over half of the inputs the model's top-scoring output is the empty translation. Beam search therefore succeeds partly because its errors conceal a serious model flaw: a systematic preference for short outputs. Length-constrained and length-normalized exact search show that simple heuristics do not repair this flaw. The work changes how modelling and search errors in NMT should be weighed.

Key points

  • Setup: Experiments use English-German newstest2015 (2,169 sentences) and a Transformer base model trained with Tensor2Tensor on WMT18 data without ParaCrawl, using 32K BPE merges. The exact search is released in the SGNMT decoder as the simpledfs and simplelendfs strategies.
  • Search errors: Greedy decoding misses the global best score on 73.6% of sentences and Beam-10 on 57.7%. Beam-100 still misses it on 53.62%, despite running ten times slower than Beam-10.
  • Empty translations: Exact search returns the empty translation for 51.8% of sentences, dropping BLEU from 30.3 (Beam-10) to 2.1. The problem also affects LSTM, SliceNet and Transformer-Big systems; even the tuned Transformer-Big has 25.8% empty translations.
  • Sentence length: Longer sources suffer more. For nearly all sentences above 40 tokens, the global best output is empty.
  • Length constraints and normalization (tested on a subset of the test set):
    • Forcing a minimum length of 0.25× the source still leaves a peak of too-short outputs.
    • An oracle constraint to the reference length gains 0.9 BLEU (37.0 → 37.9).
    • Exact search with length normalization reaches 36.4 BLEU, still below Beam-10's 37.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

91 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1331

@inproceedings{stahlberg2019nmt,
  title = {{On NMT Search Errors and Model Errors: Cat Got Your Tongue?}},
  author = {Stahlberg, Felix and Byrne, Bill},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1331},
  url = {https://doi.org/10.18653/v1/d19-1331}
}

References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013. Audio chord recognition with recurrent neural networks. In ISMIR, pages 335– 340. Citeseer. find
  3. Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. Recurrent neural networks as weighted language recognizers. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2261–2271, New Orleans, Louisiana. Association for Computational Linguistics. find
  4. Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with SMT features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 151– 157. AAAI Press. find
  5. Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? Optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134–2139, Copenhagen, Denmark. Association for Computational Linguistics. find
  6. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisbon, Portugal. Association for Computational Linguistics. find
  7. Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. 2017. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv:1706.03059. ↗
  8. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709. Association for Computational Linguistics. find
  9. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. find
  10. Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802. ↗
  11. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212–223, Belgium, Brussels. Association for Computational Linguistics. find
  12. Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2017. Analyzing neural MT search and model performance. In Proceedings of the First Workshop on Neural Machine Translation, pages 11–17. find
  13. Myle Ott, Michael Auli, David Grangier, et al. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3953–3962. find our card
  14. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. find
  15. Pavel Sountsov and Sunita Sarawagi. 2016. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516–1525, Austin, Texas. Association for Computational Linguistics. find
  16. Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2018a. The University of Cambridge’s machine translation systems for WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 504–512, Belgium, Brussels. Association for Computational Linguistics. find
  17. Felix Stahlberg, Eva Hasler, Danielle Saunders, and Bill Byrne. 2017. SGNMT – A flexible NMT decoding platform for quick prototyping of new models and search strategies. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 25–30. Association for Computational Linguistics. find
  18. Felix Stahlberg, Danielle Saunders, Gonzalo Iglesias, and Bill Byrne. 2018b. Why not be versatile? Applications of the SGNMT decoder for machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 208–216. Association for Machine Translation in the Americas. find
  19. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  20. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. ↗ our card
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  22. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  23. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054–3059, Brussels, Belgium. Association for Computational Linguistics. find