Elamur Research
☰
home / papers / 2018 / Analyzing Uncertainty in Neural Machine Translation

Analyzing Uncertainty in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.00047

Summary

This paper asks why neural machine translation (NMT) systems show familiar problems: accuracy drops as beam width grows, rare words are under-produced, and outputs lack diversity. The authors link these problems to uncertainty. Some is intrinsic, because one source sentence admits many valid translations. Some is extrinsic, introduced by noisy training data. Using pre-trained convolutional fairseq-py models on WMT'14 En-Fr and En-De (plus WMT'17 En-De), they adapt calibration and statistical tools to measure how closely the model distribution matches the data distribution and how it interacts with search. Beam search turns out to be very effective at finding high-likelihood outputs. The models are reasonably calibrated at the token and set level, yet they spread probability too thinly across the space of sequences. As a result, sampled translations are weak, and beam outputs cluster around a few references. Training pairs where the target simply repeats the source turn out to be a major cause of beam-size degradation. The work offers diagnostic tools, simple remedies and new multi-reference test data for studying NMT behaviour.

Key points

  • Between 1.1% and 2.0% of WMT'14 En-De/En-Fr training pairs are source "copies," detected by a unigram intersection-over-union threshold of 50%. This noise largely explains worse results with wide beams.
  • The authors release 10 extra human translations for each of 500 sentences from the WMT'14 En-Fr and En-De test sets (github.com/facebookresearch/analyzing-uncertainty-nmt).
  • Expected inter-sentence BLEU is 44.5 (En-Fr) and 32.1 (En-De) between human translations, versus only 28.6 and 24.2 for model samples. This gap shows the model distribution is too diffuse.
  • On En-Fr, beam 5 reaches 41.4 single-reference and 70.2 oracle-reference sentence BLEU but matches only about 1.9 of 10 references. Sampling 200 hypotheses covers 7.4 references but drops average-oracle BLEU to 39.1.
  • The authors suspect that over-smoothing stems from the class of smooth functions NMT models represent.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.00047 · arXiv 1803.00047

@misc{ott2018analyzing,
  title = {{Analyzing Uncertainty in Neural Machine Translation}},
  author = {Ott, Myle and Auli, Michael and Grangier, David and Ranzato, Marc'Aurelio},
  year = {2018},
  doi = {10.48550/arxiv.1803.00047},
  eprint = {1803.00047},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.00047}
}

References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  2. Bentivogli, L., Bisazza, A., Cettolo, M., and Federico, M. Neural versus phrase-based machine translation quality: a case study. In EMNLP, 2016. find
  3. Blunsom, P., Cohn, T., and Osborne, M. A discriminative latent variable model for statistical machine translation. In ACL-HLT, 2008. find
  4. Cho, K. Noisy parallel approximate decoding for conditional recurrent language model. arXiv:1605.03835, 2016. — [arXiv:1605.03835](https://arxiv.org/abs/1605.03835) find
  5. Dreyer, M. and Marcu, D. Hyter: Meaning-equivalent semantics for translation evaluation. In ACL, 2012. find
  6. Edunov, S., Ott, M., Auli, M., Grangier, D., et al. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), vol… find
  7. Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., and Dolan, B. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863, 2015. find
  8. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional Sequence to Sequence Learning. In Proc. of ICML, 2017. URL github.com/ facebookresearch/fairseq-py. find our card
  9. Gimpel, K., Batra, D., Dyer, C., and Shakhnarovich, G. A systematic exploration of diversity in machine translation. In Conference on Empirical Methods in Natural Language Processing, 2013. find
  10. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. find
  11. Isabelle, P., Cherry, C., and Foster, G. A challenge set approach to evaluating machine translation. arXiv:1704.07431, 2017. find our card
  12. Koehn, P. and Knowles, R. Six challenges for neural machine translation. arXiv:1706.03872, 2017. find
  13. Kuleshov, V. and Liang, P. Calibrated structured prediction. In Neural Information Processing Systems, 2015. find
  14. Li, J. and Jurafsky, D. Mutual information and diverse decoding improve neural machine translation. arXiv:1601.00372, 2016. — [arXiv:1601.00372](https://arxiv.org/abs/1601.00372) find
  15. Lin, C.-Y. and Och, F. J. Orange: a method for evaluating automatic evaluation metrics for machine translation. In Proc. of COLING, 2004. find
  16. Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015. URL http://nlp. stanford.edu/projects/nmt. find
  17. Nguyen, K. and O’Connor, B. Posterior calibration and exploratory analysis for natural language processing models. In Proc. of EMNLP, 2015. find
  18. Niehues, J., Cho, E., Ha, T.-L., and Waibel, A. Analyzing neural mt search and model performance. arXiv:1708.00563, 2017. find
  19. Padó, S., Cer, D., Galley, M., Jurafsky, D., and Manning, C. D. Measuring machine translation quality as semantic equivalence: A metric based on entailment features. Machine Translation, 23:181–193, 2009. find
  20. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002. find our card
  21. Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. In ICLR, 2016. find our card
  22. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Association for Computational Linguistics, 2016. find
  23. Smith, J. R., Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. Dirt cheap web-scale parallel text from the common crawl. In ACL, 2013. find
  24. Sokolov, A., Wisniewski, G., and Yvon, F. Lattice bleu oracles in machine translation. ACM Transactions on Speech and Language Processing, 2008. find
  25. Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. On the importance of initialization and momentum in deep learning. In ICML, 2013. find
  26. Toral, A. and Sanchez-Cartagena, V. M. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In ACL, 2017. find
  27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv, 2017. find
  28. Venuti, L. The Translator’s Invisibility. Routledge, 2008. find
  29. Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., and Batra, D. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv:1610.02424, 2016. find our card
  30. Zhang, B., Xiong, D., Su, J., Duan, H., and Zhang, M. Variational neural machine translation. In EMNLP, 2016. find
  31. Zoph, B., Ghazvininejad, M., and Knight, K. How much information does a human translator add to the original? In EMNLP, 2015. find