Elamur Research⌕
☰
home / papers / 2022 / High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

Summary

Neural machine translation usually decodes with beam search, assuming that the most probable output is also the best one. Freitag and colleagues at Google Research question this assumption. They show that model probability tracks human-judged quality only loosely, since human translations often receive much lower log-probabilities than beam output.

As an alternative, they apply sampling-based Minimum Bayes Risk (MBR) decoding, following Eikema and Aziz (2020). Unbiased samples from a transformer-big NMT model act as both candidates and pseudo-references. The candidate with the highest average utility against the others is selected. The authors compare lexical utilities (BLEU, chrF) with neural ones (YiSi, BLEURT), and are the first to use neural metrics inside MBR. Systems are tested on WMT newstest2021 En→De and De→En, with MQM judgments from professional translators.

MBR guided by BLEURT earns significantly better human ratings than beam search, even though its translations are less likely under the model and score lower on BLEU. The study argues that decoding should aim at estimated quality rather than likelihood. All outputs and ratings are released publicly.

Key points

  • In the paper's De→En example, the beam translation has logP −2.82, while two correct human translations score only −18.1 and −18.74.
  • Utilities tested are BLEU, chrF, YiSi, and BLEURT (v0.1 and v0.2); a COMET-QE reranking experiment is also reported. Samples are not deduplicated, so model probability is accounted for implicitly. Computing cross-utility costs grow quadratically with the pool size.
  • In a cross-BLEU analysis, beam output, WMT21 submissions (Facebook, Online-W, UEdin), and MBR with BLEU/chrF/YiSi form one cluster of similar translations. MBR-BLEURT outputs and the human references fall outside it, with cross-BLEU below 50.
  • The gap between the two human references (Ref-C, Ref-D) is roughly the same as the gap between either reference and the MBR-BLEURT output.
  • In the MQM evaluation for En→De, major "Terminology/Inappropriate for context" errors drop by roughly one third with MBR. Hypotheses, candidate lists, and ratings are available on Kaggle (machine-translation-mbr-with-neural-metrics).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

92 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00491

@article{freitag2022high,
  title = {{High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics}},
  author = {Freitag, Markus and Grangier, David and Tan, Qijun and Liang, Bowen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00491},
  url = {https://doi.org/10.1162/tacl_a_00491}
}

References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference… find our card
  2. Peter J. Bickel and Kjell A. Doksum. 1977. Mathematical statistics: Basic ideas and selected topics. Holder-Day Series in Probability and Statistics, Holder-Day, San Francisco. find
  3. Pinzhen Chen, Jindřich Helcl, Ulrich Germann, Laurie Burchell, Nikolay Bogoychev, Antonio Valerio Miceli Barone, Jonas Waldendorf, Alexandra Birch, and Kenneth Heafield. 2021. The University of Edinburgh’s English-German Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine trans… find
  4. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics. https://doi.org/10 .18653/… find our card
  5. Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. find our card
  6. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metric shared task. In Proceedings of the Sixth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  7. Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. arXiv preprint arXiv:2104.00054. https://doi.org/10.1162/tacl a 00417 ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243–1252. PMLR. find our card
  10. Vaibhava Goel and William J. Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115–135. https://doi.org/10.1006/csla.2000.0138 — [doi:10.1006/csla.2000.0138](https://doi.org/10.1006/csla.2000.0138) ↗
  11. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Classical structured prediction losses for sequence to sequence learning. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). https://doi.org/10.18653/v1/N18-1033 ↗ our card
  12. Joshua Goodman. 1996. Parsing algorithms and metrics. In Proceedings of the 34th Annual Meeting on Association for Computational Linguistics, ACL ’96, pages 177–183, USA. Association for Computational Linguistics. https://doi.org/10.3115/981863.981887 ↗
  13. Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? The inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4506–4520, Barcelona, Spain (Online). International Committee on Computational Linguisti… find our card
  14. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. https://doi.org/10 .18653/v1/D18-2012 find our card
  15. Shankar Kumar and William Byrne. 2002. Minimum bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 140–147. https://doi.org/10 .3115/1118693.1118712 find
  16. Bryan Eikema and Wilker Aziz. 2021. Sampling-based minimum Bayes risk decoding for neural machine translation. find
  17. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. https://doi.org/10.1162/tacl_a_00437 ↗ our card
  18. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston, Massachusetts, USA… find
  19. Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Jean-Baptiste Lespiau, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. 2021. Machine translation decoding beyond beam search. https://doi.org/10.18653/v1/2021.emnlp-main.662 ↗
  20. Chin-Yew Lin and Franz Josef Och. 2004. Orange: A method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507. find our card
  21. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3956–3965. PMLR. https://doi .org/10.1865… find our card
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  23. Chi-kiu Lo. 2019. YiSi—A unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. Association for Comp… find our card
  24. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W15-3049 find our card
  25. Chi-kiu Lo. 2020. Extended study on using pretrained language models and YiSi-1 for machine translation evaluation. In Proceedings of the Fifth Conference on Machine Translation, pages 895–902, Online. Association for Computational Linguistics. find
  26. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W18-6319 find our card
  27. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM) : A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, pages 455–463. https://doi.org/10.5565/rev/tradumatica.77 — [doi:10.5565/rev/tradumatica.77](https://doi.org/10.5565/rev/tradumatica.77) ↗
  28. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  29. Ricardo Rei, Ana C. Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? Unbabel-ist 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translatio… find our card
  30. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /20… find
  31. Khalil Sima’an. 2003. On maximizing metrics for syntactic disambiguation. In Proceedings of the Eighth International Conference on Parsing Technologies, pages 183–194, Nancy, France. find
  32. David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787–794. https://doi.org/10 .3115/1273073.1273174 find
  33. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur P. Parikh. 2020b. Learning to evaluate translation beyond english: Bleurt submissions to the WMT metrics 2020 shared task. arXiv preprint arXiv: 2010.04297. find our card
  34. Miloš Stanojević and Khalil Sima’an. 2014. Fitting sentence level translation evaluation with many dense features. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 202–206, Doha, Qatar. Association for Computational Linguistics. https://doi .… find
  35. Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, Yuan Cao, Chung-Cheng Chiu, Yanzhang He, Jan Chorowski, Smit Hinsu, Stella Laurenzo, James Qin, Orhan Firat, Wolfgang Macherey, Suyog Gupta, Ankur Bapna, Shuyuan Zhang, Ruoming Pang, Ron J.… find
  36. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  37. Antonio Toral. 2020. Reassessing claims of human parity and super-human performance in machine translation at WMT 2019. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 185–194, Lisbon, Portugal. European Association for Machine Translation. find
  38. Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021. Facebook AI’s WMT21 news translation task submission. In Proceedings of the Sixth Conference on Machine Translation, Online. Association for Computational Linguistics. find our card
  39. Roy Tromble, Shankar Kumar, Franz Josef Och, and Wolfgang Macherey. 2008. Lattice minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 620–629. https://doi.org/10.3115/1613715 .1613792 ↗
  40. Hans Uszkoreit and Arle Lommel. 2013. Multidimensional quality metrics: A new unified paradigm for human and machine translation quality assessment. Localization World, London, pages 12–14. on Machine Translation: Research Papers, pages 133–143, Brussels, Belgium. Association for Computational Lingu… find our card
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  42. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  43. Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the Third Conference Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. B… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/freitag-2022-high-quality-rather-than-high/