Elamur Research
☰
home / papers / 2020 / BLEU might be Guilty but References are not Innocent

BLEU might be Guilty but References are not Innocent

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.5.pdf

Summary

The authors argue that weak agreement between automatic MT metrics and human judgments, especially for strong systems, is partly caused by the reference translations rather than only the metrics. Standard human references tend toward translationese: they follow the source's structure and wording, so overlap metrics favour system outputs with the same style and penalise equally accurate but more natural ones. Working on WMT 2019 English→German newstest2019, they commissioned an extra independent professional translation and asked separate linguists to paraphrase the existing references as freely as possible while keeping the meaning. They then measured how well various metrics, computed against each reference type, correlate with human ratings. Paraphrased references gave the most human-like system rankings across all metrics tested. They also correctly ranked back-translation and APE systems, which standard references tend to underrate. Plain multi-reference BLEU brought no improvement for high-quality output. These results suggest that earlier research may have wrongly rejected techniques that make MT output more fluent.

Key points

  • Reference types: original WMT, an additional reference (AR) by 10 professional linguists with CAT tools and copy-paste disabled and sentences shuffled, and paraphrased versions of both (WMT.p, AR.p).
  • Paraphrased references improved correlation with human judgments for BLEU, chrF, METEOR, BERTScore and Yisi, most clearly among the top WMT 2019 submissions.
  • Standard multi-reference BLEU did not beat single-reference BLEU. Picking the best-rated reference per sentence worked better, but the authors recommend a single paraphrased reference as cheaper and reliable.
  • Monotonicity (average alignment distance via fast-align; higher means less word-by-word): WMT 5.17, AR 5.27, WMT.p 6.43, AR.p 6.88.
  • BLEU against paraphrases is far lower. Standard-reference matches rely on generic n-grams (e.g. ", sagte er ." matched 28 times for Facebook's winning system). All references are released at github.com/google/wmt19-paraphrased-references.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

117 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.5

@inproceedings{freitag2020bleu,
  title = {{BLEU might be Guilty but References are not Innocent}},
  author = {Freitag, Markus and Grangier, David and Caswell, Isaac},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.5},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.5}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find our card
  2. Chris Callison-Burch. 2009. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286–295, Singapore. Association for Computational Linguistics. find
  3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  4. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further Meta-evaluation of Machine Translation. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT ’08, pages 70–106, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  5. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. find our card
  6. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of Bleu in Machine Translation Research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. find
  7. Marine Carpuat and Michel Simard. 2012. The Trouble with SMT Consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449. Association for Computational Linguistics. find
  8. Rachel Bawden, Biao Zhang, Lisa Yankovskaya, Andre Tättar, and Matt Post. 2020. Explicit representation of the translation space: Automatic paraphrasing for machine translation evaluation. find
  9. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  10. Nikolay Bogoychev and Rico Sennrich. 2019. Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation. arXiv preprint arXiv:1911.03362. ↗
  11. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Markus Dreyer and Daniel Marcu. 2012. Hyter: Meaning-equivalent semantics for translation evaluation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162–171. Association for Computational Linguistics. shared task on pronoun translation. In Proceedings of the Second Workshop on Discourse in Machine Translation, pages 1–16, Lisbon, Portugal. Association for Computational Linguistics. find
  12. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. find
  13. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 455–462, New York City, USA. Association for Computational Linguistics. find
  14. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. find
  15. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press. find
  16. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics. find
  17. Sergey Edunov, Myle Ott, Marc’Aurelio Ranzato, and Michael Auli. 2019. On The Evaluation of Machine Translation Systems Trained With Back-Translation. arXiv preprint arXiv:1908.05204. ↗ our card
  18. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 1318–1326, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  19. Marina Fomicheva and Lucia Specia. 2016. Reference Bias in Monolingual Machine Translation Evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 77–82, Berlin, Germany. Association for Computational Linguistics. find
  20. David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proceedings of MT-Summit XII, pages 81–88. find
  21. Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at Scale and Its Implications on MT Evaluation Biases. In Proceedings of the Fourth Conference on Machine Translation, pages 34–44, Florence, Italy. Association for Computational Linguistics. find our card
  22. Sara Laviosa. 1997. How comparable can’comparable corpora’be? Target. International Journal of Translation Studies, 9(2):289–319. find
  23. Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2017. A Convolutional Encoder Model for Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 123–135. find
  24. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL ’12, pages 255– 265, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  25. Martin Gellerstam. 1986. Translationese in swedish novels translated from english. In Lars Wollin and Hans Lindquist, editors, Translation Studies in Scandinavia, page 88–95. CWK Gleerup. find
  26. Chi-kiu Lo. 2019. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513. find our card
  27. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  28. Qingsong Ma, Yvette Graham, Timothy Baldwin, and Qun Liu. 2017. Further investigation into reference bias in monolingual evaluation of machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2476–2485, Copenhagen, Denmark. Association for Computational Linguistics. find
  29. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in Machine Translation Evaluation. CoRR, abs/1906.09833. ↗
  30. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? Reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113–123, Belgium, Brussels. Association for Computational Linguistics. 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  31. Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty. 2018. Translation Quality Assessment: From Principles to Practice. Springer. find
  32. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece. European Language Resources Association (ELRA). find
  33. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, 6:407–420. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  36. John S. White. 1994. The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Further Approaches. In Proceedings of the 1994 Conference of the Association for Machine Translation in the Americas, pages 193–205. find
  37. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. find our card
  38. Rabih Zbib, Gretchen Markiewicz, Spyros Matsoukas, Richard Schwartz, and John Makhoul. 2013. Systematic comparison of professional and crowdsourced reference translations for machine translation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 612–616, Atlanta, Georgia. Association for Computational Linguistics. find
  39. Maja Popović. 2019. On reducing translation shifts in translations intended for MT evaluation. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 80–87. find
  40. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  41. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. CoRR, abs/1906.08069. ↗
  42. Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2019. Translationese as a language in ”multilingual” nmt. find
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. Arxiv, 1904.09675. find
  44. Aurko Roy and David Grangier. 2019. Unsupervised paraphrasing without translation. In ACL (1), pages 6033–6039. Association for Computational Linguistics. find
  45. Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006. Re-evaluating Machine Translation Results with Paraphrase Support. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 77–84, Sydney, Australia. Association for Computational Linguistics. find
  46. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics. find
  47. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  48. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. arXiv preprint arXiv:2004.14564. ↗ our card