Elamur Research⌕
☰
home / papers / 2006 / Re-evaluating the Role of Bleu in Machine Translation Research

Re-evaluating the Role of Bleu in Machine Translation Research

Summary

The authors question whether the Bleu metric really reflects human judgments of machine translation quality well. The statistical MT community relies on it to evaluate changes to systems, to tune them via minimum error rate training, and to compare participants in evaluation campaigns. The paper has a theoretical and a practical part. The theoretical part shows that Bleu places almost no constraints on the order of n-gram matches or on which reference they are drawn from. As a result, millions of different translation variants can receive the same score, and many of them are grammatically nonsensical. The practical part presents two counterexamples. The first is taken from the NIST MT Evaluation 2005, where the Bleu ranking diverged from human judgments. The second is the authors' own experiment on Europarl (French→English), in which two SMT systems were compared with the rule-based system Systran. The conclusion is that a higher Bleu score does not guarantee a better translation, and a translation can be improved without an increase in Bleu. This matters because research directions that failed under Bleu, such as syntax or word sense disambiguation, may deserve a second, manual evaluation.

Key points

  • Bleu computes modified n-gram precision (up to 4-grams) against multiple reference translations and adds a brevity penalty. In the example from NIST 2005, an 18-token hypothesis obtains precisions of 0.83 / 0.59 / 0.31 / 0.2.
  • Segments between the points where bigrams do not match can be permuted without any loss in score. A lower bound on the number of such permutations is (k − b)!. For the example sentence this gives at least 40 320 variants, and for some system translations from NIST 2005, more than 10^73.
  • Europarl experiment: three judges rated the fluency and adequacy of translations of 300 sentences produced by three systems. Both SMT systems had a higher Bleu score than Systran, yet humans rated Systran considerably higher.
  • Bleu is appropriate for tracking incremental changes to a single system, for comparing systems with similar strategies (phrase-based SMT systems against each other), and for tuning feature weights. It is inappropriate for comparing systems with fundamentally different approaches or for detecting improvements that are rare or poorly modeled by Bleu.
  • Other automatic metrics may have the same shortcomings: Meteor, Precision and Recall (Melamed et al.).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · AI & machine learning

Topics: Machine translation · Evaluation & benchmarks

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2006re,
  title = {{Re-evaluating the Role of Bleu in Machine Translation Research}},
  author = {Callison-Burch, Chris and Osborne, Miles and Koehn, Philipp},
  booktitle = {11th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2006},
  url = {https://aclanthology.org/E06-1032/}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bogdan Babych and Anthony Hartley. 2004. Extending the Bleu MT evaluation method with frequency weightings. In Proceedings of ACL. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for MT evaluation with improved correlation with human judgments. In Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, Michigan. find our card
  3. Chris Callison-Burch. 2005. Linear B system description for the 2005 NIST MT evaluation exercise. In Proceedings of the NIST 2005 Machine Translation Evaluation Workshop. find
  4. Marine Carpuat and Dekai Wu. 2005. Word sense disambiguation vs. statistical machine translation. In Proceedings of ACL. find
  5. Eugene Charniak, Kevin Knight, and Kenji Yamada. 2003. Syntax-based language models for machine translation. In Proceedings of MT Summit IX. find
  6. Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX. find
  7. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Human Language Technology: Notebook Proceedings, pages 128–132, San Diego. find
  8. Eduard Hovy and Deepak Ravichandra. 2003. Holy and unholy grails. Panel Discussion at MT Summit IX. find
  9. Philipp Koehn. 2004. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of AMTA. find
  10. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit. find
  11. LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Revision 1.5. find
  12. Audrey Lee and Mark Przybocki. 2005. NIST 2005 machine translation evaluation official results. Official release of automatic evaluation scores for all submissions, August. find
  13. Chin-Yew Lin and Ed Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of HLT-NAACL. find our card
  14. Dan Melamed, Ryan Green, and Jospeh P. Turian. 2003. Precision and recall of machine translation. In Proceedings of HLT/NAACL. find our card
  15. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of ACL. find our card
  16. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of NAACL-04, Boston. find our card
  17. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of ACL, Sapporo, Japan, July. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  19. Radu Soricut and Eric Brill. 2004. A unified framework for automatic evaluation using n-gram co-occurrence statistics. In Proceedings of ACL. find
  20. Henry Thompson. 1991. Automatic evaluation of translation quality: Outline of methodology and report on pilot experiment. In (ISSCO) Proceedings of the Evaluators Forum, pages 215–223, Geneva, Switzerland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/burch-2006-re-evaluating-role-bleu-machine/