Elamur Research
☰
home / papers / 2014 / A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU

A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W14-3346.pdf

Summary

Sentence-level BLEU is unreliable because it multiplies its n-gram precisions together and takes the root: if a short hypothesis has no matching 4-grams, its score collapses to zero no matter how many unigrams or bigrams it shares with the reference. Chen and Cherry systematically compare seven smoothing schemes for this problem. Four come from earlier work: an epsilon replacement, add-one (Lin and Och), the geometric back-off in NIST mteval-v13a.pl, and prior interpolation (Gao and He). Three are new: length-scaled pseudo-counts, averaging of neighbouring n-gram counts, and a combination of the two. The authors test each scheme in two ways. First, they measure correlation with human rankings on WMT 2012/2013 metrics-task data. Second, they use each scheme as the objective when tuning phrase-based SMT on NIST Chinese-English and Arabic-English with expected-BLEU optimization. The combined method (Smoothing 7) correlates best with human judgments at the segment level. In tuning, however, the choice of smoothing barely changes test BLEU. This gives practitioners evidence about when the smoothing choice matters and when it does not.

Key points

  • Segment-level correlation: Kendall's τ improves with every smoothing method. Smoothing 7 performs best, reaching 0.236 into English (vs. 0.165 with no smoothing) and 0.187 out of English (vs. 0.119).
  • Data and settings: Free parameters were tuned on WMT 2008 data (ε = 0.1, K = 5, α = 5). Test sets cover French, Spanish, German, Czech and Russian, in both directions with English.
  • System level: A length-weighted average of sentence BLEU correlates better with humans than the original IBM corpus-level BLEU. The particular smoothing method makes little difference here; Spearman ρ stays unchanged in 13 of 14 cases.
  • Small-feature tuning: With SGD-based expected-BLEU tuning, no pair of methods differs significantly, as tested with MultEval over 5 replications.
  • Large-feature tuning (~7,000 sparse features): Smoothing 7 significantly beats the unsmoothed baseline in 3 of 4 test scenarios. However, the baseline wins on Chinese-English MT08, so the authors consider these results inconclusive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

94 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3346

@inproceedings{chen2014systematic,
  title = {{A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU}},
  author = {Chen, Boxing and Cherry, Colin},
  booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},
  year = {2014},
  doi = {10.3115/v1/w14-3346},
  url = {https://doi.org/10.3115/v1/w14-3346}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  2. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada, June. Association for Computational Linguistics. find our card
  3. Daniel Cer, Christopher D. Manning, and Daniel Jurafsky. 2010. The best lexical metric for phrase-based std 0.1 0.1 0.1 0.1 0.1 0.1 statistical mt system optimization. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 555–563, Los Angeles, California, June. Association for Computational Linguistics. find
  4. Boxing Chen, Roland Kuhn, George Foster, and Howard Johnson. 2011. Unpacking and transforming feature functions: New ways to smooth phrase tables. In MT Summit 2011. find
  5. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In NAACL 2012. find
  6. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL 2011. find
  7. Michel Galley and C. D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In EMNLP 2008, pages 848–856, Hawaii, October. find
  8. Jianfeng Gao and Xiaodong He. 2013. Training mrf-based phrase translation models using gradient ascent. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 450–459, Atlanta, Georgia, June. Association for Computational Linguistics. find
  9. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In EMNLP 2011. find
  10. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 605–612, Barcelona, Spain, July. find our card
  11. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, July. ACL. find our card