Elamur Research
☰
home / papers / 2014 / Results of the WMT14 Metrics Shared Task

Results of the WMT14 Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W14-3336.pdf

Summary

This report covers the WMT14 Metrics Shared Task, a yearly evaluation of how well automatic machine translation metrics agree with human judgements. Developers scored the outputs of the WMT14 Shared Translation Task: 110 system outputs across 10 directions, English to and from Czech, French, German, Hindi and Russian, with about 3003 sentences per output in most directions. The organizers received 23 metrics from 12 groups and added six baselines (BLEU, NIST, WER, PER, TER, CDER). They compared the metrics against the official human ranking at two levels:

  • System level: Pearson correlation, replacing the earlier Spearman's ρ, with bootstrap confidence intervals.
  • Segment level: a revised Kendall's τ that changes how ties are handled.

Many metrics again beat BLEU. DiscoTK-party-tuned led into-English evaluation at both levels. BEER was the strongest segment-level metric out of English. Segment-level agreement remained much weaker than system-level agreement. The task offers a standard benchmark for choosing and improving MT metrics, and it fixes evaluation flaws that some metrics could exploit.

Key points

  • Pearson switch: System-level evaluation moved to Pearson's r because Spearman's ranks ignore how close systems' human scores are. Confidence intervals came from 1000 bootstrap samples at the 95% level.
  • Into English, system level: DiscoTK-party-tuned had the best average (Pearson about .944), and LAYERED came second while also winning Hindi-English. METEOR, the previous year's winner, degraded badly for non-Latin source scripts (Russian, Hindi).
  • Out of English, system level: English-German was left out of the averages because its 18 similar-quality systems made results unreliable. NIST had the best average Pearson, and CDER was best by Spearman's ρ.
  • Segment level: DiscoTK-party-tuned won 4 of 5 into-English directions. BEER, then METEOR, led out of English. The best τ in any language pair was only about 0.45.
  • Tie handling: The new Kendall's τ variant stops rewarding ties. DiscoTK-light-kool, which gives most segments identical scores, scored near zero under it but would have ranked first under the WMT13 method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 58 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3336

@inproceedings{machacek2014results,
  title = {{Results of the WMT14 Metrics Shared Task}},
  author = {Macháček, Matouš and Bojar, Ondřej},
  booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},
  year = {2014},
  doi = {10.3115/v1/w14-3336},
  url = {https://doi.org/10.3115/v1/w14-3336}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barančı́ková, P. (2014). Parmesan: Improving Meteor by More Fine-grained Paraphrasing. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  2. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation. find our card
  3. Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10– 51, Montréal, Canada. Association for Computational Linguistics. find our card
  4. Chen, B. and Cherry, C. (2014). A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find our card
  5. Comelles, E. and Atserias, J. (2014). VERTa participation in the WMT14 Metrics Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  6. Denkowski, M. and Lavie, A. (2014). Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find our card
  7. Doddington, G. (2002). Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, HLT ’02, pages 138–145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  8. Echizen’ya, H. (2014). Application of Prize based on Sentence Length in Chunk-based Automatic Evaluation of Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  9. Gautam, S. and Bhattacharyya, P. (2014). LAY-ERED: Description of Metric for Machine Translation Evaluation in WMT14 Metrics Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  10. Gonzàlez, M., Barrón-Cedeño, A., and Màrquez, L. (2014). IPA and STOUT: Leveraging Linguistic and Source-based Features for Machine Translation Evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  11. Guzman, F., Joty, S., Màrquez, L., and Nakov, P. (2014). DiscoTK: Using Discourse Structure for Machine Translation Evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  12. Koehn, P. and Monz, C. (2006). Manual and automatic evaluation of machine translation between european languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121, New York City. Association for Computational Linguistics. find
  13. Leusch, G., Ueffing, N., and Ney, H. (2006). CDER: Efficient MT Evaluation Using Block Movements. In In Proceedings of EACL, pages 241–248. find
  14. Libovický, J. and Pecina, P. (2014). Tolerant BLEU: a Submission to the WMT14 Metrics Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  15. Macháček, M. and Bojar, O. (2013). Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics. find
  16. Mahmoudi, A., Faili, H., Dehghan, M., and Maleki, J. (2013). ELEXR: Automatic Evaluation of Machine Translation Using Lexical Relationships. In Castro, F., Gelbukh, A., and González, M., editors, Advances in Artificial Intelligence and Its Applications, volume 8265 of Lecture Notes in Computer Science, pages 394– 405. Springer Berlin Heidelberg. find
  17. Papineni, K., Roukos, S., Ward, T., and jing Zhu, W. (2002). BLEU: a method for automatic evaluation of machine translation. pages 311–318. find our card
  18. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  19. Stanojevic, M. and Sima’an, K. (2014). BEER: A Smooth Sentence Level Evaluation Metric with Rich Ingredients. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find
  20. Vazquez-Alvarez, Y. and Huckvale, M. (2002). The reliability of the itu-t p.85 standard for the evaluation of text-to-speech systems. In Hansen, J. H. L. and Pellom, B. L., editors, IN-TERSPEECH. ISCA. find
  21. Wu, X. and Yu, H. (2014). RED, The DCU Submission of Metrics Tasks. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, USA. Association for Computational Linguistics. find