Elamur Research⌕
☰
home / papers / 2019 / Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

Summary

Automatic MT metrics are usually judged by a single global correlation with human scores. That number says little about where a metric succeeds or fails. Fomicheva and Specia run a broad meta-evaluation of many sentence-level metrics, mainly using WMT16 Metrics Task data. The metrics range from BLEU, TER and chrF to Meteor, UPF-Cobalt, BEER and trained combinations such as DPMFcomb. The study looks at three factors: the quality level of the translations, the type of MT system, and the kind of human judgment used as gold standard.

To examine behaviour across quality levels, they adopt a local dependency measure from econometrics (Tjøstheim and Hufthammer 2013). Metrics prove weakest on poor translations, not on good ones as was commonly assumed. Low-quality output is also harder for human annotators to score consistently. Metrics agree more with humans on neural MT than on phrase-based or syntax-based statistical MT. Metric rankings mostly hold across adequacy and fluency judgments. These findings show where metric development should focus and how far meta-evaluation conclusions can be generalised.

Key points

  • On WMT16 direct assessment data, Spearman correlation drops sharply for the low-quality half of the data. Meteor falls from 0.448 (high-quality half) to 0.297, and BLEU-4 from 0.381 to 0.173. Overall correlations are 0.565 for Meteor and 0.456 for BLEU-4.
  • Neural MT outputs are evaluated more reliably than statistical MT outputs:
    • Meteor: 0.486 on PBMT vs 0.613 on NMT.
    • BLEU-4: 0.388 on PBMT vs 0.497 on NMT.
    • The same pattern appears in the MTSummit17 data set.
  • The authors attribute the NMT advantage to differences in how error types are distributed across system outputs.
  • Gold standards compared include WMT16 DA and RR, WMT17, MTC adequacy and fluency, GALE, and EAMT. Adequacy (discrete or continuous scale) and fluency judgments yield similar metric orderings.
  • Post-editing effort judgments (EAMT-PE) reorder the metrics. For example, WER reaches the top correlation there (0.748). A likely reason is that some adequacy-critical errors, such as negation, are cheap to fix.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

20 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00356

@article{fomicheva2019taking,
  title = {{Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments}},
  author = {Fomicheva, Marina and Specia, Lucia},
  journal = {Computational Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00356},
  url = {https://doi.org/10.1162/coli_a_00356}
}

References

81 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amigó, Enrique, Jesús Giménez, Julio Gonzalo, and Felisa Verdejo. 2009. The contribution of linguistic features to automatic machine translation evaluation. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 306–314, Singapore. MTC -A MTC -F GALE EAMT EAMT -PE .431 .421 .390 .414 .380 .308 .301 .245 .336 .237 .243 .229 .209 .192 .185 .207 .184 .192 .461 .441 .379 .376 .372 .346 .367 .360 .325 .261 .259 .223 .213 .201 .228 .220 .223 .185 .719 .717 .712 .698 .696 .707 .737 .748 .693 .436 .360 .396 .214 .256 .251 .206 .205 .139 .165 .458 .383 .368 .307 .276 .311 .230 .201 .213 .083 .710 .704 .697 .611 .380 .405 .215 .369 .194 .719 find
  2. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR 2015, pages 1–15, San Diego, CA. ↗ our card
  3. Baig, Taimur and Ilan Goldfajn. 1999. Financial market contagion in the Asian crisis. IMF Staff Papers, 46(2):167–195. ↗
  4. Banerjee, Satanjeev and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Association for Computational Linguistics (ACL) Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, volume 29, pages 65–72, Ann Arbor, MI. find our card
  5. Berentsen, Geir Drage, Tore Selland Kleppe, Dag Tjøstheim, et al. 2014. Introducing localgauss, an R package for estimating and visualizing local Gaussian correlation. Journal of Statistical Software, 56(1):1–18. find
  6. Bertero, Elisabetta and Colin Mayer. 1990. Structure and performance: Global interdependence of stock markets around the crash of October 1987. European Economic Review, 34(6):1155–1180. ↗
  7. Blatz, John, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In Proceedings of the 20th Conference on Computational Linguistics, pages 315–321, Geneva. find
  8. Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169–214, Copenhagen. find our card
  9. Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor and Marcos Zampieri. 2016a. Findings of the 2016 Conference on Machine Translation. In First Conference on Machine Translation, Volume 2: Shared Task Papers, WMT, pages 131–198, Berlin. find our card
  10. Bojar, Ondřej, Christian Federmann, Barry Haddow, Philipp Koehn, Lucia Specia, and Matt Post. 2016b. Ten years of WMT evaluation campaigns: Lessons learnt. In Workshop on Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 27–36, Portoroz. find
  11. Bojar, Ondřej, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 Metrics Shared Task. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 489–513, Copenhagen. find our card
  12. Bojar, Ondřej, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016c. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation, pages 199–231, Berlin. find our card
  13. Burges, Chris, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, pages 89–96, Bonn. find our card
  14. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague. find our card
  15. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F. Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17–53, Uppsala. find our card
  16. Callison-Burch, Chris, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. EACL, 6:249–256, Trento. find
  17. Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. 2017. Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics, 108(1):109–120. ↗
  18. Charniak, Eugene and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 173–180, Ann Arbor, MI. find our card
  19. Coughlin, Deborah. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX, pages 63–70, New Orleans, LA. find
  20. Culy, Christopher and Susanne Z. Riehemann. 2003. The limits of N-gram translation evaluation metrics. In Proceedings of MT Summit IX, pages 71–78, New Orleans, LA. find
  21. Delicado, Pedro and Marcelo Smrekar. 2009. Measuring non-linear dependence for two random variables distributed along a curve. Statistics and Computing, 19(3):255–269. ↗
  22. Denkowski, Michael and Alon Lavie. 2010. Choosing the right evaluation for machine translation: An examination of annotator and automatic metric performance on human judgment tasks. In Proceedings of the Ninth Biennial Conference of the Association for Machine Translation in the Americas (AMTA), Denver, CO. find
  23. Denkowski, Michael and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, MD. find our card
  24. Doddington, George. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145, San Diego, CA. find
  25. Doksum, Kjell, Stephen Blyth, Eric Bradlow, Xiao-Li Meng, and Hongyu Zhao. 1994. Correlation curves as local measures of variance explained by regression. Journal of the American Statistical Association, 89(426):571–582. ↗
  26. Fomicheva, Marina and Núria Bel. 2016. Using contextual information for machine translation evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, pages 2755–2761, Portorozž. find
  27. Fomicheva, Marina, Núria Bel, Iria da Cunha, and Anton Malinovskiy. 2015. UPF-Cobalt Submission to WMT15 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 373–379, Lisbon. find
  28. Fomicheva, Marina, Núria Bel, Lucia Specia, Iria da Cunha, and Anton Malinovskiy. 2016. CobaltF: A Fluent Metric for MT Evaluation. In Proceedings of the First Conference on Machine Translation, pages 483–490, Berlin. find
  29. Forbes, Kristin J. and Roberto Rigobon. 2002. No contagion, only interdependence: Measuring stock market comovements. The Journal of Finance, 57(5):2223–2261. ↗
  30. Giménez, Jesús and Lluis Marquez. 2004. Fast and accurate part-of-speech tagging: The SVM approach revisited. Recent Advances in Natural Language Processing III, pages 153–162. find
  31. Giménez, Jesús and Lluı́s Màrquez. 2010a. Asiya: An open toolkit for automatic machine translation (meta-)evaluation. Prague Bulletin of Mathematical Linguistics, (94):77–86. ↗
  32. Giménez, Jesús and Lluı́s Màrquez. 2010b. Linguistic measures for automatic machine translation Evaluation. Machine Translation, 24(3-4):209–240. ↗
  33. Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia. find
  34. Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. find
  35. Graham, Yvette, Nitika Mathur, and Timothy Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of NAACL-HLT, pages 1183–1191, Denver, CO. find our card
  36. Gupta, Rohit, Constantin Orasan, and Josef van Genabith. 2015. ReVal: A simple and effective machine translation evaluation metric based on recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1066–1072, Lisbon. find
  37. Hjort, Nils Lid and M. Chris Jones. 1996. Locally parametric nonparametric density estimation. Annals of Statistics, 48:1619–1647. ↗
  38. Hong, Yongmiao, Jun Tu, and Guofu Zhou. 2006. Asymmetries in stock returns: Statistical tests and economic evaluation. Review of Financial Studies, 20(5):1547–1581. ↗
  39. Jones, M. C. and I. Koch. 2003. Dependence maps: Local dependence in practice. Statistics and Computing, 13(3):241–255. ↗
  40. Junczys-Dowmunt, Marcin, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready for deployment? A case study on 30 translation directions. In Proceedings of 13th International Workshop on Spoken Language Translation, volume 1. Seattle, WA. find our card
  41. Kim, Hyun, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 562–568, Copenhagen. find our card
  42. Koehn, Philipp and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of the Workshop on Statistical Machine Translation, pages 102–121, New York, NY. find our card
  43. Levene, Howard. 1961. Robust tests for equality of variances. I. Olkin, S. G. Ghurge, W. Hoeffding, W. G. Madow, and H. B. Mann, editors, Contributions to Probability and Statistics. Essays in Honor of Harold Hotelling, pages 279–292. find
  44. Lin, Chin-Yew and Franz Josef Och. 2004a. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 605–612. find our card
  45. Lin, Chin-Yew and Franz Josef Och. 2004b. Orange: A Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507. find our card
  46. Lin, Dekang. 2003. Dependency-based evaluation of minipar, In A. Abeillé, editor, Treebanks. Springer, pages 317–329. find
  47. Liu, Ding and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32, Ann Arbor, MI. find our card
  48. Lommel, Arle Richard, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 12:455–463. ↗
  49. Longin, Francois and Bruno Solnik. 2001. Extreme correlation of international equity markets. Journal of Finance, 56(2):649–676. ↗
  50. Machácek, Matouš and Ondrej Bojar. 2014. Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD. find our card
  51. Melamed, Dan, Ryan Green, and Joseph P. Turian. 2003. Precision and recall of machine translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL 2003–short papers-Volume 2, pages 61–63, Edmonton. find
  52. Moore, Robert C. and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220–224, Uppsala. find our card
  53. Nießen, Sonja, Franz Josef Och, Gregor Leusch, Hermann Ney et al. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00). pages 39–45, Athens. find our card
  54. Padó, Sebastian, Michel Galley, Dan Jurafsky, and Chris Manning. 2009. Robust machine translation evaluation with entailment features. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 297–305, Singapore. find
  55. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311–318, Philadelphia, PA. find our card
  56. Popovic, Maja. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon. find our card
  57. Popovic, Maja. 2016. CHRF deconstructed: β parameters and n-gram weights. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, 2, pages 499–504, Berlin. find
  58. Popović, Maja and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657–688. ↗
  59. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation, pages 371–376, Berlin. find our card
  60. Shen, Shiqi, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1683–1692, Berlin. find our card
  61. Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, volume 200, pages 223–231, Cambridge, MA. find our card
  62. Snover, Matthew G, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009. TER-Plus: Paraphrase, semantic, and alignment enhancements to translation edit rate. Machine Translation, 23(2-3):117–127. ↗
  63. Specia, Lucia. 2011. Exploiting objective annotations for measuring translation post-editing effort. In 15th Conference of the European Association for Machine Translation, pages 73–80, Leuven. find
  64. Specia, Lucia, Najeh Hajlaoui, Catalina Hallett, and Wilker Aziz. 2011. Predicting machine translation adequacy. In Machine Translation Summit XIII, pages 513–520, Xiamen. find
  65. Specia, Lucia, Kim Harris, Aljoscha Burchardt, Marco Turchi, Matteo Negri, and Inguna Skadina. 2017. Translation quality and productivity: A study on rich morphology languages. In Proceedings of the 16th Machine Translation Summit, pages 55–71, Nagoya. find
  66. Specia, Lucia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the sentence-level quality of machine translation systems. In 13th Annual Conference of the European Association for Machine Translation, pages 28–37, Barcelona. find our card
  67. Stanojević, Miloš and Khalil Sima’an. 2014. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore, MD. find
  68. Stanojević, Miloš, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273, Lisbon. find
  69. Støve, Bård, Dag Tjøstheim, and Karl Ove Hufthammer. 2014. Using local Gaussian correlation in a nonlinear re-examination of financial contagion. Journal of Empirical Finance, 25:62–82. ↗
  70. Surdeanu, Mihai and Jordi Turmo. 2005. Semantic role labeling using complete syntactic analysis. In Proceedings of the Ninth Conference on Computational Natural Language Learning, pages 221–224, Ann Arbor, MI. find
  71. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. ↗
  72. Tillmann, Christoph, Stephan Vogel, Hermann Ney, Arkaitz Zubiaga, and Hassan Sawaf. 1997. Accelerated DP based search for statistical translation. In Fifth European Conference on Speech Communication and Technology, pages 2667–2670, Rhodes. find
  73. Tjøstheim, Dag and Karl Ove Hufthammer. 2013. Local Gaussian correlation: A new measure of dependence. Journal of Econometrics, 172(1):33–48. ↗
  74. Tong, Yung Liang. 1990. The Multivariate Normal Distribution. Springer-Verlag, New York. ↗
  75. Toral, Antonio, Sudip Naskar, Federico Gaspari, and Declan Groves. 2012. DELiC4MT: A tool for diagnostic MT evaluation over user-defined linguistic phenomena. Prague Bulletin of Mathematical Linguistics, 98:121–131. ↗
  76. Toral, Antonio and Vı́ctor M. Sanchez-Cartagena. 2017. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1063–1073, Valencia. find
  77. Williams, Evan James. 1959. Regression Analysis, volume 14. Wiley, New York. find
  78. Williams, Philip, Rico Sennrich, Maria Nadejde, Matthias Huck, Barry Haddow, and Ondrej Bojar. 2016. Edinburgh’s statistical machine translation systems for WMT16. In Proceedings of the First Conference on Machine Translation, pages 399–410, Berlin. find
  79. Yu, Hui, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015. CASICT-DCU participation in WMT2015 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 417–421, Lisbon. find
  80. Zhang, Hao and Daniel Gildea. 2007. Factorization of synchronous context-free grammars in linear time. In Proceedings of the NAACL-HLT 2007/AMTA Workshop on Syntax and Structure in Statistical Translation, pages 25–32, Rochester, NY. find
  81. Zhang, Lilin, Zhen Weng, Wenyan Xiao, Jianyi Wan, Zhiming Chen, Yiming Tan, Maoxi Li, and Mingwen Wang. 2016. Extract domain-specific paraphrase from monolingual corpus for automatic evaluation of machine translation. In Proceedings of the First Conference on Machine Translation, pages 511–517, Berlin. find

↑ back to top