Elamur Research⌕
☰
home / papers / 2023 / Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

Paper · 2023 · Daniel Deutsch, George Foster and Markus Freitag

Summary

Segment-level evaluation of machine translation metrics usually relies on some form of Kendall's τ. Each variant handles tied scores differently, and that choice is becoming important. Stronger MT systems produce many error-free outputs, and MQM annotations use integer error counts. Both produce frequent ties in human scores. Newer metrics such as GEMBA (LLM prompting) and MaTESe also output only a few discrete values. Using WMT'22 MQM ratings for en→de, zh→en and en→ru, the authors show three things:

  • No existing τ variant handles simple tie situations well.
  • WMT's τ variants never reward a metric for correctly predicting a tie.
  • Some variants can be gamed.

They propose two fixes. The first is acc_eq, a pairwise accuracy that counts matching ties as correct. The second is tie calibration, which searches for a threshold ε below which two metric scores count as tied. This lets metrics with continuous outputs compete fairly with discrete ones. The combination, acc*_eq, ranks metrics without systematic bias toward or against ties. The approach should carry over to evaluating other generative-AI metrics.

Key points

  • Data and ties: The study uses WMT'22 MQM data with 13–15 systems per language pair. Under group-by-item grouping, 42–53% of human-score pairs are ties, and most of these are pairs of error-free translations.
  • Ties by metric type: Regression metrics (Metric-X, COMET-22) tie on roughly 0.1–1.3% of pairs. MaTESe and the GEMBA metrics tie on about 40–81% of pairs.
  • Gaming τ₁₀: Randomly breaking ties raises τ₁₀ for MaTESe from −0.459 to about 0.15, with a similar gain for GEMBA-GPT-3.5. COMET-22 improves by only about 0.005. Under acc*_eq, MaTESe and the GEMBA metrics move up from the bottom of the ranking to 4th, 6th and 15th.
  • Transfer of ε: With WMT'21 used as held-out data, ε transfers well for en-de, with about a 2% relative change in accuracy. It fails for zh-en, where the share of ties differs between years (23% vs 41%).
  • Where ties land and code: Calibrated ties mostly fall on high-scoring, near-perfect translations. Code is released in google-research/mt-metrics-eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Daniel Deutsch, George Foster and Markus Freitag. 2023. Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.798

@inproceedings{deutsch2023ties,
  title = {{Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration}},
  author = {Daniel Deutsch, George Foster and Markus Freitag},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.798},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.798}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 Metrics Shared Task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics. our card
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. Advances in neural information processing systems, 33:1877–1901. our card
  3. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metri… our card
  4. Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-Aware Decoding for Neural Machine Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… our card
  5. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. our card
  6. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics. Transactions of the Association for Computational Linguistics, 10:811– 825. our card
  7. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust. In Proceedings of the Seventh Confere… our card
  8. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain. In Proceedings of the Sixth Conference on Machine Trans… our card
  9. Maurice G Kendall. 1938. A New Measure of Rank Correlation. Biometrika, 30(1/2):81–93. find
  10. Maurice G Kendall. 1945. The Treatment of Ties in Ranking Problems. Biometrika, 33(3):239–251. find
  11. Tom Kocmi and Christian Federmann. 2023. Large Language Models Are State-of-the-Art Evaluators of Translation Quality. find
  12. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478–494, Online.… find
  13. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics. Tradumàtica, (12):0455–463. find
  14. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational L… our card
  15. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics. find
  16. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics. our card
  17. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997. our card
  18. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages… our card
  19. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. our card
  20. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  21. Alan Stuart. 1953. The Estimation and Comparison of Strengths of Association in Contingency Tables. Biometrika, 40(1/2):105–110. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deutsch-2023-ties-matter-meta-evaluating-modern/