Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.wmt-1.2.pdf

Summary

LLMs now produce much of the output of modern MT systems. The WMT24 Metrics Shared Task therefore tested whether automatic MT evaluation metrics still judge such translations reliably. The organizers took paragraph-level outputs from the WMT24 General MT Shared Task for three directions: English→German, English→Spanish (Latin America) and Japanese→Chinese. Expert MQM error annotations, provided by Google and Unbabel, served as the gold standard. Both reference-based and reference-free (QE) metrics were scored on how often they agreed with human judgments. The meta-evaluation dropped Pearson correlation in favour of pairwise ranking accuracy, which better matches how metrics are used in practice. Final ranks came from a weighted average over six tasks. Separately, a challenge-set subtask let external teams probe specific metric weaknesses. Trained neural metrics again clearly beat surface-overlap measures such as BLEU, spBLEU and chrF, and the organizers found no sign that they struggle with LLM-generated translations. The ESA human evaluation confirmed these conclusions, which supports continued use of fine-tuned metrics for modern MT.

Key points

• Meta-evaluation: system-level scoring used soft pairwise accuracy, and segment-level scoring used pairwise accuracy with tie calibration. All data and results were computed with and released via MTME.
• Human annotation:
  • Google's professional translators annotated en→de (10 raters) and ja→zh (8 raters), using pSxS rater assignment.
  • Unbabel annotated en→es following its MQM 3.0 typology, with "Critical" errors mapped to Major.
• Winners and scores: MetricX-24-Hybrid and XCOMET were named the winners. MetaMetrics-MT shared the top rank cluster (average score about 0.72), while BLEU averaged roughly 0.59.
• Challenge sets: four were submitted:
  • AfriMTE: African languages, 13 directions.
  • BioMQM: biomedical domain, 4,641 items.
  • DFKI: 137,000 items covering more than 100 linguistic phenomena, for en→de and en→ru.
  • MSLC24: low- to mid-quality MT plus corner cases such as empty strings and wrong or mixed language output.
• Challenge-set findings: trained metrics may overfit to WMT domains (BioMQM). Extremely low-resource pairs such as English–Luo and English–Twi remain difficult.