Results of the WMT16 Metrics Shared Task

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2302.pdf

Summary

The WMT16 Metrics Shared Task measures how well automatic MT evaluation metrics, which compare system outputs to reference translations, agree with human judgments of quality. Participants scored outputs from the WMT16 News, IT, Tuning and HimL translation systems: 171 outputs across 18 language pairs pairing English with 12 languages, in three domains (news, IT, medical/health). Metrics were assessed at two levels. At system level, they were correlated with official manual rankings. At segment level, they were checked for agreement with humans on individual sentences. This edition introduced three kinds of human gold standard: Relative Ranking (RR) by researchers, crowd-sourced Direct Assessment (DA) of adequacy, and HUME semantic judgments. It also used a new way to compute confidence intervals. To make system-level conclusions more decisive, the organisers generated large sets of hybrid systems. Character-level metrics such as BEER, chrF and CharacTER performed strongly again. Trained metrics also correlated highly with humans, and BLEU was clearly outperformed by many alternatives. The authors argue that MT development should look beyond BLEU and encourage adding new metrics to the Moses scorer.

Key points

• Participation and baselines: 16 metrics came from 9 research groups. For comparison, the organisers also ran several baseline metrics of their own, among them the error-rate measures WER, PER, TER and CDER as well as BLEU, its sentence-level variant SentBLEU, and NIST.
• Test sets and official methods: The test sets were newstest2016, it-test2016 (1,000 sentences) and himl2015 (about 340 judged sentences per pair). The official evaluation still relies on RR, using TrueSkill at system level and Kendall's τ at segment level.
• Hybrid systems: 10K hybrid systems per language pair were built by combining outputs of two newstest2016 systems. This was applied to DA evaluation only.
• Limits of crowd-sourced DA: DA could be collected only for English and Russian as target languages. At least 15 assessments per segment were used, and repeat judgments from the same worker still correlated above 0.9 with judgments from distinct workers.
• HUME segment-level results: In the Williams significance test, chrF3 was the sole winner for en-cs and en-ro. Its Pearson r reached .544 on en-cs and .639 on en-ro.