📄 The Effect of Translationese in Machine Translation Test Sets

Summary

WMT news test sets are typically built so that half of the source sentences were originally written in the source language and the other half are human translations from the target language (translationese). Earlier work on the WMT 2017 Chinese→English set found that the translationese half was easier for MT systems to translate. That finding concerned a single test set and direction. Zhang and Toral therefore test whether the effect holds more broadly. They recompute direct assessment (DA) human evaluation scores for 17 translation directions from the WMT16–18 news tasks, splitting each test set by the original language of the source documents. They then compare system scores and rankings on the full set, the original-source subset (ORG) and the translationese-source subset (TRS). Across directions, scores on TRS are inflated and scores on ORG are deflated relative to the official results, and rankings sometimes shift. The size of this distortion is strongly and negatively correlated with the quality the best systems reach, while language relatedness explains little. These findings matter for fair MT evaluation and for claims of human parity.

Key points

• Data and method: The study uses newstest2016–2018 with standardized DA (z-)scores. Ranking agreement between the full set and ORG is measured with Kendall's τ, and system clusters follow WMT's Wilcoxon rank-sum procedure.
• Consistent inflation: For the top systems, ORG scores fall below the official WMT score and TRS scores rise above it almost everywhere. English→Finnish at WMT18 is a striking case, at roughly −7.7 on ORG versus +8.0 on TRS.
• Rankings: In Chinese→English, most top systems move position and clusters shift. A perfect τ of 1 appears mainly in directions with few submitted systems.
• Quality correlation: At WMT18, the best system's ORG score correlates negatively with the gap between the full set and ORG (Pearson R = −0.84 relative, −0.78 absolute, p < 0.001). Directions translating out of English tend to show larger gaps.
• Language similarity: Similarity was computed with lang2vec/URIEL features and showed only weak, non-significant correlations (R ≈ −0.11 and −0.15). Code and data are released on GitHub.