📄 To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translation

Summary

Machine translation output is often described as more literal than human translation, but this gap had rarely been measured at a fine structural level. Luo, Cherry and Foster quantify it through morphosyntactic divergence, meaning cases where the target structure departs from the source. For each of En→De, En→Zh and En→Fr, they hold out one million WMT sentence pairs. They parse both sides using the Universal Dependencies scheme, align words automatically, and compare thousands of patterns in human references and in outputs of bilingual Transformer-base systems. Two pattern types are used: word-based (POS plus parent and child dependencies) and arc-based (a source dependency arc versus the corresponding target path). In every pair, MT proves more cautious. It shows less structural variety, more convergent patterns and more one-to-one alignments, and it drifts further from HT for rarer source patterns. Experiments with different decoders identify beam search as the main driver of this pull toward convergence. Finally, most frequent divergences in references coincide with lower MT scores. The work matters for building more natural MT and for interpreting evaluation against human references.

Key points

• Data and models. The study uses WMT19 for En→De and En→Zh and WMT15 for En→Fr. Each training set was lightly filtered and deduplicated, and a 1M-sentence analysis subset was held out to avoid domain mismatch. The systems reach BLEU scores of 39.9 (Fr, newstest2014), 46.3 (De, newstest2018) and 34.4 (Zh, newstest2018).
• Tooling. Dependency parsers were trained on UD treebanks (EWT for English, GSD for French, German and Chinese). The automatic word aligner achieves an F1 of about 83.6 (Fr) and 83.7 (Zh).
• Control verbs as a case study. For English control constructions (xcomp) translated into French, MT keeps the source structure roughly 20% more often than human translators do.
• Role of beam search. The bias toward convergence comes from beam search, not from ancestral sampling. It is strongest when the convergent rendering occurs in about half of the training examples for a source pattern.
• Effect on translation quality. Experiment and control groups were built for each divergence, requiring at least 100 sentences per group. Divergences in HT are often linked to lower BLEU and BLEURT, but pattern frequency explains this only weakly. Pearson correlations with BLEURT are about 0.32 for Fr and 0.29 for De, and there is no correlation for Zh.