📄 Linguistic Evaluation of German-English Machine Translation Using a Test Suite

Summary

Generic metrics and human rankings show how good a machine translation system is overall, but not which grammatical constructions it handles badly. The authors address this for German→English. They apply the manually built DFKI test suite to the 16 systems in the WMT19 news translation task. The suite has 5560 source sentences covering 107 mostly grammatical phenomena in 14 categories. Each system output is checked by hand-written positive and negative regular expressions in the TQ-AutoTest tool. Outputs that match neither, or both, are flagged as warnings and resolved by a human annotator, who also refines the rules. Accuracy is the share of correctly translated items per phenomenon, and system differences are checked with a one-tailed Z-test. Even strong neural systems still get roughly a quarter of the items wrong. Idioms, modal constructions, pseudo-clefts, MWE and verb valency are the weakest areas. Compared with WMT18, function words, non-verbal agreement and punctuation improved. The study shows how fine-grained diagnostic testing exposes specific weaknesses that aggregate scores hide.

Key points

• About 10% of outputs were initially flagged as warnings. One linguistics student spent roughly 70 hours reducing this to 3%, leaving 5393 fully valid test items.
• Average accuracy across all items was 75.6%. Idioms reached only 11.6% and resultative predicates 17.8%, while MWE and verb valency were around 66%.
• Facebook was in the top significance cluster for 11 categories, averaging 87.5% when categories are weighted equally. RWTH and Online-A led when all items are weighted equally.
• Overall accuracy improved by about 6% over WMT18. The largest gains were in function words (+12.5%), non-verbal agreement (+9.7%) and punctuation (+8%).
• Among systems submitted under the same name in both years, Online-G improved most (+18.7%). UCAM declined in several categories.