A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6307.pdf

Summary

Pronouns are a persistent difficulty for neural machine translation. Choosing the right target form often depends on an antecedent in an earlier sentence, and sentence-level systems cannot see it. Context-aware models usually show only small BLEU gains, but BLEU measures overall quality and says little about whether the extra context helps with pronouns. To address this, the authors built ContraPro, a large, automatically extracted English→German test suite for translating "it" as es, sie or er.

Candidate examples were filtered conservatively using several tools:

• word alignment with fast_align
• parsing with CoreNLP and ParZu
• coreference resolution with CoreNLP and CorZu

Each example pairs the correct translation with contrastive variants that use the wrong pronoun. Models are evaluated by checking whether they score the reference higher than these variants.

The authors tested RNN multi-encoder, concatenation and Transformer-based context models. BLEU changed little, but pronoun accuracy improved markedly, especially when the antecedent lay outside the current sentence. Sharing parameters between the main and context encoders also proved clearly helpful. The resource lets researchers measure discourse-level progress that standard metrics miss.

Key points

• ContraPro is freely available on GitHub (ZurichNLP) and contains 4,000 examples for each reference pronoun (es, sie, er). Metadata such as antecedent location and distance supports fine-grained analysis.
• The strongest model, s-hier-to-2.tied, gains about +16 percentage points of accuracy over the RNN baseline, while its BLEU scores on newstest2017 and newstest2018 change only slightly.
• Parameter tying raises accuracy by 5 points for s-hier-to-2 and by 4 points for s-t-hier. The authors suggest that context encoders get too weak a training signal when they are trained without sharing.
• Transformer models do best on intra-segmental antecedents, beating the RNN baseline by 9–18 points. The source-only model of Voita et al. (2018) outperforms the comparable RNN system, s-hier.tied.
• All evaluated models use one previous sentence as context. Many antecedents lie further back, so the authors leave variable-size context models for future work.