📄 Neural Machine Translation with Extended Context

Summary

Conventional NMT systems translate each sentence in isolation, so they lose cues from neighbouring sentences that are needed for discourse phenomena such as pronoun reference. Tiedemann and Scherrer present a pilot study that tests whether a standard attention-based encoder-decoder can use wider context when the architecture stays unchanged and only the training and test segments are enlarged. They use German→English movie subtitles from OpenSubtitles2016 and the Helsinki NMT system (HNMT). A sentence-level baseline is compared with two variants. In 2+1, the previous source unit is prepended, either with its tokens tagged by a cc prefix or separated by break tokens. In 2+2, both source and target segments are joined with their predecessors using a BREAK marker. The extended models stay robust in translation quality and learn to tell context apart from the segment being translated. They also attend across sentence boundaries in plausible ways. A manual check of the ambiguous pronoun sie/Sie shows small gains, mainly for plural readings, but these are not statistically significant. The study offers a simple, architecture-free starting point for document-level NMT.

Key points

• Data: OpenSubtitles2016 German–English covers 16,910 movies and TV series, with 400 movies held out for development and testing. The corpus has about 13.9M translation units (roughly 8 tokens each), of which 13.5M are training instances. The first 5,000 test units are used for automatic evaluation.
• Setup: HNMT with BPE subwords is used. The 2+1 models use 30k BPE codes and a 60k model vocabulary, because the cc-prefixed context words double the vocabulary. The 2+2 model uses joint 60k BPE codes. Each system is decoded as an ensemble of its last four savepoints. BLEU and chrF3 serve as automatic metrics, and for 2+2 only the final segment is scored.
• Attention: In the extended-source model, about 7.1% of the total attention weight goes to the preceding context. This share is small but not negligible.
• Pronoun evaluation: 565 of the 1,143 sie/Sie occurrences were judged manually. Overall accuracy is 90.1% for the baseline and 91.7% for 2+2. For plural readings it rises from 69.8% to 79.1%. None of the differences is significant under χ² tests at p = 0.05.
• Limitations: A one-unit context window can miss antecedents that appear further back, which sometimes causes errors the baseline avoids. It is also not yet clear that the observed gains come from cross-segment attention.