Improving the Transformer Translation Model with Document-Level Context

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1049.pdf

Summary

Standard Transformer translation handles each sentence in isolation, so it cannot resolve discourse-dependent issues such as ambiguous word senses, coreference or lexical cohesion. The authors add a separate context encoder that uses multi-head self-attention to represent the preceding source sentences of a document. Target-side history is deliberately left out to avoid propagating errors. Extra context-attention sub-layers then feed this representation into every encoder and decoder layer, with a gating mechanism in place of plain residual connections.

Document-aligned parallel data is scarce, so training happens in two stages. First, the ordinary sentence-level Transformer is fitted on large sentence corpora. Then those weights are frozen and only the new document-level components are learned on the smaller document corpus.

Tested on NIST Chinese-English and IWSLT French-English, the model beats the Transformer baseline and an adapted cache-based approach, with little extra computational cost. The work shows that document awareness can be added to Transformer cheaply and is useful even with limited document-level data.

Key points

• Reported gains over the Transformer baseline are +1.96 BLEU on Chinese-English (NIST) and +0.89 BLEU on French-English (IWSLT). The model also outperforms the cache-based method of Kuang et al. (2017) adapted to Transformer.
• Two-step training is essential:
  • Training only on the 940K-sentence document corpus gives 36.52 BLEU, below the sentence-level baseline's 39.53.
  • Combining 2M sentence pairs with the 940K document corpus reaches 47.93 BLEU, versus 45.97 for the baseline.
• Context helps in either location, with the best result when integrated into both:
  • encoder only: 47.51 BLEU
  • decoder only: 47.48 BLEU
  • both: 47.93 BLEU
• Context gating contributes about +0.38 BLEU compared with residual connections.
• Source code is released as Document-Transformer on GitHub (Glaceon31).