Unsupervised Quality Estimation for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.tacl-1.35.pdf

Summary

Quality estimation (QE) predicts how good a machine translation is when no reference is available. Leading supervised QE systems, however, depend on large parallel corpora and in-domain quality-labelled data, which exist for only a few languages. The authors instead treat the NMT model as a glass box. They derive unsupervised quality indicators directly from decoding: sequence-level output probabilities, the same probabilities refined with uncertainty quantification (Monte Carlo Dropout, ensembling), and Transformer attention weights. The approach requires no additional training or external resources.

For evaluation, they built a new sentence-level QE dataset from Wikipedia. It covers six language pairs spanning high-, medium- and low-resource settings, with 10K segments per pair scored by Direct Assessment (DA). The NMT systems that produced the translations are released alongside it. Indicators based on well-calibrated probabilities correlate with human DA about as well as strong supervised SOTA QE models. Multihead attention proved much harder to exploit. The work offers QE for languages lacking annotated data and provides the first resource for both black-box and glass-box QE.

Key points

• Dataset: 6 language pairs, 10K Wikipedia segments each, labelled with DA on a 1–100 scale rather than HTER/post-editing distance. The generating NMT models are included.
• Indicators: The paper proposes probability-based indicators (e.g., D-TP, which uses MC dropout) and attention-based ones. This is the first analysis of Transformer attention for unsupervised QE.
• System comparison (Et-En, 400 sentences, Fairseq, Pearson r with DA):
  • TP-Beam: 0.482
  • TP-Ensemble: 0.538, which also had the best translation quality (DA 61.19)
  • D-TP: 0.526
  • Sampling: 0.533, despite low quality (DA 42.02)
  • Diverse beam search (0.424) and MoE (0.449) brought no gain.
• Effect of training length: Over 60 training epochs, Meteor quality levels off while the link between model probabilities and quality weakens. Longer training therefore harms calibration.
• Recommendation: The authors recommend ensembling or MC dropout for NMT systems that use unsupervised QE. A separate test on 500 out-of-domain Et-En segments checks confidence under domain shift.