📄 Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation

Summary

The paper describes a POSTECH/Chonbuk system for the WMT17 shared task on translation quality estimation (QE). QE predicts how good a machine translation is, without a reference, at the sentence, word, and phrase levels. The system builds on the authors' two-stage Predictor-Estimator. In this design, an attention-based bidirectional RNN word predictor is trained on large parallel corpora (Europarl, common crawl, news commentary, rapid, plus src-pe pairs). It produces QE feature vectors (QEFVs), which a Bi-RNN quality estimator uses.

Previously the two stages were trained separately. The authors instead apply stack propagation, alternating updates between the word-prediction and QE objectives so that gradients reach the predictor. They also add multilevel task learning, which shares common network parts across all three QE granularities. The rationale is that every level's labels come from the same HTER-based comparison with post-edits.

Single models improve step by step from these additions. Ensembles of 15 instances, varied by dimensionality and data shuffling, took first place in every WMT17 QE subtask for English-German and German-English. This shows the value of jointly training pre-task and task networks.

Key points

• Three variants are compared: the base Predictor-Estimator (Pre&Post-QEFV/Bi-RNN), a single-level Stackprop variant, and a MultiLevel Stackprop variant. The last one was consistently best. For example, sentence-level Pearson's r on the En-De dev set rose from 0.6436 to 0.6785.
• For phrase-level QE in the multilevel setup, a word-level architecture is used. A phrase is labeled BAD if any word inside it is tagged BAD.
• The main single-model configuration used a 70,000-word vocabulary, 500-dimensional embeddings, 700 hidden units in the predictor, and 100 hidden units in the estimator.
• The best submission was PredictorEstimator-Combined-MultiLevel-Ensemble. On the En-De test set it reached Pearson's r 0.6954 (baseline 0.397), word F1-mult 0.5679 (baseline 0.361), and phrase F1-mult 0.5859 (baseline 0.327).
• On the De-En test set, the same ensemble scored Pearson's r 0.7280, word F1-mult 0.5347, and phrase F1-mult 0.5611, ranking first in all subtasks.