📄 QuEst - A translation quality estimation framework

Summary

The paper presents QuEst, an open-source toolkit for machine translation quality estimation (QE), meaning predicting how good a translated segment is when no human reference is available. Building QE models usually requires a lot of manual feature engineering. QuEst addresses this by bundling many feature extractors with a learning layer. A Java module computes sentence-level indicators from four sources: the source text, the MT output, external resources such as corpora, language models and GIZA++ alignments, and optionally the internals of the MT system. A separate Python module connects these features to scikit-learn and GPy for regression, classification, feature selection and grid search.

The authors tested it on nine datasets drawn from WMT12, EAMT11, EAMT09 and GALE11, covering en-es, fr-en and ar-en. Adding features to the standard 17-feature baseline usually lowered error. Feature selection, especially Gaussian Process–based ranking, often gave the best results. Combining system-dependent and system-independent features produced significantly lower errors. QuEst matters because it lets researchers concentrate on the learning side of QE instead of rebuilding features.

Key points

• QuEst has two independent parts:
  • a Java feature-extraction module with abstract classes for features, resources and preprocessing, controlled by configuration files;
  • a Python ML module built on scikit-learn (SVR, SVC, Lasso) and GPy (Gaussian Processes).
• Features are grouped into four families: complexity (source side), fluency (target side), adequacy (source–target comparison) and confidence (MT-system internals).
• Black-box (BB) features number 80–123 depending on the language pair. Glass-box (GB) features number 39–48 and rely on Moses-like outputs such as n-best lists and search-graph statistics. Some word-level features are also included.
• Four system types were compared: baseline (BL, 17 features), all features (AF), Randomised Lasso selection (FS(RL)), and GP selection (FS(GP), top 17). Errors were measured with MAE and RMSE.
• On WMT12, MAE dropped from 0.6802 (BL) to 0.6537 with FS(GP) using BB features, and to 0.6501 with FS(GP) using BB+GB features. GB features alone performed worse than BB features.

Interesting ideas

• I1. Black-box vs. glass-box split. Separating features that need access to the MT system from those that only look at the text lets the same tool run on outputs from unknown systems. It also allows richer signals to be added when the system is available. The same split could apply to any task that evaluates model outputs.
• I2. GP relevance for feature ranking. Per-feature RBF widths (an ARD kernel) in a Gaussian Process serve as importance scores, and a cheaper SVR is then trained on the top-ranked features. This turns a strong but expensive model into a feature selector for a simpler one.
• I3. Stability-style Lasso selection. The training data is resampled 1000 times, and a feature survives only when Lasso assigns it a weight other than zero in 25% or more of those fits. This makes selection more robust when features are numerous and correlated, a common situation with small annotated datasets.
• I4. Fixed feature budget for fair comparison. Limiting GP selection to the same number of features as the baseline (17) shows the gain comes from choosing better features, not from using more of them. The authors also note that 25–35 features worked best on most tasks.
• I5. Language model built from n-best lists. Training a small LM on the system's own n-best translations, then scoring the chosen output against it, measures how consistent the system is with its alternatives. This idea could carry over to confidence estimation in other generation tasks.