Elamur Research⌕
☰
home / papers / 2009 / Estimating the Sentence-Level Quality of Machine Translation Systems

Estimating the Sentence-Level Quality of Machine Translation Systems

Summary

The paper addresses confidence estimation (CE) for machine translation, which means judging the quality of an MT output sentence when no reference translation is available. Earlier work treated this as a binary good/bad decision. Specia, Turchi and colleagues instead frame it as regression that outputs a continuous sentence-level score. They compute 84 features and divide them into two groups. Black-box features need only the source, the target and optionally external corpora. Glass-box features draw on the decoder's internals and its n-best lists. The features are fed to Partial Least Squares regression, which handles strongly correlated inputs well, together with an explicit feature selection step. Experiments use three kinds of data: French-English Europarl translations labelled automatically with sentence-level NIST, WMT-2006 outputs from several SMT systems and one rule-based system with human adequacy judgements, and sets scored on a 1–4 quality scale or by post-editing time. Smaller selected feature subsets consistently outperform the full feature set and produce more stable models. On one dataset, the CE score agrees with human judgements much better than standard reference-based metrics. This shows that reference-free quality prediction is practical, for example to filter poor output before post-editing.

Key points

  • The feature set has 84 features in total, 54 of which are glass-box (e.g. SMT model score, phrase/word probabilities, n-best list statistics). Features that require parsers or WordNet were deliberately left out.
  • The NIST dataset contains about 6k French-English sentences from the WMT-2008 Europarl dev-test sets, each with a 1,000-best list. It was divided into 1,000 subsamples, each split 50/30/20 into train, validation and test.
  • The WMT datasets come from WMT-2006. They group outputs from three phrase-based systems (S1–S3) and one rule-based system (RB), each with roughly 1,300–2,000 sentences across 4–6 language pairs and 1–5 adequacy labels. These models use only black-box features plus source and target language indicators.
  • Prediction errors were modest. On the 1–4 datasets (P-ES-1 to P-ES-4), predictions deviate by about 0.6–0.7 points. On post-editing time (P-ER-1), the error is about 1.95 seconds per source word.
  • On P-ES-1gb, correlation with human annotation was 0.602 for the CE score, compared with BLEU-2 0.342, NIST 0.298, TER −0.263 and Meteor 0.376.

Interesting ideas

  • I1. Black-box vs. glass-box split. Sorting features by whether they need access to the system's internals lets the same quality estimator work across commercial, rule-based or otherwise opaque systems. The same split could help when evaluating third-party NLP tools.
  • I2. Predicting a continuous score instead of a pass/fail label. A continuous score avoids drawing an arbitrary quality threshold. It can also target practical quantities such as editing effort per word, which is more useful for workflow planning than a good/bad flag.
  • I3. Regression that tolerates collinear features, plus explicit selection. Partial Least Squares regression copes with many redundant features, and selecting a compact subset improved both accuracy and stability. This approach suits any task where many overlapping handcrafted features are available.
  • I4. Cheap surface-mismatch features. Comparing the source and output in terms of numbers, punctuation, length ratio and language-model fluency proved among the strongest quality signals. Such checks are easy to reuse in data-cleaning pipelines for parallel corpora.
  • I5. Pooling small annotated sets with language indicators. Small per-language-pair sets were merged into one dataset per system, with source and target language added as features. This makes it possible to train a single model when annotations for each pair are scarce.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 132 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{specia2009estimating,
  title = {{Estimating the Sentence-Level Quality of Machine Translation Systems}},
  author = {Specia, Lucia and Turchi, Marco and Cancedda, Nicola and Cristianini, Nello and Dymetman, Marc},
  booktitle = {Proceedings of the 13th Annual Conference of the European Association for Machine Translation},
  year = {2009},
  url = {https://aclanthology.org/2009.eamt-1.5/}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albrecht, J. and R. Hwa. 2007a. A re-examination of machine learning approaches for sentence-level mt evaluation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 880–887, Prague. find
  2. Albrecht, J. and R. Hwa. 2007b. Regression for sentence-level mt evaluation with pseudo references. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 296– 303, Prague. find
  3. Blatz, J., E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2003. Confidence estimation for machine translation. Technical report, Johns Hopkins University, Baltimore. find
  4. Blatz, J., E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence estimation for machine translation. In Proceedings of the 20th Conference on Computational Linguistics, pages 315–321, Geneva. find
  5. Callison-Burch, C., C. Fordyce, P. Koehn, C. Monz, and J. Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the 3rd Workshop on Statistical Machine Translation, pages 70–106, Columbus. find our card
  6. Doddington, G. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2nd Conference on Human Language Technology Research, pages 138– 145, San Diego. find
  7. Frenich, A. G., A. G. Jouan-Rimbaud, D. Massart, D. L. Kuttatharmmakul, S. Martinez Galera, and J. L. M. Martinez Vidal. 1995. Wavelength selection method for multicomponent spectrophotometric determinations using partial least squares. Analist, 120(12):2787–2792. ↗
  8. Gamon, M., A. Aue, and M. Smets. 2005. Sentence-level mt evaluation without reference translations: beyond language modeling. In Proceedings of the European Association for Machine Translation Conference, Budapest. find
  9. Gandrabur, S. and G. Foster. 2003. Confidence estimation for translation prediction. In Proceedings of the 7th Conference on Natural Language Learning, pages 95–102, Edmonton. find
  10. Giménez, J. and L. Màrquez. 2008. Heterogeneous automatic mt evaluation through non-parametric metric combinations. In Proceedings of the 3rd International Joint Conference on Natural Language Processing, pages 319–326, Hyderabad. find
  11. Jong, S De. 1993. Simpls: An alternative approach to partial least squares regression. Chemometrics and Intelligent Laboratory Systems, 18:251–263. ↗
  12. Kadri, Y. and J. Y. Nie. 2006. Improving query translation with confidence estimation for cross language information retrieval. In Proceedings of the 15th Conference on Information and Knowledge Management, pages 818–819, Arlington. find
  13. Koehn, P. and C. Monz. 2006. Manual and automatic evaluation of machine translation between european languages. In Proceedings of the Workshop on Statistical Machine Translation, pages 102–121, New York. find our card
  14. Koehn, P. 2004. Statistical significance tests for machine translation evaluation. In Conference on Empirical Methods in Natural Language Processing, Barcelona. find
  15. Kulesza, A. and A. Shieber. 2004. A learning approach to improving sentence-level mt evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation, Baltimore. find
  16. Papineni, K., S. Roukos, T. Ward, and W. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, page 311318, Morristown. find our card
  17. Paul, M., A. Finch, and E. Sumita. 2007. Reducing human assessment of machine translation quality to binary classifiers. In Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation, pages 154–162, Skovde. find
  18. Quirk, C. B. 2004. Training a sentence-level machine translation confidence measure. In Proceedings of the 4th Conference on Language Resources and Evaluation, pages 825–828, Lisbon. find
  19. Rosipal, R. and L. J. Trejo. 2001. Kernel partial least squares regression in reproducing kernel hilbert space. Machine Learning Research, 2:97–123. ↗
  20. Ueffing, N. and H. Ney. 2005. Application of word-level confidence measures in interactive statistical machine translation. In Proceedings of the 10th Conference of the European Association for Machine Translation, pages 262–270, Budapest. find
  21. Wold, S., A. Ruhe, H. Wold, and W. J. Dunn. 1984. The covariance problem in linear regression. the partial least squares (pls) approach to generalized inverses. SIAM Journal on Scientific Computing, 5:735–743. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/specia-2009-estimating-sentence-level-quality-machine/