Elamur Research⌕
☰
home / papers / 2019 / OpenKiwi: An Open Source Framework for Quality Estimation

OpenKiwi: An Open Source Framework for Quality Estimation

Summary

Translation quality estimation (QE) predicts how good a machine translation is without a reference translation. It works at the word level, tagging target words, gaps and source words as OK or BAD, and at the sentence level, for example predicting HTER. Strong QE systems had become hard to reproduce because they rely on ensembles, intricate architectures or poorly documented pre-training, and existing open toolkits trailed the WMT leaders.

The authors release OpenKiwi, a PyTorch framework that reimplements the top WMT 2015–18 QE systems under one roof: QUETCH, NuQE, Predictor-Estimator and a stacked linear ensemble. It can be trained from YAML configs on the command line or imported as a Python package. On the WMT 2018 English-German SMT and NMT data, averaged and stacked combinations of its models give the best word-level scores reported. They also beat deepQuest everywhere and come close to the strongest sentence-level systems.

The framework matters because it turns a scattered body of research into a shared, reproducible baseline. It went on to serve as the official WMT 2019 QE baseline and as the base for that year's winning systems.

Key points

  • Implements four systems:
    • QUETCH, a window-based MLP.
    • NuQE, feed-forward layers plus a bi-directional GRU, trained only on shared-task data.
    • Predictor-Estimator, a predictor pre-trained on large parallel data that feeds an estimator, with multi-task HTER prediction.
    • A stacked ensemble with a feature-based linear model.
  • Benchmark data from WMT 2018 En-De, each split with 1,000 development triplets:
    • SMT: 26,273 training triplets.
    • NMT: 13,442 training triplets.
  • Extra resources:
    • 526,368 artificial triplets built through round-trip translation.
    • 3,396,364 in-domain parallel sentences, used to pre-train the predictor.
  • Test-set results:
    • SMT: word-level F1-mult of 62.70 (MT), 52.14 (gaps) and 48.88 (source), with Pearson r of 71.08. QE Brain reached 73.97 on r.
    • NMT: MT-tag F1-mult of 44.77.
  • Engineering for reproducibility:
    • Poetry for dependency management.
    • Tests covering about 80% of the code.
    • Optional MLflow tracking.
    • Affero GPL license.

Interesting ideas

  • I1. Using APE output as a labeler: an automatic post-editing model produces pseudo post-edits, which are then turned into word tags and HTER with the same tool (TERCOM) used for the gold labels. This approach could apply to any task where labels are derived from corrected outputs.
  • I2. Two ways to combine models: simple averaging of model outputs is compared with stacking them as features in a linear model. Stacking was the stronger option for MT-word tags, which suggests a cheap linear layer over diverse neural systems is worth trying before building larger ensembles.
  • I3. Pre-training on related data: the predictor first learns to reconstruct target tokens from large parallel corpora, and the estimator is then trained on scarce QE labels. This is a transferable recipe for low-resource labeled tasks that have plentiful raw parallel or monolingual text.
  • I4. Synthetic data recipe: in-domain sentences are selected by cross-entropy filtering, round-trip translated, and stratified-sampled to create artificial triplets. The small genuine dataset is oversampled (×20) when mixed in, so it is not swamped by the synthetic data.
  • I5. Fixed pipeline, config-driven experiments: data handling, training loop and evaluation are fixed, and experiments vary only through config files or a single model class. Combined with run tracking, this makes comparisons across architectures fair and easy to repeat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

103 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-3020

@inproceedings{kepler2019openkiwi,
  title = {{OpenKiwi: An Open Source Framework for Quality Estimation}},
  author = {Kepler, Fabio and Trénous, Jonay and Treviso, Marcos and Vera, Miguel and Martins, André F. T.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/p19-3020},
  url = {https://doi.org/10.18653/v1/p19-3020}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. find
  2. Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón Astudillo, and André F. T. Martins. 2018a. Findings of the wmt 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 702–722, Belgium, Brussels. Association for Computational Linguistics. find our card
  3. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. DeepQuest: a framework for neural-based Quality Estimation. In International Conference on Computational Linguistics (COLING). find
  4. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. find our card
  5. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, António Góis, M. Amin Farajian, António V. Lopes, and André F. T. Martins. 2019. Unbabel’s Participation in the WMT19 Translation Quality Estimation Shared Task. In Conference on Machine Translation (WMT). find
  6. Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018b. Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. ↗
  7. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation. In Conference on Machine Translation (WMT). find our card
  8. Jiayi Wang, Kai Fan, Bo Li, Fengming Zhou, Boxing Chen, Yangbin Shi, and Luo Si. 2018. Alibaba Submission for WMT18 Quality Estimation Task. In Conference on Machine Translation (WMT). find
  9. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. ↗ our card
  10. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation. In Workshop on Machine Translation (WMT). find
  11. Varvara Logacheva, Chris Hokamp, and Lucia Specia. 2016. Marmot: A toolkit for translation quality estimation at the word level. In LREC. find
  12. André F. T. Martins, Ramon Astudillo, Chris Hokamp, and Fábio Kepler. 2016. Unbabel’s Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task. In Conference on Machine Translation (WMT). 7 http://www.statmt.org/wmt19/ qe-results.html ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kepler-2019-openkiwi-open-source-framework-quality/