Elamur Research
☰
home / papers / 2018 / Findings of the WMT 2018 Shared Task on Quality Estimation

Findings of the WMT 2018 Shared Task on Quality Estimation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6451.pdf

Summary

The WMT 2018 Quality Estimation (QE) shared task asked systems to judge machine translation output without reference translations, at four granularities: sentence, word, phrase and document. The organisers released new training and test data in three domains: IT, life sciences, and sports/outdoor product listings. The data covers English-German, English-Czech, English-Latvian, German-English and English-French, and the translations come from in-house statistical and neural MT systems and, for one domain, an online engine. This edition added several new annotations: gaps (missing words), source words responsible for errors, human-assigned phrase labels, post-editing logs, and document scores computed from word-level errors weighted by severity. Teams from ten institutions took part. Each task had a feature-based baseline (SVR or CRF). Detecting faulty source tokens proved harder than flagging target errors. Gap detection was also harder, though SMT outputs still allowed good scores. Participation in the phrase-level task was too thin for firm conclusions. The task matters because it provides the first QE benchmark on NMT output, and its CodaLab leaderboards stay open for blind evaluation.

Key points

  • Four tasks: sentence-level post-editing effort (Task 1), word-level labels including gaps and source errors (Task 2), phrase-level labels (Task 3), and severity-weighted document scoring on English-French (Task 4).
  • The sentence baseline uses 17 QuEst++ features with an RBF-kernel SVR in SCIKIT-LEARN (γ=0.01, ε=0.0825, C=20). The word baseline uses 28 MARMOT features plus 6 combined features in a CRFSuite CRF. The phrase baseline uses 72 "black-box" features.
  • No baseline was provided for missing words or source-side errors.
  • Organisers offered post-editing time, keystrokes and editor IDs, but no team made use of them, so the value of this information remains untested.
  • CodaLab competitions remain open for each level, and each task and language pair accepts at most five entries daily.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

59 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 56 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6451

@inproceedings{specia2018findings,
  title = {{Findings of the WMT 2018 Shared Task on Quality Estimation}},
  author = {Specia, Lucia and Blain, Frédéric and Logacheva, Varvara and Astudillo, Ramón F. and Martins, André F. T.},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6451},
  url = {https://doi.org/10.18653/v1/w18-6451}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  2. Hervé Abdi. 2007. The bonferroni and šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics, 3:103–107. find
  3. Wilker Aziz, Sheila Castilho Monteiro de Sousa, and Lucia Specia. 2012. Pet: a tool for post-editing and assessing machine translation. In Eighth International Conference on Language Resources and Evaluation, LREC, pages 3982–3987, Istanbul, Turkey. find
  4. Prasenjit Basu, Santanu Pal, and Sudip Kumar Naskar. 2018. Keep it or not: Word level quality estimation for post-editing. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46, Lisbon, Portugal. Association for Computational Linguistics. find our card
  6. Ergun Biçici. 2017. Predicting translation performance with referential translation machines. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 540– 544, Copenhagen, Denmark. Association for Computational Linguistics. find
  7. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. find
  8. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 297–303, Lisboa, Portugal. Association for Computational Linguistics. find
  9. Findings of the 2012 workshop on statistical machine translation. In Seventh Workshop on Statistical Machine Translation, WMT, pages 10–51, Montréal, Canada. find our card
  10. Melania Duma and Wolfgang Menzel. 2018. The benefit of pseudo-reference translations in quality estimation of mt output. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  11. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning, pages 1188–1196. find our card
  12. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. find
  13. Maoxi Li, Qingyu Xiang, Zhiming Chen, and Ming-weng Wang. 2018. A unified neural network for quality estimation of machine translation. IEICE Trans. Information and Systems, E101-D(9). find
  14. Varvara Logacheva, Chris Hokamp, and Lucia Specia. 2016. Marmot: A toolkit for translation quality estimation at the word level. In Tenth International Conference on Language Resources and Evaluation, LREC, pages 3671–3674, Portoroz, Slovenia. find
  15. Thierry Etchegoyhen, Eva Martı́nez Garcia, and Andoni Azpeitia. 2018. Supervised and unsupervised minimalist quality estimators: Vicomtech’s participation in the wmt 2018 quality estimation task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  16. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0(12):455–463. find
  17. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517. International World Wide Web Conferences Steering Committee. find
  18. Ngoc Quang Luong, Laurent Besacier, and Benjamin Lecouteux. 2014. Word confidence estimation for smt n-best list re-ranking. In Proceedings of the EACL 2014 Workshop on Humans and Computer-assisted Translation, pages 1–9, Gothenburg, Sweden. Association for Computational Linguistics. find
  19. Junjie Hu, Wei-Cheng Chang, Yuexin Wu, and Graham Neubig. 2018. Contextual encoding for translation quality estimation. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  20. André F. T. Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel’s participation in the wmt16 word-level translation quality estimation shared task. In Proceedings of the First Conference on Machine Translation, pages 806–811, Berlin, Germany. Association for Computational Linguistics. find
  21. Julia Ive, Frédéric Blain, and Lucia Specia. 2018a. DeepQuest: a framework for neural-based quality estimation. In Proceedings of COLING 2018, the 27th International Conference on Computational Linguistics: Technical Papers, Santa Fe, New Mexico. find
  22. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43–52. ACM. find
  23. Julia Ive, Carolina Scarton, Frédéric Blain, and Lucia Specia. 2018b. Sheffield submissions for the wmt18 quality estimation shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  24. Sylvain Raybaud, David Langlois, and Kamel Smaili. 2011. ”this sentence is wrong.” detecting errors in machine-translated sentences. Machine Translation, 25(1):1–34. find
  25. Felipe Sánchez-Martı́ı́nez, Miquel Esplà-Gomis, and Mikel L. Forcada. 2018. Ualacant machine translation quality estimation at wmt 2018: a simple approach using phrase tables and feed-forward neural networks. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  26. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Tasks Papers, pages 562–568, Copenhagen, Denmark. Association for Computational Linguistics. find
  27. Marina Sanchez-Torron and Philipp Koehn. 2016. Machine translation quality and post-editor productivity. AMTA 2016, Vol., page 16. find
  28. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadina, Matteo Negri, , and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Machine Translation Summit XVI, pages 55–71, Nagoya, Japan. find
  29. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. find
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  31. Jiayi Wang, Kai Fan, Bo Li, Fengming Zhou, Boxing Chen, Yangbin Shi, and Luo Si. 2018. Alibaba submission for wmt18 quality estimation task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  32. Elizaveta Yankovskaya, Andre Tattar, and Mark Fishel. 2018. Quality estimation with force-decoded attention and cross-lingual embeddings. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tasks Papers, Brussels, Belgium. Association for Computational Linguistics. find
  33. Alexander Yeh. 2000. More Accurate Tests for the Statistical Significance of Result Differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953, Saarbrücken, Germany. find