Elamur Research
☰
home / papers / 2019 / Findings of the WMT 2019 Shared Tasks on Quality Estimation

Findings of the WMT 2019 Shared Tasks on Quality Estimation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5401.pdf

Summary

This report describes the WMT 2019 Quality Estimation (QE) shared task. QE means judging machine translation output when only the source sentence and the system's hypothesis are available, with no reference translation. The organizers ran three tracks:

  • Task 1 covered word- and sentence-level QE. Systems tagged words and gaps as OK or BAD and predicted HTER. It used the English-German IT-domain data from the previous year plus a new English-Russian set of Microsoft interface messages.
  • Task 2 covered document-level QE on English-French Amazon product reviews. Systems predicted MQM scores and also located fine-grained error spans, which was a new subtask.
  • Task 3 was new. It treated QE as a reference-free metric, run jointly with the WMT19 metrics task, and scored outputs from the news translation systems against human judgments.

All Task 1 and Task 2 data came from neural MT, so the setup was simpler than in earlier years. Eleven institutions took part. The strongest systems typically relied on pretrained contextual encoders such as BERT and XLM, combined with transfer learning. In Task 3, reference-free metrics usually fell short of the reference-based BLEU and chrF, especially at segment level. This shows both how far reference-free evaluation has come and how much room it still has to improve.

Key points

  • Word-level evaluation: the main metric is F1-Mult, the product of the OK and BAD F1 scores, with systems ranked on the source side. MCC was added because BAD labels are rare. Labels were derived using TERCOM and fast_align alignments.
  • Document-level scoring: MQM penalizes errors by severity, with weights of 1 for minor, 5 for major and 10 for critical, and scores are truncated at 0. Error-span predictions are scored by F1 based on character overlap.
  • Task 3 setup:
    • Evaluation used Kendall τ at segment level and Pearson correlation at system level.
    • LASER and LogProb served as baselines.
    • Submissions included YiSi-2, UNI, USFD and the ibm1-based metrics.
  • Task 2 participation: only two teams entered. The organizers plan to lower the entry barrier, for example by releasing preprocessed data or word-level predictions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5401

@inproceedings{fonseca2019findings,
  title = {{Findings of the WMT 2019 Shared Tasks on Quality Estimation}},
  author = {Fonseca, Erick and Yankovskaya, Lisa and Martins, André F. T. and Fishel, Mark and Federmann, Christian},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)},
  year = {2019},
  doi = {10.18653/v1/w19-5401},
  url = {https://doi.org/10.18653/v1/w19-5401}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Abdi. 2007. Bonferroni and Šidák corrections for multiple comparisons. In N. J. Salkind, editor, Encyclopedia of Measurement and Statistics, pages 103– 107. Sage, Thousand Oaks, CA. find
  2. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. CoRR, abs/1812.10464. ↗ our card
  3. Shamil Chollampatt and Hwee Tou Ng. 2018. Neural quality estimation of grammatical error correction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. ↗
  5. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of NAACL. find
  6. Yvette Graham, Qingsong Ma, Johnny Tian-Zheng Wei, and Ondřej Bojar. 2019. Results of the WMT19 Metrics Shared Task. In Proceedings of the Fourth Conference on Machine Translation (WMT), Florence, Italy. find
  7. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. Deepquest: a framework for neural-based quality estimation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3146–3157. find
  8. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of ACL 2019 System Demonstrations. find
  9. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv preprint arXiv:1901.07291. ↗
  10. Chi kiu Lo. 2019. YiSi - A unified semantic MT quality evaluation and estimation metric for languages with Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. find our card
  11. Maja Popović, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: Ibm1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 99–103, Edinburgh, Scotland. Association for Computational Linguistics. find
  12. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. Association for Computational Linguistics and The Asian Federation of Natural Language Processing. find
  13. Lisa Yankovskaya, Andre Tättar, and Mark Fishel. 2019. Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT), Florence, Italy. find
  14. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953. find