Elamur Research
☰
home / papers / 2020 / Findings of the WMT 2020 Shared Task on Quality Estimation

Findings of the WMT 2020 Shared Task on Quality Estimation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.wmt-1.79.pdf

Summary

Quality estimation (QE) means judging how good a machine translation is at run time, without any reference translation. The WMT 2020 shared task, its ninth edition, benchmarks QE for neural MT output at word, sentence and document granularity. The organisers released several new resources. One is a sentence-level set, drawn mostly from Wikipedia, covering seven language pairs from high- to low-resource and scored 0–100 with direct assessment (DA) by professional translators. Another provides HTER scores and OK/BAD word tags derived from post-editing for En-De and En-Zh. The third is a new English–French test set of Amazon product texts with MQM error annotations. The fairseq NMT models that generated the translations were also shared, which enabled "glass-box" features. OpenKiwi Predictor-Estimator models served as baselines. Nineteen teams from 27 institutions sent 1374 submissions. The strongest systems used large pre-trained multilingual encoders such as XLM-R, often as ensembles above 500M parameters, and outperformed the baselines everywhere. Glass-box methods were weaker than these systems but still clearly beat the baselines. All data and submissions are public benchmarks for QE and related tasks such as automatic post-editing.

Key points

  • Task 1 (DA) language pairs:
    • High-resource: En-De, En-Zh, Ru-En.
    • Medium-resource: Ro-En, Et-En.
    • Low-resource: Si-En, Ne-En.
    • Each pair has 7,000 training, 1,000 dev and 1,000 test sentences.
    • The data is a superset of MLQE.
    • A multilingual track required predictions for all six Wikipedia-based pairs.
  • Task 2 (post-editing effort):
    • HTER was computed with TERCOM and capped at 1.
    • Mean HTER was 0.32 for En-De and 0.62 for En-Zh.
    • Word-level tags cover source words, target words and gaps, using FastAlign alignments.
  • Task 3 (document-level MQM):
    • The score is MQM = 1 − (minor + 5·major + 10·critical)/n, so it can be negative.
    • Papago reached Pearson r 0.573, compared with 0.389 for the baseline.
    • The best error-span F1 was 0.472 (IST and Unbabel), compared with 0.416 for the baseline.
  • Open questions:
    • No team used the offered Wikipedia article titles, so the value of document-level context remains unanswered.
    • Most multilingual entries were per-language models. Bergamot-LATTE was the exception, training a single model for all languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

100 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.79

@inproceedings{specia2020findings,
  title = {{Findings of the WMT 2020 Shared Task on Quality Estimation}},
  author = {Specia, Lucia and Blain, Frédéric and Fomicheva, Marina and Fonseca, Erick and Chaudhary, Vishrav and Guzmán, Francisco and Martins, André F. T.},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.79},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.79}
}

References

39 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yujin Baek, Zae Myung Kim, Jihyung Moon, Hyun-joong Kim, and Eunjeong Park. 2020. Patquest: Papago translation quality estimation. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  3. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32, pages 7059–7069. find
  4. Qu Cui, Xiang Geng, Shujian Huang, and Jiajun Chen. 2020. Nju’s submission for wmt2020 qe shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. find
  7. Kai Fan, Jiayi Wang, Bo Li, Fengming Zhou, Boxing Chen, and Luo Si. 2019. “bilingual expert” can find translation errors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6367–6374. find
  8. Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2020a. MLQE-PE: A multilingual quality estimation and post-editing dataset. arXiv preprint arXiv:2010.04480. ↗
  9. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Vishrav Chaudhary, Mark Fishel, Francisco Guzmán, and Lucia Specia. 2020b. Bergamot-latte submissions for the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  10. Ergun Biçici. 2020. Rtm ensemble learning results at quality estimation task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  11. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020c. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computation Linguistics. find our card
  12. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28. find
  14. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098–6111, Hong Kong, China. Association for Computational Linguistics. find our card
  15. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517. International World Wide Web Conferences Steering Committee. find
  16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751. ↗
  17. Chi Hu, Hui Liu, Kai Feng, Chen Xu, Nuo Xu, Zefan Zhou, Shiqin Yan, Yingfeng Luo, Chenglong Wang, Xia Meng, Tong Xiao, and Jingbo Zhu. 2020. The niutrans system for the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  18. Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020. NU-BIA: Neural based interchangeability assessor for text generation. find
  19. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, António Góis, M Amin Farajian, António V Lopes, and André FT Martins. 2019a. Unbabel’s participation in the wmt19 translation quality estimation shared task. arXiv preprint arXiv:1907.10352. ↗
  20. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019b. OpenKiwi: An open source framework for quality estimation. In Proceedings of ACL 2019 System Demonstrations. find
  21. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, pages 562–568. find
  22. Dongjun Lee. 2020. Two-phase cross-lingual language model fine-tuning for machine translation quality estimation. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  23. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. find
  24. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43–52. ACM. find
  25. João Moura, Miguel Vera, Daan van Stigt, Fabio Kepler, and André F. T. Martins. 2020. Ist-unbabel participation in the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  26. Akifumi Nakamachi, Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2020. Tmuou submission for wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  27. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair wmt19 news translation task submission. In Proc. of WMT, pages 1–4. find
  28. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. find our card
  29. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. Transquest at wmt2020: Sentence-level direct assessment. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  31. Raphael Rubino. 2020. Nict kyoto submission for the wmt’20 quality estimation task: Intermediate training for domain and task adaptation. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  32. Marina Sanchez-Torron and Philipp Koehn. 2016. Machine translation quality and post-editor productivity. AMTA 2016, Vol., page 16. find
  33. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. find
  34. Shuo Sun, Francisco Guzmán, and Lucia Specia. 2020. Are we estimating or guesstimating translation quality? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6262–6267, Online. Association for Computational Linguistics. find
  35. Minghan Wang, Hao Yang, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Lizhi Lei, Ying Qin, Shimin Tao, Shiliang Sun, Yimeng Chen, and Liangyou Li. 2020a. Hw-tsc’s participation at wmt 2020 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  36. Zixuan Wang, Haijiang Wu, Xiaoli Wang, Xinjie Wen, Ruichen Wang, and Qingsong Ma. 2020b. Tencent submission for wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find
  37. Evan J. Williams. 1959. Regression Analysis, volume 14. Wiley, New York, USA. find
  38. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  39. Lei Zhou, Liang Ding, and Koichi Takeda. 2020. Zero-shot translation quality estimation with explicit cross-lingual patterns. In Proceedings of the Fifth Conference on Machine Translation, Volume 2: Shared Tasks Papers, Online. Association for Computational Linguistics. find