Elamur Research⌕
☰
home / papers / 2020 / Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

Summary

Learned metrics for machine translation evaluation mostly target English output, while the WMT Metrics 2020 Shared Task also judges translation into other languages, some of which have no human ratings. The authors adapt BLEURT, a BERT-based regression metric originally built for English, to this multilingual setting. Instead of the English pipeline's synthetic mid-training stage, they fine-tune multilingual encoders directly on all available WMT human ratings, pooled across languages. They use two encoders: public MBERT and MBERT-WMT, a larger in-house model pre-trained for this purpose. They also submit simple BERT regression baselines and, for English→German, ensembles of BLEURT and YiSi that use several reference translations. On WMT19 data, the systems match or beat strong competitors such as PRISM and BERTSCORE. MBERT-WMT does well even on target languages it was never fine-tuned on. This shows that one pooled multilingual model can evaluate translation into lower-resource languages, which is relevant for languages like Ukrainian.

Key points

  • Evaluation covers 14 language pairs with fine-tuning data and 4 zero-shot pairs, including En→Gu, En→Kk and En→Lt. Training used WMT ratings from 2015–2018 (216,541 sentence pairs, 8 languages), with WMT19 as the development set. The final submissions added 2019, contributing 289,895 more pairs and 4 more languages.
  • MBERT-WMT has 24 Transformer layers and a new WordPiece vocabulary covering Pashto, Khmer and Inuktitut. It was pre-trained on 19 WMT languages from sources such as Wikipedia, News Crawl, C4, OPUS, ParaCrawl and Nunavut Hansard, for 1M steps with LAMB on 64 TPU v3 chips.
  • On non-English targets at segment level, MBERT-WMT averaged 0.502, against 0.482 for PRISM. It was best on 7 of 11 pairs but weaker for En→Ru and En→Zh, so BERT-CHINESE-L2 was used for Chinese.
  • For to-English pairs, English BLEURT reached the top segment-level average of 0.339, against PRISM's 0.336.
  • English→German submissions were YISI-COMB, which takes the minimum recall and maximum precision across three references, and ALL-COMB, which averages BLEURT, YiSi-1 and YiSi-2 predictions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 19 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.102

@inproceedings{sellam2020learning,
  title = {{Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task}},
  author = {Sellam, Thibault and Pu, Amy and Chung, Hyung Won and Gehrmann, Sebastian and Tan, Qijun and Freitag, Markus and Das, Dipanjan and Parikh, Ankur},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.102},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.102}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loı̈c Barrault, Ondřej Bojar, Marta R Costa-Jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. Findings of the 2019 conference on machine translation. In Proceedings of WMT. find
  2. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of nlg systems. In Proceedings of EACL. *Figure 1: Correlations with respect to different α settings for Yisi-1. The system-level correlation is highest when α = 0.7, which is the α we use for the submission.* Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of WMT. find our card
  3. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in machine translation research. In Proceedings of EACL. A LL - COMB We combined the predictions of Y I S I -1 with those of B LEURT and Y I S I -2. Y I S I -2 usually performs worse than the reference-based metrics, but we found that incorporating its predictions can help. Having three different metrics (B LEURT , Y I S I -1, Y I S I -2) and three different reference translations, we take all seven predictions and average the scores for each segment. The combined prediction A LL - COMB outperforms every single metric at the segment level, though the system-level correlation drops in comparison to the best Y I S I -1 score on paraphrased references. This submission aims to maximize the segment-level correlation. find
  4. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv. find
  5. Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In Proceedings of ACL. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT. find
  7. Miquel Esplà-Gomis, Mikel L Forcada, Gema Ramı́rez-Sánchez, and Hieu Hoang. 2019. Paracrawl: Web-scale parallel corpora for the languages of the eu. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks. Summary We submit the following systems to the WMT Metrics shared task: • B LEURT as previously published, fine-tuned on the human ratings of the WMT Metrics shared task 2015 to 2019, to-English. find our card
  8. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be Guilty but References are not Innocent. In Proceedings of EMNLP. • A multi-lingual extensions of B LEURT based on a 20 languages variant of M BERT and BERT-C HINESE . find our card
  9. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG. find our card
  10. Eric Joanis, Rebecca Knowles, Roland Kuhn, Samuel Larkin, Patrick Littell, Chi-kiu Lo, Darlene Stewart, and Jeffrey Micher. 2020. The nunavut hansard inuktitut–english parallel corpus 3.0 with preliminary machine translation results. find
  11. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2019. Machine translation evaluation with bert regressor. arXiv. ↗
  12. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of ACL. ↗ our card
  13. Kaliyaperumal Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2019. Cross-lingual ability of multilingual bert: An empirical study. In Proceedings of ICLR. ↗
  14. Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. Proceedings of EMNLP. find our card
  15. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out. find
  16. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of The 8th Language Resources and Evaluation Conference. find
  17. Chi-kiu Lo. 2019. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of WMT. find our card
  18. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv. ↗
  19. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of WMT. find our card
  20. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of WMT. find our card
  21. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training bert in 76 minutes. In Proceedings of ICLR. ↗
  22. Nitika Mathur, Tim Baldwin, and Trevor Cohn. 2020. Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics. Proceedings of ACL. find our card
  23. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of ACL. find
  24. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Proceedings of ICLR. ↗
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  26. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Proceedings of EMNLP. ↗ our card
  27. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv. find our card
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. In Journal of Machine Learning Research. ↗
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv. find our card
  30. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In Proceedings of ICASSP. find
  31. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. Proceedings of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/