Elamur Research
☰
home / papers / 2020 / Findings of the 2020 Conference on Machine Translation (WMT20)

Findings of the 2020 Conference on Machine Translation (WMT20)

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.wmt-1.1.pdf

Summary

This report describes two shared tasks run at WMT20, held online alongside EMNLP 2020. The tasks benchmark current machine translation (MT) systems on common, newly built test data. In the news task, teams built systems for 11 language pairs, which the paper counts as 22 translation directions. The pairs span high- and low-resource languages. Inuktitut, Japanese, Polish and Tamil were new, and Khmer and Pashto were reused from the corpus filtering task. Test sets were mostly fresh news translations and were source-original wherever possible. For some pairs, translators worked on whole paragraphs instead of single sentences. Participants received curated training corpora, including the newly added Wikimatrix.

The official ranking came from human judges, who used direct assessment on a 100-point adequacy scale. In a few directions, top systems scored in the same cluster as human references. However, earlier claims of human parity were not reproduced this year. A test suite showed that every system failed on gender bias. The Similar Language Translation task covered five pairs of closely related languages, and results varied widely between them. All data, outputs and judgments are public, which makes them reusable benchmarks.

Key points

  • Test set design: New language pairs got about 1,000 test sentences plus a development set. Previously used pairs got about 2,000 sentences. Common translator briefs were sent to all translation agencies.
  • Paragraph-split sources: English↔Czech, English↔German and English→Chinese sources were given to translators as paragraphs. About 70% of English paragraphs contained only one sentence. Sentence alignment used hunalign with bitextor dictionaries.
  • Human parity: Some systems clustered with human translation for German↔English, Chinese→English and English→Inuktitut. Apparent wins over humans reported in 2018–2019 for English→German and English→Czech did not hold up in 2020.
  • Similar Language Translation: The task covered Hindi–Marathi, Catalan/Spanish/Portuguese and Croatian/Serbian/Slovene. Systems were scored with BLEU, RIBES and TER. Catalan–Spanish systems clearly scored best.
  • Public resources: All data and outputs are released, and systems can be compared interactively with MT-ComparEval at wmt.ufal.cz. Inuktitut data came from Nunatsiaq News and the Nunavut Hansard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

222 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 206 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.wmt-1.1

@inproceedings{barrault2020findings,
  title = {{Findings of the 2020 Conference on Machine Translation (WMT20)}},
  author = {Barrault, Loïc and Biesialska, Magdalena and Bojar, Ondřej and Costa-jussà, Marta R. and Federmann, Christian and Graham, Yvette and Grundkiewicz, Roman and Haddow, Barry and Huck, Matthias and Joanis, Eric and Kocmi, Tom and Koehn, Philipp and Lo, Chi-kiu and Ljubešić, Nikola and Monz, Christof and Morishita, Makoto and Nagata, Masaaki and Nakazawa, Toshiaki and Pal, Santanu and Post, Matt and Zampieri, Marcos},
  booktitle = {Proceedings of the Fifth Conference on Machine Translation},
  year = {2020},
  doi = {10.18653/v1/2020.wmt-1.1},
  url = {https://doi.org/10.18653/v1/2020.wmt-1.1}
}

References

133 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ife Adebara, El Moatez Billah Nagoudi, and Muhammad Abdul Mageed. 2020. Translating similar languages: Role of mutual intelligibility in multilingual transformers. In Proceedings of WMT. find
  2. Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. Tico-19: the tra… find
  3. Chao Bei, Hao Zong, Qingmin Liu, and Conghu Yuan. 2020. Gtcom neural machine translation systems for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  4. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, Aljoscha Burchardt, and Sebastian Möller. 2020. Fine-grained linguistic evaluation for state-of-the-art machine translation. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  5. Pushpak Bhattacharyya. 2010. IndoWordNet. In Proceedings of LREC. find
  6. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Eleftherios Avramidis, Vivien Macketanz, Ursula Strohri… find
  7. Pere Vergés Boncompte and Marta R. Costa-jussà. 2020. Multilingual neural machine translation: Case-study for catalan, spanish and portuguese romance languages. In Proceedings of WMT. find
  8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceed… find our card
  9. Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. find
  10. Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further Meta-Evaluation of Machine Translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, Ohio. Association for Computational Linguistics. find
  11. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A… find our card
  12. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metri… find
  13. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… find our card
  14. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–48, Montreal, Canada. Association for Computation… find our card
  15. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… find our card
  16. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1–28, Athens, Greece. Association for Computational Linguistics. find
  17. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics. find
  18. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proc… find our card
  19. Sheila Castilho, Maja Popović, and Andy Way. 2020. On context span needed for machine translation evaluation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3735– 3742, Marseille, France. European Language Resources Association. find
  20. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors. 2019. Proceedings of th… find
  21. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT3: web inventory of transcribed and translated talks. In Proc. of EAMT, pages 261– 268. find our card
  22. Rajen Chatterjee, Markus Freitag, Matteo Negri, and Marco Turchi. 2020. Findings of the wmt 2020 find
  23. Findings of the wmt 2020 shared task on chat translation. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. shared task on automatic post-editing. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Co… find
  24. Peng-Jen Chen, Ann Lee, Changhan Wang, Naman Goyal, Angela Fan, Mary Williamson, and Jiatao Gu. 2020a. Facebook ai’s wmt20 news translation task submission. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  25. Christian Federmann. 2012. Appraise: an open-source toolkit for manual evaluation of mt output. The Prague Bulletin of Mathematical Linguistics, 98:25– 35. find
  26. Mikel L. Forcada, Francis M. Tyers, and Gema Ramírez Sánchez. 2009-11. The apertium machine translation platform: five years on. find our card
  27. Tanfang Chen, Weiwei Wang, Wenyang Wei, Xing Shi, Xiangang Li, Jieping Ye, and Kevin Knight. 2020b. The didi machine translation system for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  28. Alexander Fraser. 2020. The wmt 2020 shared tasks in unsupervised mt and very low resource supervised mt. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  29. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059– 7069. Curran Associates, Inc. find
  30. Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 34–44, Florence, Italy. Association for Computational Linguistics. find our card
  31. Marta R. Costa-jussà, Roger Creus, Oriol Domingo, Albert Domínguez, Miquel Escobar, Cayetana López, Marina Garcia, and Margarita Geleta. 2020. MT-Adapted Datasheets for Datasets: Template and Repository. arXiv e-prints, page arXiv:2005.13156. — [arXiv:2005.13156](https://arxiv.org/abs/2005.13156) find
  32. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics. find our card
  33. Marta R. Costa-jussà, Marcos Zampieri, and Santanu Pal. 2018. A Neural Approach to Language Variety Translation. In Proceedings of VarDial. find
  34. Ulrich Germann. 2020. The university of edinburgh’s submission to the german-to-english and english-to-german tracks in the wmt 2020 news translation and zero-shot translation robustness tasks. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Lingu… find
  35. Farhan Dhanani and Muhammad Rafi. 2020. Attention transformer model for translation of similar languages. In Proceedings of WMT. find
  36. Prajit Dhar, Arianna Bisazza, and Gertjan van Noord. 2020. Linguistically motivated subwords improve english-tamil translation: University of groningen’s submission to wmt-2020. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  37. Ulrich Germann, Roman Grundkiewicz, Martin Popel, Radina Dobreva, Nikolay Bogoychev, and Kenneth Heafield. 2020. Speed-optimized, compact student models that distill knowledge from a larger teacher model: the uedin-cuni submission to the wmt 2020 news translation task. In Proceedings of the Fifth Co… find
  38. Tomaž Erjavec, Nikola Ljubešić, and Nataša Logar. 2015. The slwac corpus of the sloveneweb. Informatica, 39(1). find
  39. Vikrant Goyal, Anoop Kunchukuttan, Rahul Kejriwal, Siddharth Jain, and Amit Bhagwat. 2020. Contact relatedness can help improve multilingual nmt: Microsoft stci-mt @ wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  40. Carlos Escolano, Marta R. Costa-jussà, and José A. R. Fonollosa. 2020. The talp-upc machine translation systems for wmt20 news translation task: Multilingual adaptation for low resource mt. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguisti… find
  41. Yvette Graham, George Awad, and Alan Smeaton. 2018. Evaluation of automatic video captioning using direct assessment. PLOS ONE, 13(9):1–20. find
  42. Miquel Esplà-Gomis. 2009. Bitextor: a free/open-source software to harvest translation memories from multilingual websites. In MT Summit Workshop on New Tools for Translators. International Association for Machine Translation. find
  43. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, M. Amin Farajian, António V. Lopes, André F. T. Martins, Sa… find
  44. Empirical Methods in Natural Language Processing, pages 944–952, Cambridge, MA. Association for Computational Linguistics. find
  45. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443– 451, Gothenburg, Sweden. Association for Computational… find
  46. Eric Joanis, Rebecca Knowles, Roland Kuhn, Samuel Larkin, Patrick Littell, Chi-kiu Lo, Darlene Stewart, and Jeffrey Micher. 2020. The Nunavut Hansard Inuktitut–English parallel corpus 3.0 with preliminary machine translation results. In Proceedings of The 12th Language Resources and Evaluation Confe… find
  47. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, pages 1–28. find
  48. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Ass… find
  49. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in Machine Translation Evaluation. arXiv e-prints, page arXiv:1906.09833. find
  50. Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Virtual. find our card
  51. Jiwan Kim, Soyoon Park, Sangha Kim, and Yoonjung Choi. 2020. An iterative knowledge transfer nmt system for wmt20 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  52. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho… find our card
  53. Shun Kiyono, Takumi Ito, Ryuto Konno, Makoto Morishita, and Jun Suzuki. 2020. Tohoku-aip-ntt at wmt 2020 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  54. Rebecca Knowles, Darlene Stewart, Samuel Larkin, and Patrick Littell. 2020. Nrc systems for the 2020 inuktitut-english news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  55. Jeremy Gwinnup and Tim Anderson. 2020. The afrl wmt20 news-translation systems. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  56. Tom Kocmi. 2020. Cuni submission for inuktitut language in wmt news 2020. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  57. Barry Haddow and Faheem Kirefu. 2020. PMIndia – A Collection of Parallel Corpora of Languages of India. arXiv e-prints, page arXiv:2001.09907. — [arXiv:2001.09907](https://arxiv.org/abs/2001.09907) find
  58. Tom Kocmi, Tomasz Limisiewicz, and Gabriel Stanovsky. 2020. Gender coreference and bias evaluation at wmt 2020. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  59. Sami Ul Haq, Sadaf Abdul Rauf, Arsalan Shaukat, and Abdullah Saeed. 2020. Document level nmt of low-resource languages with backtranslation. In Proceedings of WMT. find
  60. Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the wmt 2020 shared task on parallel corpus filtering and alignment. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics… find
  61. Hossein Hassani. 2017. Kurdish Interdialect Machine Translation. Proceedings of VarDial. find
  62. François Hernandez and Vincent Nguyen. 2020. The ubiqus english-inuktiut system for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  63. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Christopher J. Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translatio… find
  64. Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
  65. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121, New York City. Association for Computational Linguistics. find
  66. Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2019. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. find
  67. Nikola Ljubešić and Tomaž Erjavec. 2011. hrwac and slwac: Compiling web corpora for croatian and slovene. In International Conference on Text, Speech and Dialogue, pages 395–402. Springer. find
  68. Rihards Krišlauks and Mārcis Pinnis. 2020. Tilde at wmt 2020: News task systems. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  69. Nikola Ljubešić and Filip Klubička. 2014. {bs, hr, sr} wac-web corpora of bosnian, croatian and serbian. In Proceedings of the 9th web as corpus workshop (WaC-9), pages 29–35. find
  70. Mateusz Krubiński, Marcin Chochowski, Bartłomiej Boczek, Mikołaj Koszowski, Adam Dobrowolski, Marcin Szymański, and Paweł Przybysz. 2020. Samsung r&d institute poland submission to wmt20 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for C… find
  71. Qingsong Ma, Yvette Graham, Timothy Baldwin, and Qun Liu. 2017. Further investigation into reference bias in monolingual evaluation of machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2466–2475, Copenhagen, Denmark. Association for… find
  72. Amit Kumar, Rupjyoti Baruah, Rajesh Kumar Mundotiya, and Anil Kumar Singh. 2020. Transformer-based neural machine translation system for hindi - marathi. In Proceedings of WMT. find
  73. Anoop Kunchukuttan. 2020a. corpus. find
  74. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, and Hans Uszkoreit. 2018. Fine-grained evaluation of German-English Machine Translation based on a Test Suite. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  75. Indowordnet parallel Anoop Kunchukuttan. 2020b. The IndicNLP Library. https://github.com/anoopkunchukuttan/ indic\_nlp\_library/blob/master/docs/ indicnlp.pdf. ↗
  76. Lovish Madaan, Soumya Sharma, and Parag Singla. 2020. Transfer learning for related languages: Iit delhi’s submissions to the wmt20 similar language translation task. In Proceedings of WMT. find
  77. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  78. Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Combination of neural machine translation systems at wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  79. Luıs Marujo, Nuno Grazina, Tiago Luıs, Wang Ling, Luısa Coheur, and Isabel Trancoso. 2011. BP2EP– Adaptation of Brazilian Portuguese texts to European Portuguese. In Proceedings of EAMT. find
  80. Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Partha Pakray, and Sivaji Bandyopadhyay. 2020. Hindi-marathi cross lingual model. In Proceedings of WMT. find
  81. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the wmt20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find our card
  82. Samuel Laubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human–machine parity in language translation. Journal of Artificial Intelligence Research (JAIR), 67. find
  83. Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xianfeng Zeng, Qinsong Zeng, Peng Li, Ming Chen, Jie Zhou, Sifan Liu, and Hao Zhou. 2020. Wechat neural machine translation systems for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguist… find
  84. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Neural Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. In EMNLP 2018, Brussels, Belgium. Association for Computational Linguistics. find
  85. Luis A. Menéndez-Salazar, Grigori Sidorov, and Marta R. Costa-Jussà. 2020. The ipn-cic team system submission for the wmt 2020 simmilar language task. In Proceedings of WMT. of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Philadelphia, Pennsylvania. A… find
  86. Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, and Leo Wanner. 2018. The first multilingual surface realisation shared task (sr’18): Overview and evaluation results. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 1–12. Association for Computationa… find our card
  87. Venkatesh Parthasarathy, Akshai Ramesh, Rejwanul Haque, and Andy Way. 2020. The adapt system description for the wmt20 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  88. Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014. The language demographics of Amazon Mechanical Turk. Transactions of the Association for Computational Linguistics, 2(Feb):79–92. find
  89. Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019. The second multilingual surface realisation shared task (SR’19): Overview and evaluation results. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 1–17, Hong Kong, China. Association fo… find
  90. Martin Popel. 2018. CUNI transformer neural MT system for WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 482–487, Belgium, Brussels. Association for Computational Linguistics. find
  91. Alexander Molchanov. 2020. Promt systems for wmt 2020 shared news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  92. Martin Popel. 2020. Cuni english-czech and english-polish systems in wmt20: Robust document-level training. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  93. Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2020. JParaCrawl: A large scale web-based English-Japanese parallel corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3603–3609. find
  94. Martin Popel and Ondřej Bojar. 2018. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics, 110(1):43–70. find
  95. Vandan Mujadia and Dipti Sharmaa. 2020. Nmt based similar language translation for hindi - marathi. In Proceedings of WMT. find
  96. Martin Popel, Marketa Tomkova, Jakub Tomek, Łukasz Kaiser, Jakob Uszkoreit, Ondřej Bojar, and Zdeněk Žabokrtský. 2020. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11(4381):1–15. find
  97. Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt. ↗
  98. Atul Kr. Ojha, Priya Rani, Akanksha Bansal, Bharathi Raja Chakravarthi, Ritesh Kumar, and John P. McCrae. 2020. Nuig-panlingua-kmi hindi– marathi mt systems for similar language translation task @ wmt 2020. In Proceedings of WMT. find
  99. Maja Popovic. 2019. On reducing translation shifts in translations intended for MT evaluation. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 80–87, Dublin, Ireland. European Association for Machine Translation. find
  100. Csaba Oravecz, Katina Bontcheva, László Tihanyi, David Kolovratnik, Bhavani Bhaskar, Adrien Lardilleux, Szymon Klocek, and Andreas Eisele. 2020. etranslation’s submissions to the wmt 2020 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Co… find
  101. Maja Popović, Alberto Poncelas, Marija Brkic, and Andy Way. 2020. Neural machine translation for translating into Croatian and Serbian. In Proceedings of the Seventh Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). find
  102. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find our card
  103. Maja Popović and Alberto Poncelas. 2020. Neural machine translation between similar south-slavic languages. In Proceedings of WMT. find
  104. R. Pryzant, Y. Chung, D. Jurafsky, and D. Britz. 2017. JESC: Japanese-English Subtitle Corpus. arXiv preprint arXiv:1710.10639. find
  105. Santanu Pal and Marcos Zampieri. 2020. Neural machine translation for similar languages: The case of indo-aryan languages. In Proceedings of WMT. find
  106. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation… find
  107. Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  108. Kamalkumar Rathinasamy, Amanpreet Singh, Bal-aguru Sivasambagupta, Prajna Prasad Neerchal, and Vani Sivasankaran. 2020. Infosys machine translation system for wmt20 similar language translation task. In Proceedings of WMT. find
  109. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  110. Christian Roest, Lukas Edman, Gosse Minnema, Kevin Kelly, Jennifer Spenader, and Antonio Toral. 2020. Machine translation for english–inuktitut with segmentation,data acquisition and pre-training. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Li… find
  111. Roman Sudarikov, Martin Popel, Ondřej Bojar, Aljoscha Burchardt, and Ondřej Klejch. 2016. Using MT-ComparEval. In Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 76–82. find
  112. Yves Scherrer, Stig-Arne Grönroos, and Sami Virpioja. 2020a. The university of helsinki and aalto university submissions to the wmt 2020 news and low-resource translation tasks. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  113. Jörg Tiedemann. 2009. News from OPUS - a collection of multilingual parallel corpora with tools and interfaces. In Proceedings of the Conference on Recent Advances in Natural Language Processing, pages 237–248. John Benjamins. find
  114. Yves Scherrer, Alessandro Raganato, and Jörg Tiedemann. 2020b. The mucow word sense disambiguation test suite at wmt 2020. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  115. Jörg Tiedemann and Lars Nygaard. 2004. The opus corpus-parallel and free: http://logos.uio.no/opus. In Proceedings of LREC. ↗
  116. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wiki-Matrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia. arXiv e-prints, page arXiv:1907.05791. find our card
  117. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find our card
  118. Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei Dang, Xue Zhengshan, and JIE HAO. 2020. Oppo’s machine translation systems for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  119. Dániel Varga, Péter Halaácsy, András Kornai, Vok-tor Nagy, László Németh, and Viktor Trón. 2005. Parallel corpora for medium density languages. In Proceedings of the RANLP 2005 Conference, pages 590–596. find
  120. Shashank Siripragada, Jerin Philip, Vinay P. Namboodiri, and C V Jawahar. 2020. A multilingual parallel corpora collection effort for Indian languages. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3743–3751, Marseille, France. European Language Resources Association… find
  121. Tereza Vojtěchová, Michal Novák, Miloš Klouček, and Ondřej Bojar. 2019. SAO WMT19 Test Suite: Machine Translation of Audit Reports. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  122. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and Ralph Weischedel. 2006. A study of translation error rate with targeted human annotation. In In Proceedings of the Association for Machine Transaltion in the Americas (AMTA 2006. find
  123. Daimeng Wei, Hengchao Shang, Zhanglin Wu, Zhengzhe Yu, Liangyou Li, Jiaxin Guo, Minghan Wang, Hao Yang, Lizhi Lei, Ying Qin, and Shiliang Sun. 2020a. Hw-tsc’s participation in the wmt 2020 news translation shared task. In Proceedings of the Fifth Conference on Machine Translation, Online. Associatio… find
  124. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. 2020a. Findings of the wmt 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguis… find our card
  125. Xiangpeng Wei, Ping Guo, Yunpeng Li, Xingsheng Zhang, Luxi Xing, and Yue Hu. 2020b. Iie’s neural machine translation systems for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  126. Liwei Wu, Xiao Pan, Zehui Lin, Yaoming ZHU, Mingxuan Wang, and Lei Li. 2020a. The volctrans machine translation system for wmt20. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find
  127. Shuangzhi Wu, Xing Wang, Longyue Wang, Fangxu Liu, Jun Xie, Zhaopeng Tu, Shuming Shi, and Mu Li. 2020b. Tencent neural machine translation systems for the wmt20 news translation task. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. ar… find
  128. Saumitra Yadav and Manish Shrivastava. 2020. A3-108 machine translation system for similar language translation shared task 2020. In Proceedings of WMT. find
  129. Lei Yu, Laurent Sartran, Po-Sen Huang, Wojciech Stokowiec, Domenic Donato, Srivatsan Srinivasan, Alek Andreev, Wang Ling, Sona Mokra, Agustin Dal Lago, Yotam Doron, Susannah Young, Phil Blunsom, and Chris Dyer. 2020. The deepmind chinese–english document translation system at wmt2020. In Proceedings… find
  130. Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huan-Bo Luan, and Yang Liu. 2017. THUMT: an open source toolkit for neural machine translation. CoRR, abs/1706.06415. — [arXiv:1706.06415](https://arxiv.org/abs/1706.06415) find
  131. Xiaoheng Zhang. 1998. Dialect MT: A Case Study Between Cantonese and Mandarin. In Proceedings of ACL. find
  132. Yuhao Zhang, Ziyang Wang, Runzhe Cao, Binghao Wei, Weiqiao Shan, Shuhan Zhou, Abudurexiti Reheman, Tao Zhou, Xin Zeng, Laohu Wang, Yongyu Mu, Jingnan Zhang, Xiaoqian Liu, Xuanjun Zhou, Yinqiao Li, Bei Li, Tong Xiao, and Jingbo Zhu. 2020. The niutrans machine translation systems for wmt20. In Proceed… find
  133. Vilém Zouhar, Tereza Vojtěchová, and Ondřej Bojar. 2020. Wmt20 document-level markable error exploration. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics. find