Elamur Research
☰
home / papers / 2019 / Findings of the 2019 Conference on Machine Translation (WMT19)

Findings of the 2019 Conference on Machine Translation (WMT19)

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5301.pdf

Summary

WMT19 is the yearly shared evaluation of machine translation systems. This report covers the news translation task and a related task on similar languages. Teams built systems for 18 language pairs. These included established pairs (English with Chinese, Czech, German, Finnish and Russian) and new lower-resourced pairs: Gujarati, Kazakh and Lithuanian with English, plus French–German. There was also an unsupervised German→Czech sub-track.

The organizers supplied training corpora and freshly translated news test sets. Each test set contained only original source-language text, so translationese no longer appeared on the source side. Systems were ranked mainly by crowd-sourced and researcher-provided Direct Assessment (DA) judgments on a 0–100 scale. Several language pairs were judged with document context, against the source text, and with whole-document scores. Test suites added targeted checks of specific phenomena.

Top systems again appeared to match human quality for some pairs. The authors warn that this claim is fragile, since last year's parity results did not hold up. All data, outputs and judgments are released publicly as a reusable benchmark.

Key points

  • Evaluation changes this year:
    • English→X and X→English test sets no longer overlap, which removes "flipped" translationese input.
    • Assessors saw the surrounding document for many pairs.
    • Source-based DA was extended to more pairs.
    • Whole documents were scored as well as single segments.
  • Test sets for existing pairs have about 2,000 sentences. The new pairs (Gujarati, Kazakh, Lithuanian) have test sets about half that size, plus roughly 1,000-sentence development sets.
  • Training resources include Europarl, News Commentary, Common Crawl, ParaCrawl, the EU Press Release corpus, the UN corpus, WikiTitles and CzEng v1.7 (about 57M Czech–English sentence pairs). For the low-resource pairs, crawled Kazakh and Gujarati parallel data (e.g., Bible, localisation, Wikipedia) was also provided.
  • In the similar-language task, quality varied greatly by pair:
    • Portuguese→Spanish reached 64.7 BLEU (MLLPUPV).
    • Czech→Polish peaked at only 7.9 BLEU (UPC-TALP).
    • Hindi↔Nepali was also covered.
  • Test suites, such as Vojtěchová et al., 2019, found phenomena that even the best systems still mishandle.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

377 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 339 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5301

@inproceedings{barrault2019findings,
  title = {{Findings of the 2019 Conference on Machine Translation (WMT19)}},
  author = {Barrault, Loïc and Bojar, Ondřej and Costa-jussà, Marta R. and Federmann, Christian and Fishel, Mark and Graham, Yvette and Haddow, Barry and Huck, Matthias and Koehn, Philipp and Malmasi, Shervin and Monz, Christof and Müller, Mathias and Pal, Santanu and Post, Matt and Zampieri, Marcos},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)},
  year = {2019},
  doi = {10.18653/v1/w19-5301},
  url = {https://doi.org/10.18653/v1/w19-5301}
}

References

133 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ergun Biçici. 2019. Machine Translation with parfda, Moses, kenlm, nplm, and PRO. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of the Sixth International Conference on Learning Representations. find our card
  3. Nikolay Bogoychev, Marcin Junczys-Dowmunt, Kenneth Heafield, and Alham Fikri Aji. 2018. Accelerating asynchronous stochastic gradient descent for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  4. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, and Hans Uszkoreit. 2019. Linguistic Evaluation of German-English Machine Translation Using a Test Suite. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  5. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the… find our card
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find our card
  7. Pau Baquero-Arnal, Javier Iranzo-Sánchez, Jorge Civera, and Alfons Juan. 2019. The MLLP-UPV Spanish-Portuguese and Portuguese-Spanish Machine Translation Systems for WMT19 Similar Language Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association… find
  8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceed… find our card
  9. Rachel Bawden, Nikolay Bogoychev, Ulrich Germann, Roman Grundkiewicz, Faheem Kirefu, Antonio Valerio Miceli Barone, and Alexandra Birch. 2019a. The University of Edinburgh’s Submissions to the WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Ital… find
  10. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A… find our card
  11. Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mariana Neves, Felipe Soares, Amy Siu, Karin Verspoor, and Maika Vicente Navarro. 2019b. Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluati… find
  12. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… find our card
  13. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Chao Bei, Hao Zong, Conghu Yuan, Qingming Liu, and Baoyong Fan. 2019. GTCOM Neural Machine Translation Systems for WMT19. In Proceedings 2010. Findings of th… find
  14. Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 Conference on Machine Translation. In Proceedings of the First Conference on M… find our card
  15. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–48, Montreal, Canada. Association for Computation… find our card
  16. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proc… find our card
  17. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1–28, Athens, Greece. Association for Computational Linguistics. find
  18. Fethi Bougares, Jane Wottawa, Anne Baillot, Loïc Barrault, and Adrien Bardet. 2019. LIUM’s Contributions to the WMT2019 News Translation Task: Data and Systems for German-French Language Pairs. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computati… find
  19. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 Workshop on Statistical Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics. find
  20. Eleftheria Briakou and Marine Carpuat. 2019. The University of Maryland’s Kazakh-English Neural Machine Translation System at WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  21. Noe Casas, José A. R. Fonollosa, Carlos Escolano, Christine Basta, and Marta R. Costa-jussà. 2019. The TALP-UPC Machine Translation Systems for WMT19 News Translation Task: Pivoting Techniques for Low Resource MT. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Assoc… find
  22. Sari Dewi Budiwati, Al Hafiz Akbar Maulana Siagian, Tirana Noor Fatyanosa, and Masayoshi Aritsugi. 2019. DBMS-KU Interpolation for WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  23. Rajen Chatterjee, Christian Federmann, Matteo Negri, and Marco Turchi. 2019. Findings of the WMT 2019 Shared Task on Automatic Post-Editing. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  24. Franck Burlot. 2019. Lingua Custodia at WMT’19: Attempts to Control Terminology. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  25. Yu Chen and Tania Avgustinova. 2019. Machine Translation from an Intercomprehension Perspective. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  26. Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) Evaluation of Machine Translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. find
  27. Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361, Berlin, Germany. Association for Computational Linguistics. find our card
  28. Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further Meta-Evaluation of Machine Translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, Ohio. Association for Computational Linguistics. find
  29. Marta R. Costa-jussà, Marcos Zampieri, and Santanu Pal. 2018. A Neural Approach to Language Variety Translation. In Proceedings of VarDial. find
  30. Fabien Cromieres and Sadao Kurohashi. 2019. Kyoto University Participation to the WMT 2019 News Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. find
  31. Shared Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. Conference of the European Chapter of the Association for Computational Linguistics, pages 443– 451, Gothenburg, Sweden. Association for Computational Linguistics. find
  32. Raj Dabre, Kehai Chen, Benjamin Marie, Rui Wang, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019. NICT’s Supervised Neural Machine Translation Systems for the WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Comput… find
  33. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, pages 1–28. find
  34. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019a. Translationese in machine translation evaluation. CoRR, abs/1906.09833. find
  35. Liang Ding and Dacheng Tao. 2019. The University of Sydney’s Machine Translation System for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  36. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019b. Translationese in machine translation evaluation and mt checklist. CoRR. find
  37. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium. Association for Computational Linguistics. find
  38. Stig-Arne Grönroos, Sami Virpioja, and Mikko Kurimo. 2018. Cognate-aware morphological segmentation for multilingual neural translation. CoRR, abs/1808.10791. find
  39. Xinze Guo, Chang Liu, Xiaolong Li, Yiran Wang, Guoliang Li, Feng Wang, Zhitao Xu, Liuyi Yang, Li Ma, and Changliang Li. 2019. Kingsoft’s Neural Machine Translation System for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguis… find
  40. Cristina España-Bonet and Dana Ruiter. 2019. UdS-DFKI Participation at WMT 2019: Low-Resource (en-gu) and Coreference-Aware (en-de) Systems. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  41. Jeremy Gwinnup, Grant Erdmann, and Tim Anderson. 2019. The AFRL WMT19 Systems: Old Favorites and New Tricks. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  42. Christian Federmann. 2012. Appraise: an open-source toolkit for manual evaluation of mt output. The Prague Bulletin of Mathematical Linguistics, 98:25– 35. find
  43. Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 Shared Task on Quality Estimation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  44. Jan Hajič, Eva Hajičová, Jarmila Panevová, Petr Sgall, Ondřej Bojar, Silvie Cinková, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří Semecký, Jana Šindlerová, Jan Štěpánek, Josef Toman, Zdeňka Urešová, and Zdeněk Žabokrtský. 2012. Announcing Prague Czech-English Dependency Treeba… find
  45. Pablo Gamallo, José Ramom Pichel, and Iñaki Alegria. 2017. From language identification to language distance. Physica A: Statistical Mechanics and its Applications, 484:152 – 162. find
  46. Vikrant Goyal and Dipti Misra Sharma. 2019. The IIIT-H Gujarati-English Machine Translation System for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  47. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… find our card
  48. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… find
  49. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is Machine Translation Getting Better over Time? In Proceedings of the 14th Hossein Hassani. 2017. Kurdish Interdialect Machine Translation. Proceedings of VarDial. find
  50. Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  51. Chris Hokamp, John Glover, and Demian Gholipour Ghalandari. 2019. Evaluating the Supervised and Zero-shot Performance of Multilingual Translation Models. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  52. Philipp Koehn, Kevin Duh, and Brian Thompson. 2018. The JHU Machine Translation Systems for WMT 2018. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  53. Javier Iranzo-Sánchez, Gonçal Garcés Díaz-Munío, Jorge Civera, and Alfons Juan. 2019. The MLLP-UPV Supervised Machine Translation Systems for WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  54. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  55. Marcin Junczys-Dowmunt. 2018a. Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  56. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Christopher J. Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translatio… find
  57. Marcin Junczys-Dowmunt. 2018b. Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  58. Marcin Junczys-Dowmunt. 2019. Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  59. Philipp Koehn and Christof Monz. 2006. Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121, New York City. Association for Computational Linguistics. find
  60. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast Neural Machine Translation in C++. In Proceedings of ACL 2018, System… find our card
  61. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 1318–1326, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  62. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226. find our card
  63. Jyotsana Khatri and Pushpak Bhattacharyya. 2019. Utilizing Monolingual Data in NMT for Similar Languages: Submission to Similar Language Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  64. Ivana Kvapilíková, Dominik Macháček, and Ondřej Bojar. 2019. CUNI Systems for the Unsupervised News Translation Task in WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  65. James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural netwo… find
  66. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. find
  67. Sahinur Rahman Laskar, Partha Pakray, and Sivaji Bandyopadhyay. 2019. Neural Machine Translation: Hindi-Nepali. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  68. Tom Kocmi and Ondřej Bojar. 2019. CUNI Submission for Low-Resource Languages in WMT News 2019. In Proceedings of the Fourth Conference on evaluation of German-English Machine Translation based on a Test Suite. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Associa… find
  69. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Neural Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. In EMNLP 2018, Brussels, Belgium. Association for Computational Linguistics. find
  70. Kelly Marchisio, Yash Kumar Lal, and Philipp Koehn. 2019. Johns Hopkins University Submission for WMT News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  71. Bei Li, Yinqiao Li, Chen Xu, Ye Lin, Jiqiang Liu, Hui Liu, Ziyang Wang, Yuhao Zhang, Nuo Xu, Zeyang Wang, Kai Feng, Hexuan Chen, Tengbo Liu, Yanyang Li, Qiang Wang, Tong Xiao, and Jingbo Zhu. 2019a. The NiuTrans Machine Translation Systems for WMT19. In Proceedings of the Fourth Conference on Machin… find
  72. Benjamin Marie, Raj Dabre, and Atsushi Fujita. 2019a. NICT’s Machine Translation Systems for the WMT19 Similar Language Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  73. Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Pino, and Hassan Sajjad. 2019b. Findings of the First Shared Task on Machine Translation Robustness. In Proceedings of the Fourth Conference on Machine Translation, Florenc… find
  74. Benjamin Marie and Atsushi Fujita. 2018. A smorgasbord of features to combine phrase-based and neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), page 111–124, Boston, MA. Association for Machine T… find
  75. Zhenhao Li and Lucia Specia. 2019. A Comparison on Fine-grained Pre-trained Embeddings for the WMT19Chinese-English News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  76. Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019b. NICT’s Unsupervised Neural and Statistical Machine Translation Systems for the WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. A… find
  77. Patrick Littell, Chi-kiu Lo, Samuel Larkin, and Darlene Stewart. 2019. Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  78. Luıs Marujo, Nuno Grazina, Tiago Luıs, Wang Ling, Luısa Coheur, and Isabel Trancoso. 2011. BP2EP– Adaptation of Brazilian Portuguese texts to European Portuguese. In Proceedings of EAMT. find
  79. Zihan Liu, Yan Xu, Genta Indra Winata, and Pascale Fung. 2019. Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  80. Alexander Molchanov. 2019. PROMT Systems for WMT 2019 Shared Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  81. Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018. Neural architecture optimization. In Advances in neural information processing systems, pages 7816–7827. find
  82. Riktim Mondal, Shankha Raj Nayek, Aditya Chowdhury, Santanu Pal, Sudip Kumar Naskar, and Josef van Genabith. 2019. JU-Saarland Submission to the WMT2019 English–Gujarati Translation Shared Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computat… find
  83. Jerry Ma and Denis Yarats. 2018. Quasi-hyperbolic momentum and Adam for deep learning. arXiv preprint arXiv:1810.06801. — [arXiv:1810.06801](https://arxiv.org/abs/1810.06801) find
  84. Roberto Navigli and Simone Paolo Ponzetto. 2012. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell., 193:217–250. find
  85. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT 2019 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  86. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 News Translation Task Submission. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find our card
  87. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, and Hans Uszkoreit. 2018. find
  88. Fine-grained Tommi Pirinen. 2019. Apertium-fin-eng–Rule-based Shallow Machine Translation for WMT 2019 Shared Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find our card
  89. Atul Kr. Ojha, Ritesh Kumar, Akanksha Bansal, and Priya Rani. 2019. Panlingua-KMI MT System for Similar Language Translation Task at WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  90. Martin Popel. 2018. CUNI Transformer Neural MT System for WMT18. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  91. Csaba Oravecz, Katina Bontcheva, Adrien Lardilleux, László Tihanyi, and Andreas Eisele. 2019. eTranslation’s Submissions to the WMT 2019 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  92. Martin Popel, Dominik Macháček, Michal Auersperger, Ondřej Bojar, and Pavel Pecina. 2019. English-Czech Systems in WMT19: Document-Level Transformer. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  93. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  94. Maja Popović. 2019. Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  95. Santanu Pal, Marcos Zampieri, and Josef van Genabith. 2019. UDS–DFKI Submission to the WMT2019 Czech–Polish Similar Language Translation Shared Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  96. Michael Przystupa and Muhammad Abdul-Mageed. 2019. Neural Machine Translation of Low-Resource and Similar Languages with Backtranslation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  97. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Philadelphia, Pennsylvania. find our card
  98. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association f… find
  99. Mārcis Pinnis. 2018. Tilde’s Parallel Corpus Filtering Methods for WMT 2018. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find
  100. Matı̄ss Rikters. 2018. Impact of Corpora Quality on Neural Machine Translation. In In Proceedings of the 8th Conference Human Language Technologies - The Baltic Perspective (Baltic HLT 2018), Tartu, Estonia. find
  101. Mārcis Pinnis, Rihards Krišlauks, Daiga Deksne, and Toms Miks. 2017. Neural Machine Translation for Morphologically Rich Languages with Improved Sub-word Units and Synthetic Data. In Proceedings of the 20th International Conference of Text, Speech and Dialogue (TSD2017), volume 10415 LNAI, Prague,… find
  102. Jan Rosendahl, Christian Herold, Yunsu Kim, Miguel Graça, Weiyue Wang, Parnia Bahar, Yingbo Gao, and Hermann Ney. 2019. The RWTH Aachen University Machine Translation Systems for WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational… find
  103. Marcis Pinnis, Rihards Krišlauks, and Matiss Rikters. 2019. Tilde’s Machine Translation Systems for WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  104. Kateřina Rysová, Lucie Rysová, Magdaléna Poláková, Tomáš Musil, and Ondřej Bojar. 2019. Manual Evaluation of Discourse Relations Translation Accurracy in Document Level NMT. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  105. Mārcis Pinnis, Andrejs Vasil , jevs, Rihards Kalnin , š, Roberts Rozis, Raivis Skadin , š, and Valters Šics. 2018. Tilde MT Platform for Developing Client Specific MT Solutions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan… find
  106. Yves Scherrer, Raúl Vázquez, and Sami Virpioja. 2019. The University of Helsinki Submissions to at WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. the WMT19 Similar Language Translation Task. In Proceedings of the Fou… find
  107. Dario Stojanovski, Viktor Hangya, Matthias Huck, and Alexander Fraser. 2019. The LMU Munich Unsupervised Machine Translation System for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  108. Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019. IITP-MT System for Gujarati-English News Translation Task at WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  109. Roman Sudarikov, Martin Popel, Ondřej Bojar, Aljoscha Burchardt, and Ondřej Klejch. 2016. Using MT-ComparEval. In Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 76–82. find
  110. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The University of Edinburgh’s Neural MT Systems for WMT17. In Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark. Assoc… find
  111. Meng Sun, Bojian Jiang, Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2019. Baidu Neural Machine Translation Systems for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  112. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  113. Víctor M. Sánchez-Cartagena, Juan Antonio Pérez-Ortiz, and Felipe Sánchez-Martínez. 2019. The Universitat d’Alacant Submissions to the English-to-Kazakh News Translation Task at WMT 2019. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational L… find
  114. Leslie N. Smith. 2018. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. CoRR, abs/1803.09820. — [arXiv:1803.09820](https://arxiv.org/abs/1803.09820) find
  115. Aarne Talman, Umut Sulubacak, Raúl Vázquez, Yves Scherrer, Sami Virpioja, Alessandro Raganato, Arvi Hurskainen, and Jörg Tiedemann. 2019. The University of Helsinki Submissions to the WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Associ… find
  116. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, pages 223–231. find our card
  117. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926–5936. find our card
  118. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  119. Felix Stahlberg, Adrià de Gispert, Eva Hasler, and Bill Byrne. 2017. Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short… find
  120. Jörg Tiedemann and Lars Nygaard. 2004. The opus corpus-parallel and free: http://logos.uio.no/opus. In Proceedings of LREC. ↗
  121. Felix Stahlberg, Danielle Saunders, Adrià de Gispert, and Bill Byrne. 2019. CUED@WMT19:EWC&LMs. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  122. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium. Association for Computational Linguistics. find our card
  123. Dario Stojanovski and Alexander Fraser. 2019. find
  124. Combining Local and Document-Level Context: The LMU Munich Neural Machine Translation System Antonio Toral, Lukas Edman, Galiya Yeshmagambetova, and Jennifer Spenader. 2019. Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data. In Proceedings of the Fourth… find
  125. Andre Tättar, Elizaveta Korotkova, and Mark Fishel. 2019. University of Tartu’s Multilingual Multi-domain WMT19 News Translation Shared Task Submission. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  126. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  127. Tereza Vojtěchová, Michal Novák, Miloš Klouček, and Ondřej Bojar. 2019. SAO WMT19 Test Suite: Machine Translation of Audit Reports. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  128. Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, Cheng Xiang Zhai, and Tie-Yan Liu. 2019. Multi-agent dual learning. In International Conference on Learning Representations. find
  129. Yingce Xia, Xu Tan, Fei Tian, Fei Gao, Weicong Chen, Yang Fan, Linyuan Gong, Yichong Leng, Renqian Luo, Yiren Wang, Lijun Wu, Jinhua Zhu, Tao QIN, and Tie-Yan Liu. 2019. Microsoft Research Asia’s Systems for WMT19. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Asso… find
  130. Jingyi Zhang and Josef van Genabith. 2019. DFKI-NMT Submission to the WMT19 News Translation Task. In Proceedings of the Fourth Conference on Machine Translation, Florence, Italy. Association for Computational Linguistics. find
  131. Xiaoheng Zhang. 1998. Dialect MT: A Case Study Between Cantonese and Mandarin. In Proceedings of ACL. find
  132. Jinhua Zhu, Fei Gao, Lijun Wu, Yingce Xia, Tao Qin, Wengang Zhou, Xueqi Cheng, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. CoRR, abs/1905.10523. find
  133. NEU BTRANS ONLINE -B BTRANS- ENSEMBLE UEDIN ONLINE -Y NICT 0.10† 0.07 0.01 0.00 --0.01 -0.03 -0.09 -0.14 -0.14 -0.19 -0.26 -0.39 -0.68 -1.15 0.11† 0.08? 0.02 0.01 0.01 --0.02 -0.08 -0.13 -0.14 -0.19 -0.25 -0.39 -0.67 -1.14 0.13‡ 0.10† 0.04 0.03 0.03 0.02 --0.06 -0.11 -0.11 -0.16 -0.23 -0.36 -0.64 -1… find