Elamur Research⌕
☰
home / papers / 2019 / Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Findings of the WMT 2019 Shared Task on Automatic Post-Editing

Summary

This report covers the fifth round of the WMT shared task on Automatic Post-Editing (APE). In this task, systems must repair translations from a hidden "black-box" MT engine by learning from (source, MT output, human post-edit) triplets. The 2019 edition used Information Technology data and only neural MT output, in two directions. English-German had 13,442 training triplets and reused the 2018 test set of 1,023 sentences, so results could be compared directly with the previous year. English-Russian was new and was built from Microsoft Office localization data. Organizers also supplied synthetic training corpora, eSCAPE and the 4.5M-triplet "Artificial" set.

Almost every submission was a neural, multi-source model, typically based on the Transformer and trained with synthetic data. On English-German, four teams outperformed the 2018 winner. The best run improved on the baseline by -0.78 TER and +1.23 BLEU. On English-Russian, more than 60% of MT outputs already needed no edits, and no run beat the baseline. The study shows that APE gains shrink as MT quality rises, which makes avoiding over-correction the main open challenge.

Key points

  • Participation: English-German drew 7 teams and 18 runs. English-Russian drew 2 teams with 2 runs each.
  • English-German progress: The baseline scored 16.84 TER and 74.73 BLEU. The top system reached 16.06 TER and 75.96 BLEU, compared with the 2018 best gain of -0.38 TER and +0.8 BLEU. The differences among the top four primary runs were not statistically significant.
  • English-Russian difficulty: The Microsoft Translator baseline scored 16.16 TER and 76.20 BLEU. The best submission reached only 16.59 TER and 75.27 BLEU. This dataset also had the highest repetition rate of any APE dataset released so far.
  • Synthetic data: eSCAPE provides 7.25M phrase-based and 7.25M neural triplets for English-German. A new 7.7M-triplet eSCAPE-NMT set was released for English-Russian.
  • Human evaluation: Direct Assessment via Appraise was run only for English-German. Human post-edits ranked highest, and UNBABEL was the top-scoring system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

38 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 34 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5402

@inproceedings{chatterjee2019findings,
  title = {{Findings of the WMT 2019 Shared Task on Automatic Post-Editing}},
  author = {Chatterjee, Rajen and Federmann, Christian and Negri, Matteo and Turchi, Marco},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)},
  year = {2019},
  doi = {10.18653/v1/w19-5402},
  url = {https://doi.org/10.18653/v1/w19-5402}
}

References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David M. Blei, Andrew Y. Ng, Michael I. Jordan, and John Lafferty. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:2003. find our card
  2. Rajen Chatterjee, Matteo Negri, Marco Turchi, Frédéric Blain, and Lucia Specia. 2018b. Combining quality estimation and automatic post-editing to enhance machine translation output. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research… find
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473. find our card
  5. Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions… find
  6. Gonçalo Correia and André Martins. 2019. A simple and effective approach to automatic post-editing with transfer learning. In To appear at Proceedings of the 57th annual meeting on association for computational linguistics. Association for Computational Linguistics. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  8. Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. find
  9. M Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation, pages 127–137. find our card
  10. WonKee Lee, Jaehun Shin, and Jong-Hyeok Lee. 2019. Transformer-based Automatic Post-Editing Model with Joint Encoder and Multi-source Attention of Decoder. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Italy, August. find
  11. Christian Federmann. 2012. Appraise: An open-source toolkit for manual evaluation of machine translation output. The Prague Bulletin of Mathematical Linguistics, 98:25–35, September. find
  12. Jindr̆ich Libovický, Jindr̆ich Helcl, Marek Tlustý, Ondr̆ej Bojar, and Pavel Pecina. 2016. CUNI at Post-editing and Multimodal Translation Tasks. In Proceedings of the 11th Workshop on Statistical Machine Translation (WMT). find
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria, August. Association for Compu… find
  14. António V. Lopes, M. Amin Farajian, Gonçalo M. Correia, Jonay Trénous, and André F. T. Martins. 2019. Unbabel’s Submission to the WMT2019 APE Shared Task: BERT-based Encoder-Decoder for Automatic Post-Editing. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Ital… find
  15. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting better over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451. find
  16. Matteo Negri, Marco Turchi, Rajen Chatterjee, and Nicola Bertoldi. 2018. eSCAPE: a Large-scale Synthetic Corpus for Automatic Post-Editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018. find
  17. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. find
  18. Santanu Pal, Nico Herbig, Antonio Krüger, and Josef van Genabith. 2019. USAAR-DFKI – The Transference Architecture for English–German Automatic Post-Editing. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Italy, August. find
  19. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing. In Proceedings of the First Conference on Machine Translation, pages 751– 758, Berlin, Germany, August. find our card
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. find our card
  21. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2018. Microsoft and University of Edinburgh at WMT2018: Dual-Source Transformer for Automatic Post-Editing. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium, October. find
  22. Ofir Press and Lior Wolf. 2017. Using the Output Embedding to Improve Language Models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163, Valencia, Spain, April. Association for Computational Linguist… find our card
  23. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72. Association for Computational Linguistics. find our card
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Br… find
  25. Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany, August. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a Toolkit… find our card
  26. Hongfei Xu and Qiuhui Liu. 2019. Neutron: An Implementation of the Transformer Translation Model and its Variants. arXiv preprint arXiv:1903.07402, March. — [arXiv:1903.07402](https://arxiv.org/abs/1903.07402) find
  27. Dimitar Shterionov, Joachim Wagner, and do Carmo Félix. 2019. APE through neural and statistical MT with augmented data. ADAPT/DCU submission to the WMT 2019 APE Shared task. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Italy, August. find
  28. Hongfei Xu, Qiuhui Liu, and Josef van Genabith. 2019. UdS Submission for the WMT 19 Automatic Post-Editing Task. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Italy, August. find
  29. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Brussels, Belgium, Oct… find our card
  30. Michel Simard, Cyril Goutte, and Pierre Isabelle. 2007. Statistical Phrase-Based Post-Editing. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT), pages 508–515, Rochester, New York. find
  31. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231, Cambridge, Massachusetts, USA. find our card
  32. Barret Zoph and Kevin Knight. source neural translation. arXiv:1601.00710. — [arXiv:1601.00710](https://arxiv.org/abs/1601.00710) find
  33. Lucia Specia, Kim Harris, Aljoscha Burchardt, Marco Turchi, Matteo Negri, and Inguna Skadina. 2017. Translation Quality and Productivity: A Study on Rich Morphology Languages. In Proceedings of the 16th Machine Translation Summit, Nagoya, Japan, September. find
  34. Amirhossein Tebbifakhr, Ruchit Agrawal, Rajen Chatterjee, Matteo Negri, and Marco Turchi. 2018. Multi-source Transformer with Combined Losses for Automatic Post-Editing. In Proceedings of the Third Conference on Machine Translation, Brussels, Belgium, October. find
  35. Amirhossein Tebbifakhr, Matteo Negri, and Marco Turchi. 2019. Effort-Aware Neural Automatic Post-Editing. In Proceedings of the Fourth Conference on Machine Translation (WMT19), Florence, Italy, August. find
  36. Marco Turchi, Matteo Negri, and Marcello Federico. 2013. Coping with the subjectivity of human judgements in MT quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 240–251, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  37. Marco Turchi, Matteo Negri, and Marcello Federico. 2014. Data-driven annotation of binary MT quality estimation corpora based on human post-editions. Machine Translation, 28(3):281–308. 2016. Multi-arXiv preprint find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/chatterjee-2019-findings-wmt-2019-shared-task/