Elamur Research
☰
home / papers / 2019 / APE at Scale and Its Implications on MT Evaluation Biases

APE at Scale and Its Implications on MT Evaluation Biases

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5204.pdf

Summary

The paper asks whether monolingual data alone can teach a model to repair the stilted "translationese" that NMT systems produce, and what such repairs reveal about BLEU-based evaluation. The authors build synthetic training pairs by round-trip translating (RTT) target-language News Crawl sentences through English. They then train a transformer in lingvo to recover each original sentence from its noisy round-tripped version. This Automatic Post-Editing (APE) model works with any translation system without retraining. It was tested on WMT18 English→German, WMT15 English→French and WMT16 English→Romanian, on outputs from bitext-only systems, noised back-translation systems and top WMT submissions.

The authors split each WMT test set by the language each sentence was originally written in. On the target-original half, BLEU goes up after post-editing. On the source-original half, whose references are human translations, BLEU falls. Human raters nonetheless judged the post-edited output more fluent and more accurate on both halves. The authors conclude that translated references reward translationese. They recommend reporting BLEU separately for each half and call for better, multi-reference test sets.

Key points

  • Training data: German News Crawl 2007–2017 gave 216.5M sentences after filtering, French 2007–2014 gave 34M, and Romanian News Crawl '16 gave 2.2M. The conclusion notes that a subset of about 24M examples is enough to train the APE model.
  • English→German, bitext-only system: APE raised BLEU by 1.5 on newstest2014 and 0.7 on newstest2017 but lowered it by 1.4 on newstest2016. Applying APE only to German-original sentences gave an average gain of 2.2 BLEU.
  • Top WMT submissions: Selectively applying APE improved even the best entries, by up to 2.5 BLEU points.
  • Reverse and iterative APE: A reverse model trained on (y, RTT(y)) pairs scores higher on the English-original half, which supports the bias explanation. Running APE a second time left 75% of sentences unchanged and slightly lowered BLEU.
  • What RTT changes: Round-tripped German scores 40.9 BLEU against the originals. It also has a smaller vocabulary (29,635 vs. 33,814 unique tokens), and APE enlarges the vocabulary of NMT output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

54 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 46 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5204

@inproceedings{freitag2019ape,
  title = {{APE at Scale and Its Implications on MT Evaluation Biases}},
  author = {Freitag, Markus and Caswell, Isaac and Roy, Scott},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)},
  year = {2019},
  doi = {10.18653/v1/w19-5204},
  url = {https://doi.org/10.18653/v1/w19-5204}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Unsupervised Statistical Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632–3642. John Dryden. 1685. Sylvae [Translator’s preface]. A Scolar Press facsimile. Scolar Press. find our card
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised Neural Machine Translation. In International Conference on Learning Representations. find our card
  3. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 489–500. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015. find our card
  5. Marina Fomicheva et al. 2017. The Role of Human Reference Translation in Machine Translation Evaluation. Ph.D. thesis, Universitat Pompeu Fabra. find
  6. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, pages 1243–1252. find our card
  7. Mona Baker. 1993. Corpus Linguistics and Translation Studies: Implications and Applications. Text and technology: in honour of John Sinclair, pages 233–252. find
  8. Martin Gellerstam. 1986. Translationese in Swedish novels translated from English. Translation Studies in Scandinavia, pages 88–95. Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, pages 1318–1326. find
  9. Çağlar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using Monolingual Corpora in Neural Machine Translation. arXiv preprint arXiv:1503.03535. ↗ our card
  10. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised Machine Translation Using Monolingual Corpora Only. In International Conference on Learning Representations. find our card
  11. Çağlar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017. On Integrating a Language Model into Neural Machine Translation. Comput. Speech Lang., pages 137–148. find
  12. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-Based & Neural Unsupervised Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual Learning for Machine Translation. In Conference on Advances in Neural Information Processing Systems (NeurIPS). find
  14. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Language Models for Machine Translation: Original vs. Translated Texts. Computational Linguistics, 38(4). find
  15. Matthieu Hermet and Désilets Alain. 2009. Using First and Second Language Models to Correct Preposition Errors in Second Language Authoring. In Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications, pages 64–72. find
  16. Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Exploring grammatical error correction with not-so-crummy machine translation. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 44–53. Association for Computational Linguistics. find
  17. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative Backtranslation for Neural Machine Translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, volume 1, pages 18–24. find
  18. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing Revisited with Neural Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 881–893. find
  19. Maria Holmqvist, Sara Stymne, Jody Foo, and Lars Ahrenberg. 2009. Improving Alignment for SMT by Reordering and Augmenting the Training Corpus. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 120–124. Association for Computational Linguistics. find
  20. Xing Niu, Michael Denkowski, and Marine Carpuat. 2018. Bi-Directional Neural Machine Translation with Synthetic Parallel Data. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 84–91. find
  21. Marcin Junczys-Dowmunt. 2018. Microsoft’s Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 429–434. Association for Computational Linguistics. find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  23. Jan-Thorsten Peter, Tamer Alkhouli, Hermann Ney, Matthias Huck, Fabienne Braune, Alexander Fraser, Aleš Tamchyna, Ondřej Bojar, Barry Haddow, Rico Sennrich, Frédéric Blain, Lucia Specia, Jan Niehues, Alex Waibel, Alexandre Allauzen, Lauriane Aufrant, Franck Burlot, elena knyazeva, Thomas Lavergne, François Yvon, Mārcis Pinnis, and Stella Frank. 2016. The QT21/HimL Combined Machine Translation System. In Proceedings of the First Conference on Machine Translation (WMT16), pages 344–355. find
  24. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016. Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 751–758. find
  25. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2018. MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 822–826. find
  26. Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating Backtranslation in Neural Machine Translation. In Proceedings of the Moshe Koppel and Noam Ordan. 2011. Translationese and Its Dialects. In Proceedings of the 49th Annual 21st Annual Conference of the European Association for Machine Translation, pages 249–258. find
  27. Marina Warner. 2018. The Politics of Translation. London Review of Books, 40 no. 19:22. find
  28. Martin Popel. 2018. CUNI Transformer Neural MT System for WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 482–487. find
  29. Tian Wu, Zhongjun He, Enhong Chen, and Haifeng Wang. 2018. Improving Neural Machine Translation with Neural Sentence Rewriting. In 2018 International Conference on Asian Language Processing (IALP). find
  30. Matt Post. 2018. A Call for Clarity in Reporting Bleu Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191. find our card
  31. Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. 2017. Dual Supervised Learning. In International Conference on Machine Learning (ICML). find
  32. Friedrich Schleiermacher. 1816. Über die verschiedenen Methoden des Übersetzens. Abhandlungen der Königlichen Akademie der Wissenschaften in Berlin. Walter de Gruyter GmbH. find
  33. Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. 2018. Joint Training for Neural Machine Translation Models with Monolingual Data. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  34. Larry Selinker. 1972. Interlanguage. International Review of Applied Linguistics, pages 209–241. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh Neural Machine Translation Systems for WMT 16. In Proceedings of the First Conference on Machine Translation, pages 371–376. find our card
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96. find
  37. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 464–468. find our card
  38. Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, and Yuan Cao et al. 2019. Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. CoRR, abs/1902.08295. ↗
  39. Felix Stahlberg, Adri de Gispert, and Bill Byrne. 2018. The University of Cambridge’s Machine Translation Systems for WMT18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 508–516. find
  40. Sara Stymne. 2017. The Effect of Translationese on Tuning for Statistical Machine Translation. In The 21st Nordic Conference on Computational Linguistics, pages 241–246. find
  41. Gideon Toury. 1995. Descriptive Translation Studies and Beyond. Benjamins translation library. John Benjamins Publishing Company. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, pages 5998–6008. find