Elamur Research
☰
home / papers / 2020 / On The Evaluation of Machine Translation Systems Trained With Back-Translation

On The Evaluation of Machine Translation Systems Trained With Back-Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.253.pdf

Summary

Back-translation (BT) is a standard way to exploit target-side monolingual data in NMT. However, BLEU gains from BT appear mostly on test sentences whose source is itself a human translation (translationese). Critics have argued that BT therefore helps only because such inputs resemble synthetic training sources. Edunov, Ott, Ranzato and Auli test this claim with fairseq Transformer Big models trained with and without BT (OP = parallel data only). They cover En-De, De-En, En-Ru and Ru-En, using WMT'18/WMT'19 data and the winning Facebook-FAIR systems. They also commissioned professional sentence-level translations and double translations of newstest subsets. These let them separate input effects from reference effects. Human evaluators prefer BT outputs for both naturally occurring and translationese sources. BLEU misses this in the direct portion because its references are translationese, and BT and OP outputs sit about equally close to them. BT outputs are measurably more fluent. The authors therefore recommend reporting a language-model perplexity next to BLEU. The work matters because it defends BT as a technique and exposes a systematic blind spot in reference-based evaluation.

Key points

  • Training data: En-De used 5.18M WMT'18 sentence pairs without ParaCrawl, plus sampled BT of 226M German newscrawl sentences. The WMT'19 directions used parallel data including ParaCrawl: 27.7M pairs for English–German and 26M for English–Russian. Monolingual data was 424M English and 76M Russian newscrawl sentences.
  • Test design: Test sets were split into direct (X→Y*) and reverse (X*→Y) portions. For En↔De, 500 source-original and 500 target-original newstest2014 sentences were sampled. Extra double translations (X**, Y**) let the authors check separately whether translationese is easier as input and as reference.
  • Key finding: Translationese inputs are easier to translate for both BT and OP models, not only for BT models. Human judgement of BT systems correlates poorly with BLEU.
  • Fluency evidence:
    • In blind pairwise ratings without the source (100 judgements per direction), native speakers of English, German and Russian rated BT significantly more fluent than OP (p=0.05).
    • Language models trained on system outputs also showed that BT outputs are closer to natural target text.
  • Proposed metric: Score outputs with a single LM trained on Common Crawl (315M English, 284M German and 120M Russian sentences), kept disjoint from the BT monolingual data. In all four directions this LM gave BT lower perplexity than OP (e.g., 57.6 vs 68.6), despite similar BLEU. The authors suggest using it to break BLEU ties.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

58 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 54 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.253

@inproceedings{edunov2020evaluation,
  title = {{On The Evaluation of Machine Translation Systems Trained With Back-Translation}},
  author = {Edunov, Sergey and Ott, Myle and Ranzato, Marc'Aurelio and Auli, Michael},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.253},
  url = {https://doi.org/10.18653/v1/2020.acl-main.253}
}

References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853. ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  3. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and technology: In honour of John Sinclair, 233:250. find
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measure for Machine Translation. find our card
  5. Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proc. of WMT. find
  6. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proc. of WMT. find our card
  7. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proc. of WMT. find our card
  8. Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proc. of WMT. find our card
  9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP. find
  10. Markus Freitag, Isaac Caswell, and Scott Roy. 2019. Text repair model for neural machine translation. arXiv, abs/1904.04790. ↗
  11. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML. find our card
  12. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proc. of 2nd Workshop on Neural Machine Translation and Generation. find our card
  13. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP. find
  14. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proc. of ACL. find
  15. David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proc. of MT Summit. find
  16. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based and neural unsupervised machine translation. In EMNLP. find
  17. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. In Proc. of EMNLP. find
  18. Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proc. of ACL: Demonstrations. find our card
  19. Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proc. of WMT. find our card
  20. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proc. of WMT. find our card
  21. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL: Demonstrations. find our card
  22. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT. find our card
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL. Alberto Poncelas, Dimitar Sht. find our card
  24. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018a. Investigating backtranslation in neural machine translation. arXiv, 1804.06189. Alberto Poncelas, Dimitar Sht. find
  25. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018b. Investigating backtranslation in neural machine translation. arXiv, 1804.06189. find
  26. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proc. of WMT. find our card
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL. find
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL. find
  29. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA. find our card
  30. Milos Stanojevic and Khalil Sima’an. 2014. Beer: Better evaluation as ranking. In Proc. of WMT. find
  31. Sara Stymne. 2017. The effect of translationese on tuning for statistical machine translation. In Proc. of NoDaLiDa. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS. find
  33. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proc. of WMT. find our card
  34. Gideon Toury. 2012. Descriptive translation studies and beyond: Revised edition, volume 100. John Benjamins Publishing. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS. find
  36. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. arXiv, abs/1906.08069. ↗
  37. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv, abs/1904.09675. ↗