Elamur Research
☰
home / papers / 2018 / On the Impact of Various Types of Noise on Neural Machine Translation

On the Impact of Various Types of Noise on Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-2709.pdf

Summary

The paper asks which kinds of flaws in parallel training data damage translation quality, and whether neural and statistical systems react differently. The motivation is an experiment on German–English: adding raw web-crawled Paracrawl data to clean WMT 2017 training data slightly improved a statistical system but sharply degraded a neural one. The authors manually labelled 200 crawled sentence pairs to see which errors occur in practice. Based on that study, they defined five synthetic noise types: misaligned sentences, misordered words, wrong language, untranslated sentences, and short segments. They mixed each type into a clean corpus in increasing proportions and trained RNN attention models in Marian and phrase-based systems in Moses. For nearly every noise type, the neural models suffered more than the statistical ones. The worst case was source text copied untranslated into the target side, which taught the neural system to copy its input. The results show that corpus cleaning matters much more for NMT, and they point to which types of noise filtering should target first.

Key points

  • Adding an equal amount of raw Paracrawl data changed BLEU from 27.2 to 17.3 (−9.9) for NMT and from 24.0 to 25.2 (+1.2) for SMT.
  • In the manual Paracrawl sample, misalignment was the most common problem (41%). Wrong-language content made up 23%, untranslated pairs 4%, and only 23% of pairs were judged okay.
  • The clean training data (Europarl, News Commentary, Rapid; about 83M tokens per language) was mixed with noise at 5–100% of its size. Evaluation used newstest2017, and the NMT models used BPE with a 50,000 vocabulary.
  • Adding just 10% untranslated-target data made 57% of NMT outputs closer to the source than to the reference, by TER. Learning curves showed the models overfitting to the copied portion of the data.
  • Both NMT and SMT were surprisingly robust to German–French data in the training set. Even at a 100% noise ratio, only 6.7% of NMT outputs were in French, which the authors attribute to the domain mismatch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 141 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-2709

@inproceedings{khayrallah2018impact,
  title = {{On the Impact of Various Types of Noise on Neural Machine Translation}},
  author = {Khayrallah, Huda and Koehn, Philipp},
  booktitle = {Proceedings of the 2nd Workshop on Neural Machine Translation and Generation},
  year = {2018},
  doi = {10.18653/v1/w18-2709},
  url = {https://doi.org/10.18653/v1/w18-2709}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web. Association for Computational Linguistics, Portland, Oregon, page… find
  2. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics 33(2). http://www.aclweb.org/anthology-new/J/J07/J07-2003.pdf. ↗ our card
  3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Edinburgh, Scotland, UK., pages 355–362. http://www.aclwe… ↗ our card
  4. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics… find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. http://arxiv.org/pdf/1409.0473v6.pdf. ↗ our card
  6. Lei Cui, Dongdong Zhang, Shujie Liu, Mu Li, and Ming Zhou. 2013. Bilingual data cleaning for smt using graph-based random walk. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Sofia, Bulga… find
  7. Eduard Barbu, Carla Parra Escartı́n, Luisa Bentivogli, Matteo Negri, Marco Turchi, Constantin Orasan, and Marcello Federico. 2016. The first automatic translation memory cleaning shared task. Machine Translation 30(3):145–166. https://doi.org/10.1007/s10590-016-9183-x. — [doi:10.1007/s10590-016-9183-x](https://doi.org/10.1007/s10590-016-9183-x) ↗
  8. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, Volume 1: Research Paper. Association for Computational Linguistics, Copenhagen, Denmark,… find
  9. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. CoRR abs/1711.02173. http://arxiv.org/abs/1711.02173. ↗
  10. Phil Blunsom and Miles Osborne. 2008. Probabilistic inference for machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 215– 223. http://www.aclweb.org/anthology/D08-1023. ↗
  11. Shuoyang Ding, Kevin Duh, Huda Khayrallah, Philipp Koehn, and Matt Post. 2016. The jhu machine translation systems for wmt 2016. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 272–280. http://www.aclweb.org/anthology/W… ↗
  12. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translat… find our card
  13. Shuoyang Ding, Huda Khayrallah, Philipp Koehn, Matt Post, Gaurav Kumar, and Kevin Duh. 2017. The jhu machine translation systems for wmt 2017. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In… find our card
  14. Task Papers. Association for Computational Linguistics, Copenhagen, Denmark, pages 276–282. http://www.aclweb.org/anthology/W17-4724. ↗
  15. Nadir Durrani, Alexander Fraser, Helmut Schmid, Hieu Hoang, and Philipp Koehn. 2013. Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT? In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sofia, B… find
  16. Liang Huang and David Chiang. 2007. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 144–151. http://www.aclweb.org/an… ↗
  17. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada,… find our card
  18. M. Junczys-Dowmunt. 2012. A phrase table without phrases: Rank encoding for better phrase table compression. In Mauro Cettolo, Marcello Federico, Lucia Specia, and Andy Way, editors, Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT). pages 245… find
  19. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the… find our card
  20. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. arXiv preprint arXiv:1804.00344 h… find our card
  21. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). http://www.aclweb.o… ↗ our card
  22. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. http://www.aclweb.org/anthology/D13-1176. ↗
  23. Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 848–856. http://www.aclweb.org/an… ↗
  24. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X). Phuket, Thailand. http://mt-archive.info/MTS-2005-Koehn.pdf. ↗
  25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Christopher J. Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translatio… find
  26. Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver,… find
  27. Barry Haddow, Matthias Huck, Alexandra Birch, Nikolay Bogoychev, and Philipp Koehn. 2015. The edinburgh/jhu phrase-based machine translation systems for wmt 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, pag… find
  28. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, Vancouver, pages 28–39. http://www.aclweb.org/anthology/W17-3204. ↗
  29. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing, John Benjamins, Amsterdam/Philadelphia, Borovets, Bulgaria, volume V. find
  30. Shankar Kumar and William J. Byrne. 2004. Minimum Bayes-Risk Decoding for Statistical Machine Translation. In HLT-NAACL. pages 169–176. find
  31. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). find our card
  32. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the 8th International Conference on Language Resources and Evaluation. find
  33. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. CoRR abs/1803.00047. http://arxiv.org/abs/1803.00047. ↗ our card
  34. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1411–1421. http://aclweb.org/anthology/D1… ↗
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages… find our card
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR abs/1706.03762. http://arxiv.org/abs/1706.03762. ↗
  37. Spencer Rarrick, Chris Quirk, and Will Lewis. 2011. MT detection in web-scraped parallel corpora. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII). International Association for Machine Translation, pages 422–430. http://www.mt-archive.info/MTS-2011-Rarrick.pdf. ↗
  38. Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. 2011. Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association fo… find
  39. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany,… find
  40. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 2935–2940. http://aclweb.org/antho… ↗
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany. http://www.aclweb.org/antholog… ↗
  42. Raivis Skadiņš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. 2014. Billions of parallel words for free: Building and using the eu bookshop corpus. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA… find
  43. Kaveh Taghipour, Shahram Khadivi, and Jia Xu. 2011. Parallel corpus refinement as an outlier detection algorithm. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII). International Association for Machine Translation, pages 414–421. http://www.mt-archive.info/MTS-2011-Taghipour.pd… ↗