Elamur Research⌕
☰
home / papers / 2019 / Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

Summary

Machine translation is usually judged on accuracy. This paper asks two further questions. Is MT output lexically poorer than human translation? Do MT systems systematically boost frequent words at the expense of rare ones? The authors link this second effect to known MT problems, including gender bias and weak handling of morphologically rich languages. Using Europarl data for English→French and English→Spanish, they train three kinds of systems: phrase-based Moses, attentional RNN and Transformer (OpenNMT-py, word-level vocabularies without BPE). Some systems use the original data, and others use back-translated sources from reverse-direction models, giving 12 systems in total. They test on both seen and unseen data. Lexical diversity is measured with TTR, Yule's I and MTLD, alongside word-frequency analysis and a small manual check of selected words. Every MT system produces less diverse text than human translation. Dominant translations become even more dominant, while rarer variants shrink or vanish. The authors argue this points to algorithmic bias that goes beyond biased training data.

Key points

  • Data: Europarl, roughly 2M sentence pairs per language pair, shuffled and split. Training sets have about 1.47M pairs (EN–FR 1,467,489; EN–ES 1,472,203), and test sets are deliberately large (about 500K and 460K pairs).
  • Systems: Moses (5-gram LM, MERT tuning), a 4-layer bidirectional LSTM RNN and a 6-layer Transformer. The neural models were trained for 150K steps and selected by development-set perplexity. Each architecture was trained on original ("FF") and back-translated ("BACK") data.
  • Lexical diversity (TTR, Yule's I, MTLD) was lower for all MT output than for human text. SMT kept the most richness, and RNNs were among the poorest. Transformer scored best on standard quality metrics.
  • In a manual check of 15 words, the most common Spanish rendering of "picture" ("imagen") became even more frequent in all systems. The rarest alternatives disappeared, and "happen" and "also" showed similar patterns.
  • Unseen test data showed slightly lower MTLD and more words lost entirely than seen data. The authors note that not using BPE may have disadvantaged the neural systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 63 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vanmassenhove2019lost,
  title = {{Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation}},
  author = {Vanmassenhove, Eva and Shterionov, Dimitar and Way, Andy},
  booktitle = {Proceedings of Machine Translation Summit XVII: Research Track},
  year = {2019},
  url = {https://aclanthology.org/W19-6622/}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of International Conference on Learning Representations (ICLR2015), San Diego, USA, May. find our card
  2. Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase-Based Machine Translation Quality: a Case Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 257–267, Austin, Texas, USA, November. find our card
  3. Berman, Antoine. 2000. Translation and the Trials of the Foreign. In The Translation Studies Reader. Routledge London. find
  4. Brezina, Vaclav. 2018. Statistics in Corpus Linguistics: A Practical Guide. Cambridge University Press. find
  5. Britz, Denny, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017. Massive Exploration of Neural Machine Translation Architectures. In Proceedings of the Association for Computational Linguistics (ACL), pages 1442–1451, Vancouver, Canada, July–August. find
  6. Clark, Jonathan H, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL): Human Language Technologies: shor… find
  7. Kingma, Diederik P. and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations: Poster Session, Banff, Canada, April. find
  8. Klebanov, Beata Beigman and Michael Flor. 2013. Associative Texture is Lost in Translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 27–32, Sofia, Bulgaria, August. find
  9. Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open-Source Toolkit for Statistical Machine Translation. In Pro… find
  10. Koehn, Philipp. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, (EMNLP 2004), A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, pages 388–395… find
  11. Koehn, Philipp. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of The Tenth Machine Translation Summit, pages 79– 86, Phuket, Thailand, September. find
  12. Kruger, Haidee. 2012. A Corpus-Based Study of the Mediation Effect in Translated and Edited Language. In Target. International Journal of Translation Studies, Volume 24:2, pages 355–388. John Benjamins Publishing Company, Amsterdam, The Netherlands. find
  13. McCarthy, Philip M and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A Validation Study of Sophisticated Approaches to Lexical Diversity Assessment. In Behavior Research Methods, Volume 2:2, pages 381–392. Springer, Berlin, Germany. find
  14. McCarthy, Philip M. 2005. An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity (MTLD). In PhD Thesis, Dissertation Abstracts International, Volume 66:12. University of Memphis, Memphis, Tennessee, USA. find
  15. Oakes, Michael P and Meng (eds) Ji. 2013. Quantitative Methods in Corpus-Based Translation Studies: A Practical Guide to Descriptive Translation Research. In Studies in Corpus Linguistics, Volume 51, page 361. John Benjamins Publishing Company, Amsterdam, The Netherlands. find
  16. Och, Franz Josef and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. In Computational Linguistics, Volume 29:1, pages 19–51. MIT Press, Cambridge, Massachusetts, USA. find
  17. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL), pages 311–318, Philadephia, PA, USA, July. find our card
  18. Passban, Peyman, Andy Way, and Qun Liu. 2018. Tailoring Neural Architectures for Translating from Morphologically Rich Languages. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 3134–3145, Santa Fe, New-Mexico, USA, August. find
  19. Poncelas, Alberto, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating Backtranslation in Neural Machine Translation. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation (EAMT), pages 249–258, Alacant, S… find our card
  20. Prates, Marcelo OR, Pedro HC Avelar, and Luis Lamb. 2019. Assessing Gender Bias in Machine Translation–A Case Study with Google Translate. In Neural Computing and Applications. Springer, Berlin, Germany, March. find
  21. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL): Long Papers, pages 86– 96, Berlin, Germany, August. find
  22. Language Processing (EMNL), pages 2979–2989, Copenhagen, Denmark, September. find
  23. Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas (AMTA) 200:6, pages 223–231, Austin, Texas, USA, October. find our card
  24. Templin, Mildred C. 1975. Certain Language Skills in Children: Their Development and Interrelationships. Greenwood Press, Westport, Connecticut, USA. find
  25. Unanue, Inigo Jauregi, Lierni Garmendia Arratibel, Ehsan Zare Borzeshi, and Massimo Piccardi. 2018. English-Basque statistical and neural machine translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), pages 880–885, Miyazaki, Japan, May… find
  26. Vanmassenhove, Eva, Jinhua Du, and Andy Way. 2016. Improving Subject-Verb Agreement in SMT. In Proceedings of the Fifth Workshop on Hybrid Approaches to Translation: HyTra (EAMT), Riga, Latvia, June. find
  27. Vanmassenhove, Eva, Christian Hardmeier, and Andy Way. 2018. Getting Gender Right in Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3003–3008, Brussels, Belgium, Novemebr–October. find our card
  28. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of The Thirty-first Annual Conference on Neural Information Processing Systems 30 (NIPS), pages 5998–6008, Long Beach, CA, US… find
  29. Wong, Billy and Chunyu Kit. 2012. Extending Machine Translation Evaluation Metrics with Lexical Cohesion to Document Level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNL), pages 1060–1068, Jeju Island… find
  30. Yule, G. Udny. 1944. The Statistical Study of Literary Vocabulary. Cambridge University Press, Cambridge, USA. find
  31. Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also Like Shopping: Reducing Gender Bias Amplification Using Corpus-Level Constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vanmassenhove-2019-lost-translation-loss-decay-linguistic/