Elamur Research⌕
☰
home / papers / 2021 / Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Paper · 2021 · Eva Vanmassenhove, Dimitar Shterionov and Matthew Gwilliam

Summary

The paper asks whether data-driven MT systems flatten language itself, in addition to amplifying social biases such as gender stereotypes. The concern is that they favour frequent patterns and drop rare ones, producing what the authors call "machine translationese." To test this, they trained three types of systems on Europarl for English–French and English–Spanish in both directions:

  • a PB-SMT system built with Moses;
  • LSTM and Transformer NMT models built with OpenNMT-py, using sub-word vocabularies.

They compared each system's output with its original training data using nine diversity measures:

  • standard lexical metrics (TTR, Yule's I, MTLD);
  • a Lexical Frequency Profile adapted from language-acquisition research;
  • Shannon entropy and Simpson's diversity index over the wordforms of each lemma;
  • a new automatic analysis of synonym frequency.

Every metric shows MT output to be lexically and morphologically poorer than the training data, for all pairs and directions. Transformer loses the least, and the losses are sharper when translating into the morphologically richer languages. This matters because MT is used at enormous scale. Shrinking inflectional variety can also stop systems from choosing grammatically correct forms.

Key points

  • Data: Europarl, about 1.47M parallel training sentences each for EN–FR and EN–ES, with roughly 500K (EN–FR) and 460K (EN–ES) test sentences. Systems were tested in four directions: EN→FR, FR→EN, EN→ES and ES→EN.
  • Systems: Moses PB-SMT (5-gram LM, MERT tuning, unknown words masked as UNK). OpenNMT-py models: a 4-layer bidirectional LSTM and a 6-layer, 8-head Transformer.
  • Morphological measure: entropy and Simpson scores are computed only for lemmas with two or more wordforms. Example Shannon H scores (×100) for French: ORIG 75.20, PB-SMT 69.00, LSTM 69.28, Transformer 73.13. For Spanish: 78.42 for ORIG versus 71.79–77.23 for the systems.
  • System ranking: LSTM beats PB-SMT when translating into French and Spanish, but PB-SMT does better into English. On average, the diversity ranking matches the BLEU/TER quality ranking.
  • Novel resources: an LFP implementation that can be applied to any NLP task, and an automatic synonym-frequency evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations🎓Tilburg UniversityTilburg · Netherlands40 works · 6,231 citations🎓Brigham Young UniversityProvo · United States15 works · 40,065 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Safety Research · 221 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Eva Vanmassenhove, Dimitar Shterionov and Matthew Gwilliam. 2021. Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.188

@inproceedings{vanmassenhove2021machine,
  title = {{Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation}},
  author = {Eva Vanmassenhove, Dimitar Shterionov and Matthew Gwilliam},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.188},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.188}
}

References

59 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Moshe Koppel, and Yoav Goldberg. 2014. Automatic detection of machine translated text and translation quality estimation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL2014), Volume 2: Short Papers, pages 289–295. find
  2. Nora Aranberri. 2020. Can translationese features help users select an MT system for post-editing? Procesamiento del Lenguaje Natural, 64:93–100. find
  3. Lyle F Bachman. 2004. Statistical analyses for language assessment. Cambridge University Press. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA. 15pp. our card
  5. Mona Baker. 1999. The role of corpora in investigating the linguistic behaviour of professional translators. International journal of corpus linguistics, 4(2):281–298. find
  6. Marco Baroni and Silvia Bernardini. 2006. A new approach to the study of translationese: Machine-learning the difference between original and translated text. Literary and Linguistic Computing, 21(3):259–274. find
  7. Frederic H. Behr, Victoria Fossum, Michael Mitzenmacher, and David Xiao. 2003. Estimating and comparing entropies across written natural languages using PPM compression. In Proceedings of the 2003 Data Compression Conference (DCC 2003), page 416, Snowbird, UT, USA. find
  8. Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. 2020. How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International Conference on Spoken Language Translation (IWS… find
  9. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349–4357. our card
  10. Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017. Massive Exploration of Neural Machine Translation Architectures. In Proceedings of the Association for Computational Linguistics (ACL2017), pages 1442–1451, Vancouver, Canada. our card
  11. Bram Bulté. 2013. The development of complexity in second language acquisition. A dynamic systems approach (Unpublished doctoral dissertation). find
  12. Bram Bulté, Alex Housen, Michel Pierrard, and Siska Van Daele. 2008. Investigating lexical proficiency development over time–the case of dutch-speaking learners of french in brussels. Journal of French Language Studies, 18(3):277–298. find
  13. Aylin Caliskan-Islam, Joanna J Bryson, and Arvind Narayanan. 2016. Semantics derived automatically from language corpora necessarily contain human biases. arXiv preprint arXiv:1608.07187, pages 1–14. find
  14. Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL2011): Human Language Technologies, V… find
  15. Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open-Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL2007), pages 177–180, Prague, Czech Republic. find
  16. Scott A Crossley, Tom Cobb, and Danielle S Mc-Namara. 2013. Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System, 41(4):965–981. find
  17. Joke Daems, Orphée De Clercq, and Lieve Macken. 2017. Translationese and post-editese: How comparable is comparable quality? Linguistica Antverpiensia New Series-Themes in Translation Studies, 16:89–103. find
  18. Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer… find
  19. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. find
  20. Martin Gellerstam. 1986. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 1:88–95. find
  21. Nizar Habash, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinflection in arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155–165. find
  22. Margherita Ippolito. 2014. Simplification, Explicitation and Normalization: Corpus-Based Research into English to Italian Translations of Children’s Classics. Cambridge Scholars Publishing. find
  23. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations: Poster Session, Banff, Canada. find
  24. Beata Beigman Klebanov and Michael Flor. 2013. Associative texture is lost in translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 27–32, Sofia, Bulgaria. find
  25. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP2004), pages 388–395, Barcelona, Spain. our card
  26. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of The Tenth Machine Translation Summit (MT Summit 2005), pages 79–86, Phuket, Thailand. find
  27. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisti… our card
  28. Svenja Kranich. 2014. Translations as a locus of language contact. In Translation: A multidisciplinary approach, pages 96–115. Springer. find
  29. Kristopher Kyle. 2019. Measuring lexical richness. The Routledge Handbook of Vocabulary Studies, page 454. find
  30. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP2018), pages 4791–4796, Brussels, Belgium. our card
  31. Batia Laufer. 1994. The lexical profile of second language writing: Does it change over time? RELC journal, 25(2):21–33. find
  32. Batia Laufer and Paul Nation. 1995. Vocabulary size and use: Lexical richness in l2 written production. Applied linguistics, 16(3):307–322. find
  33. Philip M McCarthy. 2005. An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity (MTLD). In PhD Thesis, Dissertation Abstracts International, Volume 66:12. University of Memphis, Memphis, Tennessee, USA. find
  34. Jack Merullo, Luke Yeh, Abram Handler, Alvin Grissom II, Brendan O’Connor, and Mohit Iyyer. 2019. Investigating sports commentator bias within a large corpus of American football broadcasts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna… find
  35. IS Paul Nation. 1984. Vocabulary lists: Words, affixes, and stems. English Language Institute, Victoria University of Wellington. find
  36. Michael P Oakes and Meng (eds) Ji. 2013. Quantitative Methods in Corpus-Based Translation Studies: A Practical Guide to Descriptive Translation Research. In Studies in Corpus Linguistics, Volume 51, page 361. John Benjamins Publishing Company, Amsterdam, The Netherlands. find
  37. Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. In Computational Linguistics, Volume 29:1, pages 19–51. MIT Press, Cambridge, Massachusetts, USA. find
  38. Scott E. Page. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton University Press. find
  39. Scott E. Page. 2011. Diversity and Complexity. Princeton University Press. find
  40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002), pages 311–318, Philadephia, PA, USA. our card
  41. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation (WMT 2018): Research Papers, pages 186–191, Belgium, Brussels. our card
  42. Fermın Moscoso del Prado Martın, Aleksandar Kostić, and R Harald Baayen. 2004. Putting the bits together: An information theoretical perspective on morphological processing. Cognition, 94(1):1–18. find
  43. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), Volume 1: Long Papers, pages 1715–1725, Berlin, Germany. our card
  44. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  45. Edward H Simpson. 1949. Measurement of diversity. Nature, 163(4148):688–688. find
  46. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas (AMTA 2006) 200:6, pages 223–231, Austin, Texas, USA. our card
  47. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. our card
  48. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… our card
  49. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (NIPS 2014), pages 3104–3112, Montreal, Quebec, Canada. find
  50. Mildred C. Templin. 1975. Certain Language Skills in Children: Their Development and Interrelationships. Greenwood Press, Westport, Connecticut, USA. find
  51. Antonio Toral. 2019. Post-editese: an exacerbated translationese. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 273–281, Dublin, Ireland. European Association for Machine Translation. find
  52. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT 2018): Research Papers, pages 113–123, Brussels, Belgium. Association for Comput… our card
  53. Eva Vanmassenhove. 2020. On the Integration of Linguistic Features into Statistical and Neural Machine Translation. arXiv preprint arXiv:2003.14324. — [arXiv:2003.14324](https://arxiv.org/abs/2003.14324) find
  54. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. find
  55. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019), pages 3003–3008, Brussels, Belgium. our card
  56. Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In Proceedings of Machine Translation Summit XVII (MT Summit 2019), Volume 1: Research Track, pages 222–232, Dublin, Ireland. our card
  57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of The Thirty-first Annual Conference on Neural Information Processing Systems 30 (NIPS 2017), pages 5998–6008, Long Beach, CA… our card
  58. G. Udnv Yule. 1944. The Statistical Study of Literary Vocabulary. Cambridge University Press. find
  59. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Volume 1: Long Papers, pages 654– 664, V… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vanmassenhove-2021-machine-translationese-effects-algorithmic-bias/