Elamur Research⌕
☰
home / papers / 2021 / On the Difficulty of Translating Free-Order Case-Marking Languages

On the Difficulty of Translating Free-Order Case-Marking Languages

Paper · 2021 · Arianna Bisazza, Ahmet Üstün and Stephan Sportel

Summary

The paper asks whether languages that pair flexible constituent order with case marking (e.g., Russian, Latin, Tamil) are inherently harder for neural machine translation. Earlier work showed such languages are harder for parsing and subject-verb agreement prediction. Natural corpora differ in size and domain, so the authors instead build synthetic source languages that differ only in chosen typological properties, while the target side stays constant.

They use two setups:

  • A tiny synchronous grammar modeled on English–Dutch, which gives a fully controlled setting.
  • English sentences reordered via their syntactic trees into other fixed orders or random orders, optionally with artificial case suffixes. A new challenge set tests whether argument roles are recovered.

Both BiLSTM-with-attention and Transformer systems are compared. With ample training data, order flexibility costs very little quality. Case marking resolves subject/object ambiguity in sentences that offer no semantic clues. With medium or small training sets, however, fixed-order sources stay ahead. The authors argue this makes data scarcity, combined with typology, a further obstacle to language equality in NLP.

Key points

  • The toy grammar has 6 nouns, 6 verbs and 6 adjectives, yields 10k sentence pairs (80/10/10 split), and is translated into a fixed SVO target. Its source variants are fixed VSO, fixed VOS, and a VSO/VOS mix marked with #S/#O suffixes.
  • Models were built in OpenNMT: a 2-layer BiLSTM (hidden size 500) with Luong attention, and a standard 6-layer Transformer with sinusoidal positional embeddings. Toy runs were repeated five times.
  • The main realistic experiments translate English-derived variants (following Ravfogel et al., 2019) into French. They are evaluated with BLEU and RIBES on Europarl-test and on the new challenge set.
  • Repeating experiments with Czech and Dutch targets (100k training sentences) gave similar RIBES trends. Fixed SOV sources scored best and the case-marked flexible source scored worst, while BLEU differences were minimal.
  • Toy-grammar data and code are released at github.com/573phn/cm-vs-wo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands70 works · 5,142 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Arianna Bisazza, Ahmet Üstün and Stephan Sportel. 2021. On the Difficulty of Translating Free-Order Case-Marking Languages. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00424

@article{bisazza2021difficulty,
  title = {{On the Difficulty of Translating Free-Order Case-Marking Languages}},
  author = {Arianna Bisazza, Ahmet Üstün and Stephan Sportel},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00424},
  url = {https://doi.org/10.1162/tacl_a_00424}
}

References

54 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duygu Ataman and Marcello Federico. 2018. An evaluation of two vocabulary reduction methods for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 97–110. find
  2. Matthew Baerman and Dunstan Brown. 2013. Case syncretism. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015.
  4. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translatio…
  5. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada.…
  6. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745–754, Honolulu, Hawaii. Association for Computational Linguistics. https://doi.org/10.3115 /1613715.… find
  7. Yonatan Bisk and Ke Tran. 2018. Inducing grammars with and for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 25–35, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-2704 ↗
  8. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A…
  9. Emanuele Bugliarello, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. 2020. It’s easier to translate out of English than into it: Measuring neural translation difficulty by cross-mutual information. In Proceedings of the 58th Annual Meeting of the Association for Comp… find
  10. Rahma Chaabouni, Eugene Kharitonov, Alessandro Lazaric, Emmanuel Dupoux, and Marco Baroni. 2019. Word-order biases in deep-agent emergent communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5166–5175, Florence, Italy. Association for Compu… find
  11. David Chiang and Kevin Knight. 2006. An introduction to synchronous grammars. Tutorial available at http://www.isi.edu/ ˜chiang/papers/synchtut.pdf. ↗
  12. Leshem Choshen and Omri Abend. 2019. Automatically extracting challenge sets for non-local phenomena in neural machine translation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 291–303, Hong Kong, China. Association for Computational Linguistics. ht… find
  13. Berlin, Germany. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P16-1078 find
  14. Benrard Comrie. 1981. Language Universals and Linguistic Typology. Blackwell. Book. find
  15. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015a. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336–10341. https://doi.org/10.1073/pnas.1502134112, PubMed: 26240370 — [doi:10.1073/pnas.1502134112](https://doi.org/10.1073/pnas.1502134112) ↗
  16. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
  17. Anna Currey and Kenneth Heafield. 2019. Incorporating source syntax into transformer-based neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 24–33, Florence, Italy. Association for Computational Linguistics. https://doi.org/… find
  18. Matthew S. Dryer. 2013. Order of subject, object and verb. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  19. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. https://wals.info/ ↗
  20. Jinhua Du and Andy Way. 2017. Pre-reordering for neural machine translation: Helpful or harmful? The Prague Bulletin of Mathematical Linguistics, 108(1):171–182. https://doi.org/10.1515/pralin-2017-0018 — [doi:10.1515/pralin-2017-0018](https://doi.org/10.1515/pralin-2017-0018) ↗
  21. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. https://doi.org/10.1207/s15516709cog1402 1 — [doi:10.1207/s15516709cog1402](https://doi.org/10.1207/s15516709cog1402) ↗
  22. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 823–833, Richard Futrell, Kyle Mahowald, and Edward Gibso… find
  23. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,…
  24. Daniel Gildea and David Temperley. 2010. Do grammars minimize dependency length? Cognitive Science, 34(2):286–310. https://doi.org/10.1111/j.1551-6709.2009.01073.x, PubMed: 21564213 — [doi:10.1111/j.1551-6709.2009.01073.x](https://doi.org/10.1111/j.1551-6709.2009.01073.x) ↗
  25. Joseph H. Greenberg. 1963. Some universals of grammar with particular reference to the order of meaningful elements. In Joseph H. Greenberg, editor, Universals of Human Language, pages 73–113. MIT Press, Cambridge, MA. find
  26. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  27. Kristina Gulordava and Paola Merlo. 2015. Diachronic trends in word order freedom and dependency length in dependency-annotated corpora of Latin and ancient Greek. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 121–130, Uppsala, Sweden. Uppsala U…
  28. Kristina Gulordava and Paola Merlo. 2016. Multi-lingual dependency parsing evaluation: A large-scale analysis of word order properties using artificial data. Transactions of the Association for Computational Linguistics, 4:343–356. https://doi.org/10.1162/tacl_a_00103 ↗
  29. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English. In Proceedings of the 2019 Conference on Empirical Metho…
  30. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. https://doi.org/10.1162/tacl_a_00306 ↗
  31. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… — Achieving Human Parity on Automatic Chinese to English News Translation
  32. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10 .1162/neco.1997.9.8.1735, PubMed: 9377276 find
  33. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944–952, Cambridge, MA. Association for Comput…
  34. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  35. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In The Tenth Machine Translation Summit Proceedings of Conference, pages 79–86. International Association for Machine Translation. find
  36. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. https://doi.org/10.1162 /tacl_a_0034… find
  37. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10.18653 /v1/D…
  38. Gary Lupyan and Morten H. Christiansen. 2002. Case, word order, and language learnability: Insights from connectionist modeling. In Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society. find
  39. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. https://doi.org/10.3115/v1/P14-5010 ↗
  40. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. https://doi.org/10.21236/ADA273556 ↗
  41. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic…
  42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Lingu…
  43. Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 425–435. https://doi.org/10 .18653/v1/D18-1…
  44. Martin Popel, Marketa Tomkova, Jakub Tomek, Łukasz Kaiser, Jakob Uszkoreit, Ondřej Bojar, and Zdeněk Žabokrtský. 2020. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11(1):4381. https://doi.org/1… find
  45. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
  46. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon…
  47. Benoı̂t Sagot. 2013. Comparing complexity measures. In Computational Approaches to Morphological Complexity. Paris, France. Surrey Morphology Group. find
  48. Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. 2020. Analyzing the effect of neural network architecture on training performance. In Proceedings of Machine Learning and Systems 2020, pages 9834–9845. find
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  50. Kaius Sinnemäki. 2008. Complexity trade-offs in core argument marking. Language Complexity, pages 67–88. John Benjamins. https://doi.org/10.1075/slcs.94.06sin Dan I. Slobin. 1966. The acquisition of Russian as a native language. The Genesis of Language: A Psycholinguistic Approach, pages 129–148. — [doi:10.1075/slcs.94.06sin](https://doi.org/10.1075/slcs.94.06sin) ↗
  51. Dan I. Slobin and Thomas G. Bever. 1982. Children use canonical sentence schemas: A crosslinguistic study of word order and inflections. Cognition, 12(3):229–265. https://doi.org/10.1016/0010-0277(82)90033-6 — [doi:10.1016/0010-0277(82)90033-6](https://doi.org/10.1016/0010-0277%2882%2990033-6) ↗
  52. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736, Brussels, Belgium. Association for Computational Linguistics. https://… find
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017,…
  54. Nouns president / président man / homme woman / femme minister / ministre candidate / candidat secretary / secrétaire commissioner / commissaire child / enfant teacher / enseignant student / étudiant Verbs thank / remercier support / soutenir represent / représenter defend / défendre welcome /… — The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bisazza-2021-difficulty-translating-free-order-case/