Elamur Research⌕
☰
home / papers / 2019 / What Kind of Language Is Hard to Language-Model?

What Kind of Language Is Hard to Language-Model?

Paper · 2019 · Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman et al.

Summary

The paper tests whether current language-modeling methods work equally well across languages. It revisits the authors' earlier claim (Cotterell et al., 2018) that rich inflectional morphology makes languages harder to model. That study used the fully parallel Europarl corpus, which covers few languages, most of them related. To go beyond it, the authors fit a paired-sample multiplicative mixed-effects model. The model jointly estimates how much information each aligned sentence ("intent") carries and a difficulty coefficient for each language, and it tolerates missing translations. They train open-vocabulary Char-RNNLM and BPE-RNNLM models on Bible translations from 13 language families (69 languages per the abstract, 62 per the body) and on the 21 Europarl languages. The difficulty estimates are stable across corpora and architectures. However, morphological complexity and WALS typological features do not predict them. Only vocabulary size and, more weakly, sentence length in characters correlate with difficulty. Translationese also proves no easier to model once training data are balanced. The authors warn against generalizing from small language samples and offer a reusable tool for cross-lingual evaluation.

Key points

  • Models: Char-RNNLM, an LSTM using the Merity et al. (2018) implementation with char-PTB settings, and BPE-RNNLM, which uses 0.4|V| BPE merges. Data were split 2/3 train, 1/6 dev and 1/6 test.
  • The earlier morphology finding, measured with MCC (morphological counting complexity), could not be reproduced on the larger language sample. No WALS feature showed a tenable correlation with difficulty.
  • Word inventory size was the strongest predictor of difficulty. Raw character sequence length was a weaker one. Unigram word entropy added nothing beyond vocabulary size.
  • Translationese test: Europarl was split into native and translated sub-languages (42 in total). Native text first looked slightly harder, by 0.027 ± 0.023 across 8 languages. With balanced training on 10 fully parallel Europarl languages, the gap shrank to −0.0044 ± 0.022.
  • Main recommendation: typology is a small-data problem. Future cross-linguistic studies should use far more languages, which this method makes feasible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations🎓City University of New YorkNew York · United States13 works · 2,531 citations🎓The Graduate Center, CUNYNew York · United States6 works · 182 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksMorphology · 277 works

Cite

Cite as Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark and Jason Eisner. 2019. What Kind of Language Is Hard to Language-Model?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1491

@inproceedings{mielke2019what,
  title = {{What Kind of Language Is Hard to Language-Model?}},
  author = {Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark and Jason Eisner},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1491},
  url = {https://doi.org/10.18653/v1/p19-1491}
}

References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  2. Željko Agić, Anders Johannsen, Barbara Plank, Héctor Martı́nez Alonso, Natalie Schluter, and Anders Søgaard. 2016. Multilingual projection for parsing truly low-resource languages. Transactions of the Association for Computational Linguistics, 4:301–312.
  3. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Arya McCarthy, Sabrina J. Mielke, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. Unimorph 2.0: Universal morphology. In Proceedings of the Ninth Internationa… find
  4. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and Technology: In Honour of John Sinclair, pages 233– 250. find
  5. Emily M. Bender. 2009. Linguistically naı̈ve != language independent: Why NLP needs linguistic typology. In EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics, pages 26–32. find
  6. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit, pages 79–86. find
  7. Charles J. Kowalski. 1972. On the effects of non-normality on the distribution of the sample product-moment correlation coefficient. Journal of the Royal Statistical Society. Series C (Applied Statistics), 21(1):1–12. find
  8. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289–300. find
  9. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia.
  10. Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of Statistical Software, Articles, 76(1):1–32. find
  11. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of EACL, pages 255–265. find
  12. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of NAACL, pages 536–541.
  13. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. find
  14. Mathieu Dehouck and Pascal Denis. 2018. A framework for understanding the role of morphology in universal dependency parsing. In Proceedings of EMNLP, pages 2864–2870. find
  15. Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of Life Reviews, 21:171–193. find
  16. Chris Drummond. 2009. Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML. find
  17. Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In Proceedings of LREC, pages 3158–3163.
  18. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240. find
  20. Lawrence Fenton. 1960. The sum of log-normal probability distributions in scatter transmission systems. IRE Transactions on Communications Systems, 8(1):57–67. find
  21. Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, and Yoshua Bengio. 2017. Multiscale sequence modeling with a learned dictionary. arXiv preprint arXiv:1707.00762. — [arXiv:1707.00762](https://arxiv.org/abs/1707.00762) find
  22. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336– 10341. find
  23. Sabrina J. Mielke and Jason Eisner. 2018. Spell once, summon anywhere: A two-level open-vocabulary language model. arXiv preprint arXiv:1804.08205. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) find
  24. NIST Multimodal Information Group. 2010a. find
  25. NIST 2002 Open Machine Translation (OpenMT) evaluation LDC2010T10. find
  26. Johannes Graën, Dolores Batinic, and Martin Volk. 2014. Cleaning the Europarl corpus for linguistic applications. In Konvens, pages 222–227. find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of ACL, pages 1715– 1725.
  28. NIST Multimodal Information Group. 2010b. find
  29. NIST 2003 Open Machine Translation (OpenMT) evaluation LDC2010T11. find
  30. NIST Multimodal Information Group. 2010c. find
  31. NIST 2004 Open Machine Translation (OpenMT) evaluation LDC2010T12. find
  32. Claude E. Shannon. 1951. Prediction and entropy of printed English. Bell Labs Technical Journal, 30(1):50–64. find
  33. NIST Multimodal Information Group. 2010d. find
  34. NIST 2005 Open Machine Translation (OpenMT) evaluation LDC2010T14. find
  35. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of LREC, pages 4290–4297. find
  36. NIST Multimodal Information Group. 2010e. find
  37. NIST 2006 Open Machine Translation (OpenMT) evaluation LDC2010T17. find
  38. Milan Straka and Jana Straková. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In CoNLL 2017 Shared Task: Multilingual parsing from raw text to Universal Dependencies, pages 88–99.
  39. NIST Multimodal Information Group. 2010f. find
  40. NIST 2008 Open Machine Translation (OpenMT) evaluation LDC2010T21. find
  41. NIST Multimodal Information Group. 2010g. find
  42. NIST 2009 Open Machine Translation (OpenMT) evaluation LDC2010T23. find
  43. Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of ICML, pages 1017–1024. find
  44. NIST Multimodal Information Group. 2013a. NIST 2008-2012 Open Machine Translation (OpenMT) progress test sets LDC2013T07. find
  45. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research, pages 1–8.
  46. NIST Multimodal Information Group. 2013b. find
  47. NIST 2012 Open Machine Translation (OpenMT) evaluation LDC2013T03. find
  48. Lewis M. Paul, Gary F. Simons, Charles D. Fennig, et al. 2009. Ethnologue: Languages of the world, 19 edition. SIL International, Dallas. find
  49. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, pages 2227– 2237.
  50. Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419–432. find
  51. Ella Rabinovich, Shuly Wintner, and Ofek Luis Lewinsohn. 2016. A parallel corpus of translationese. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 140–155. Springer. find
  52. Philip Resnik, Mari Broman Olsen, and Mona Diab. 1999. The bible as a parallel corpus: Annotating the ‘book of 2000 tongues’. Computers and the Humanities, 33(1):129–153. find
  53. Benoı̂t Sagot. 2013. Comparing complexity measures. In Computational Approaches to Morphological Complexity. find
  54. S. C. Schwartz and Y. S. Yeh. 1982. On the distribution function and moments of power sums with log-normal components. The Bell System Technical Journal, 61(7):1441–1462. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/