Elamur Research⌕
☰
home / papers / 2018 / Are All Languages Equally Hard to Language-Model?

Are All Languages Equally Hard to Language-Model?

Paper · 2018 · Ryan Cotterell, Sabrina J. Mielke, Jason Eisner et al.

Summary

Multilingual NLP often assumes one modeling technique works equally well for every language. Comparing language models across languages is hard, though, because corpus sizes, topics, spelling conventions and OOV handling all differ.

The authors propose a controlled setup. Open-vocabulary models are trained and tested on utterance-aligned translations from Europarl in 21 languages, so every model must encode roughly the same content. Performance is measured in bits per English character (BPEC). This is total bits per utterance, scaled by the length of the aligned English sentence, so it does not depend on a language's orthography.

Two model families are compared: a hybrid word/character Kneser–Ney n-gram model and a character-level LSTM. Inflectional richness is quantified with counting complexity computed from UniMorph lexicons. Under both models, languages with richer inflection get worse BPEC, but this link disappears once the corpora are lemmatized.

The finding shows that character-level neural models do not automatically overcome morphological richness. It remains unresolved whether such languages carry more information or current models fail to capture predictable morphemes.

Key points

  • Data and splits: the 21 Europarl languages are used, all Indo-European except the Uralic Finnish, Hungarian and Estonian. The split is about 80/10/10 train/dev/test, and characters seen fewer than 100 times are mapped to an unknown symbol.
  • Models: the baseline is a 7-gram hybrid open-vocabulary Kneser–Ney model. The neural model is a character-level PyTorch LSTM with 1024-dimensional embeddings and 2 hidden layers of 1024; it beats the n-gram model on every language.
  • Correlation with inflection: for the LSTM, Spearman's ρ between BPEC on word forms and counting complexity is 0.59 (p < 0.005). On lemmata it falls to −0.13 (p ≈ 0.56), which is not significant.
  • Effect of lemmatization: removing inflection shrinks the spread of BPEC across languages, with the standard deviation dropping from 0.065 to 0.039 bits. Finnish word forms are harder to model than English ones, yet Finnish lemmata are easier.
  • BPEC vs. BPC: ranking languages by BPC misleadingly places English mid-pack, below French. Under BPEC, English is the easiest language for the LSTM, with Danish and Swedish closest to it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Ryan Cotterell, Sabrina J. Mielke, Jason Eisner and Brian Roark. 2018. Are All Languages Equally Hard to Language-Model?. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2085

@inproceedings{cotterell2018are,
  title = {{Are All Languages Equally Hard to Language-Model?}},
  author = {Ryan Cotterell, Sabrina J. Mielke, Jason Eisner and Brian Roark},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2085},
  url = {https://doi.org/10.18653/v1/n18-2085}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and Technology: In Honour of John Sinclair 233:250. find
  2. Maximilian Bisani and Hermann Ney. 2005. Open vocabulary speech recognition with flat hybrid models. In INTERSPEECH. pages 725–728. find
  3. John McWhorter. 2001. The world’s simplest grammars are creole grammars. Linguistic Typology 5(2):125–66. find
  4. Émile Enguehard, Yoav Goldberg, and Tal Linzen. 2017. Exploring the syntactic abilities of RNNs with multi-task learning. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Association for Computational Linguistics, pages 3–14. https://doi. org/10.18653/v… find
  5. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association. pages 1045–1048. http://www.isca-speech.org/archive/ interspe… ↗
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. find
  7. Thomas Müller, Hinrich Schütze, and Helmut Schmid. 2012. A comparative investigation of morphological language modeling for the languages of the European Union. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  8. Kyuyeon Hwang and Wonyong Sung. 2017. Character-level language modeling with hierarchical recurrent neural networks. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pages 5720–5724. find
  9. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Vancouver, Canada, pages 1… find
  10. Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The Annals of Mathematical Statistics pages 400–407. find
  11. Benoı̂t Sagot. 2013. Comparing complexity measures. In Computational Approaches to Morphological Complexity. find
  12. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Arya McCarthy, Sabrina J. Mielke, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. Unimorph 2.0: Universal morphology. In Proceedings of the Ninth Internationa… find
  13. Richard Sproat, Bruno Cartoni, HyunJeong Choe, David Huynh, Linne Ha, Ravindran Rajakumar, and Evelyn Wenzel-Grondie. 2014. A database for measuring linguistic information content. In LREC. find
  14. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In LREC. find
  15. Christo Kirov, John Sylak-Glassman, Rebecca Knowles, Ryan Cotterell, and Matt Post. 2017. A rich morphological tagger for English: Exploring the cross-linguistic tradeoff between morphology and syntax. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational… our card
  16. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association. find
  17. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR abs/1409.2329. http://arxiv.org/ abs/1409.2329. ↗ our card
  18. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In International Conference on Acoustics, Speech, and Signal Processing, ICASSP. pages 181–184. find
  19. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit. volume 5, pages 79–86. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cotterell-2018-are-all-languages-equally-hard/