Elamur Research⌕
☰
home / papers / 2018 / On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling

On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling

Paper · 2018 · Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti et al.

Summary

The paper asks why supposedly language-independent language models perform so unevenly across languages, and whether linguistic typology can explain it. The authors built a new LM benchmark of 50 languages, chosen by stratified sampling to cover isolating, fusional, introflexive and agglutinative morphology. On it they evaluated n-gram, neural (LSTM) and character-aware (Char-CNN-LSTM) models. They argue that the common fixed-vocabulary setup, which maps rare words to <UNK>, makes perplexity look unrealistically good for morphologically rich languages. They therefore use a full-vocabulary setup in which every training word is kept and only unseen test words count as OOV.

The coarse morphological type of a language predicts absolute LM performance. Type-to-token ratio is the corpus statistic most strongly tied to perplexity. Fine-grained WALS features (fusion, inflectional synthesis, exponence, flexivity) help explain this, because they drive the number of rare word forms and how ambiguous character n-grams are. Character information always helps, but the gain varies by language. The study argues that future multilingual LMs should be informed by typology, especially for languages like Tamil or Korean.

Key points

  • New dataset: 50 typologically diverse languages. Previous multilingual LM sets reached at most 21 EU languages (Europarl), and the most typologically diverse earlier set had only 10 languages (Vania and Lopez, 2017).
  • Effect of the vocabulary setup: a Finnish model reported at perplexity ≈20 with a 5k fixed vocabulary reaches ≈2,000 under the full-vocabulary setup.
  • One-way ANOVA on typological features:
    • fusion has a weak effect (η² = 0.09) on performance;
    • inflectional synthesis has a medium effect (η² = 0.21);
    • exponence shows weak links to the LSTM vs. Char-CNN-LSTM perplexity gap (η² = 0.07 for nouns, 0.04 for verbs).
  • Strongest corpus predictor: type-to-token ratio (TTR), measured by Pearson's correlation with LM performance.
  • Availability: code and data are released at people.ds.cam.ac.uk/dsg40/lmmrl.html.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel28 works · 4,941 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,343 worksMultilingual & cross-lingual NLP · 467 worksMorphology · 277 works

Cite

Cite as Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart and Anna Korhonen. 2018. On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1029

@inproceedings{gerz2018relation,
  title = {{On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling}},
  author = {Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart and Anna Korhonen},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1029},
  url = {https://doi.org/10.18653/v1/d18-1029}
}

References

63 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of EACL, pages 937–947. find
  2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183–192.
  3. Dik Bakker. 2010. Language sampling. In The Oxford handbook of linguistic typology, pages 100–127. Oxford University Press. find
  4. Emily M. Bender. 2013. Linguistic fundamentals for natural language processing: 100 essentials from morphology and syntax. Morgan & Claypool Publishers. find
  5. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Proceedings of CoNLL, pages 31–39. find
  6. Balthasar Bickel and Johanna Nichols. 2013. Inflectional Synthesis of the Verb. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  7. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of EMNLP, pages 745–754. find
  8. Steven Bird. 2011. Bootstrapping the language archive: New prospects for natural language processing in preserving linguistic heritage. Linguistic Issues in Language Technology, 6(4). find
  9. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the 8th Workshop on Statistical Machine Translati…
  10. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In Proceedings of ICML, pages 1899– 1907. find
  11. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2013. One billion word benchmark for measuring progress in statistical language modeling. In Proceedings of IN-TERPSEECH, pages 2635–2639.
  12. Xie Chen, Xunying Liu, Yanmin Qian, MJF Gales, and Philip C Woodland. 2016. CUED-RNNLM: An open-source toolkit for efficient training and evaluation of recurrent neural network language models. In Proceedings of ICASSP, pages 6000 –6004. find
  13. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  14. Nicholas Evans and Stephen C. Levinson. 2009. The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and Brain Sciences, 32(5):429–448. find
  15. Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In Proceedings of EMNLP, pages 360–368.
  16. Ankur Gandhe, Florian Metze, and Ian Lane. 2014. Neural network language models for low resource languages. In Proceedings of INTERSPEECH, pages 2615–2619. find
  17. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. Transactions of the ACL, 6:451– 465.
  18. Joshua T. Goodman. 2001. A bit of progress in language modeling. Computer Speech & Language, 15(4):403–434. find
  19. Edouard Grave, Moustapha Cissé, and Armand Joulin. 2017. Unbounded cache model for online language modeling with open vocabulary. In Proceedings of NIPS, pages 6044–6054. find
  20. Derek Greene and Padraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of ICML, pages 377–384. find
  21. Martin Haspelmath and Andrea Sims. 2013. Understanding morphology. find
  22. Ryan Cotterell and Jason Eisner. 2017. Probabilistic typology: Deep generative models of vowel inventories. In Proceedings of ACL, pages 1182–1192. find
  23. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of ACL, pages 690–696.
  24. Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of NAACL-HLT.
  25. Aurelie Herbelot and Marco Baroni. 2017. High-risk learning: acquiring new word vectors from tiny data. In Proceedings of EMNLP, pages 304–309. find
  26. Ryan Cotterell, Thomas Müller, Alexander Fraser, and Hinrich Schütze. 2015. Labeled morphological segmentation with semi-Markov models. In Proceedings of CoNLL, pages 164–174. find
  27. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL, pages 328–339. find
  28. Joachim Daiber. 2018. Typologically Robust Statistical Machine Translation. Ph.D. thesis, University of Amsterdam. find
  29. Matthew S Dryer. 1989. Large linguistic areas and language sampling. Studies in Language. International Journal sponsored by the Foundation “Foundations of Language”, 13(2):257–292. find
  30. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. In Proceedings of ICML. find
  31. Dan Jurafsky and James H. Martin. 2017. Speech and Language Processing, volume 3. Pearson. find
  32. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of ACL, pages 1492–1502. find
  33. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of AAAI, pages 2741– 2749.
  34. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In Proceedings of ICASSP, pages 181–184. find
  35. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86. find
  36. Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Handwritten digit recognition with a back-propagation network. In Proceedings of NIPS, pages 396–404. find
  37. Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2017. Recurrent additive networks. CoRR, abs/1705.07393. — [arXiv:1705.07393](https://arxiv.org/abs/1705.07393) find
  38. Wang Ling, Tiago Luís, Luís Marujo, Ramón Fernández Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of EMNLP, pages 1520–1530.
  39. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of ACL, pages 1054–1063.
  40. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL, pages 142–150.
  41. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.
  42. John McWhorter. 2001. The world’s simplest grammars are Creole grammars. Linguistic Typology, 5(2):125–66. find
  43. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTERSPEECH, pages 1045–1048. find
  44. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In Proceedings of EMNLP, pages 1992–1997. find
  45. Thomas Müller, Hinrich Schütze, and Helmut Schmid. 2012. A comparative investigation of morphological language modeling for the languages of the European Union. In Proceedings of NAACL-HLT, pages 386–395. find
  46. Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, and Anna Korhonen. 2016. Survey on the use of typological information in natural language processing. In Proceedings of COLING, pages 1297– 1308. find
  47. Michael Paul, Hirofumi Yamamoto, Eiichiro Sumita, and Satoshi Nakamura. 2009. On the importance of pivot language selection for statistical machine translation. In Proceedings of NAACL-HLT, pages 221–224. find
  48. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237.
  49. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of EMNLP, pages 102–112. find
  50. Frans Plank. 2017. Split morphology: How agglutination and flexion mix. Linguistic Typology, 21(2017):1–62. find
  51. Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In Proceedings of SIGIR, pages 275–281.
  52. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2018. Modeling language variation and universals: A survey on typological linguistics for natural language processing. CoRR, abs/1807.00914.
  53. Edoardo Maria Ponti, Ivan Vulić, and Anna Korhonen. 2017. Decoding sentiment from distributed representations of sentences. In Proceedings of *SEM, pages 22–32. find
  54. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of EACL, pages 157–163.
  55. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, pages 379–389. find
  56. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. In Proceedings of the ICML Deep Learning Workshop.
  57. Martin Sundermeyer, Hermann Ney, and Ralf Schluter. 2015. From feedforward to recurrent LSTM neural networks for language modeling. IEEE Transactions on Audio, Speech and Language Processing, 23(3):517–529. find
  58. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of ACL, pages 2016–2027. find
  59. Ivan Vulić, Nikola Mrkšić, Roi Reichart, Diarmuid Ó Séaghdha, Steve Young, and Anna Korhonen. 2017. Morph-fitting: Fine-tuning word vector spaces with simple language-specific rules. In Proceedings of ACL, pages 56–68. find
  60. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In Proceedings of ACL, pages 1319–1329. find
  61. Hamed Zamani and W. Bruce Croft. 2016. Embedding-based query language models. In Proceedings of IC-TIR, pages 147–156. find
  62. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In Proceedings of ICLR (Conference Papers).
  63. George Kingsley Zipf. 1949. Human behavior and the principle of least effort: An introduction to human ecology. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gerz-2018-relation-between-linguistic-typology-limitations/