Elamur Research⌕
☰
home / papers / 2021 / Morphology Matters: A Multilingual Language Modeling Analysis

Morphology Matters: A Multilingual Language Modeling Analysis

Paper · 2021 · Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley et al.

Summary

Earlier multilingual language-modeling studies disagreed on whether rich inflectional morphology makes a language harder to model. Some linked difficulty to morphological complexity, while Mielke et al. (2019) attributed it to plain corpus statistics such as type counts. This paper returns to that question at a larger scale. The authors assemble verse-aligned Bible translations in 92 languages and fill gaps in WALS typological data by hand. They then train LSTM language models on text split in several ways: characters, BPE, Morfessor, and FST-based analyzers combined with a BPE or Morfessor fallback. Performance is measured as surprisal per verse and correlated with expert-coded and corpus-derived morphology measures. Under BPE, surprisal rises significantly with several morphological properties, so BPE does not neutralize morphology. Linguistically informed segmentation narrows this gap, and the FST hybrids give the best results wherever an analyzer exists. The work suggests that common subword methods are not truly language-agnostic and that morphological knowledge can improve modeling across languages.

Key points

  • Corpus and split: 145 Bible translations covering 92 languages from 22 families, mostly building on Mielke et al. (2019) data. The data is cut into 30-verse blocks, giving 16,926 training verses and 4,225 verses each for development and test.
  • Morphology measures: 12 WALS features plus four corpus measures: number of types, TTR, MATTR (500-token window) and MLW.
  • Which features matter: Type count was the strongest predictor of BPE surprisal, matching Mielke et al. WALS 20A (fusion) and 22A (verbal inflectional synthesis) showed no effect. Locus of marking and syncretism did show an effect, which earlier work had not reported.
  • Segmentation comparison: Among standard methods, Morfessor gave the lowest surprisal for most languages. Characters beat it for a few, mostly polysynthetic, languages. FST+BPE and FST+Morfessor were best in every language where an FST was available.
  • Code: Released at github.com/hayleypark/MorphologyMatters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations🎓University of ChicagoChicago · United States45 works · 50,617 citations🎓Indiana UniversityBloomington · United States41 works · 3,663 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksMorphology · 405 works

Cite

Cite as Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu and Lane Schwartz. 2021. Morphology Matters: A Multilingual Language Modeling Analysis. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00365

@article{park2021morphology,
  title = {{Morphology Matters: A Multilingual Language Modeling Analysis}},
  author = {Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu and Lane Schwartz},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00365},
  url = {https://doi.org/10.1162/tacl_a_00365}
}

References

41 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antti Arppe, Atticus Harrigan, Katherine Schmirler, Lene Antonsen, Trond Trosterud, Sjur Nørstebø Moshagen, Miikka Silfverberg, Arok Wolvengrey, Conor Snoek, Jordan Lachler, Eddie Antonio Santos, Jean Okimāsis, and Dorothy Thunder. 2014–2019. Finite-state transducer-based computational model of Pla… find
  2. Eric Axelson, Sam Hardwick, Krister Lindén, Kimmo Koskenniemi, Flammie Pirinen, Mikka Silfverberg, and Senka Drobac. 2015. Helsinki finite-state technology resources. find
  3. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B amh, arz, ayr, cmn, esu, heb, ike, ikt, kal, quh, tel, xho. BPE outperformed Morfessor for cmn and heb. find
  4. (Methodological), 57(1):289–300. DOI: https://doi.org/10.1111/j.2517-6161.1995 .tb02031.x Christian Bentz, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardžić. 2016. A comparison between morphological complexity measures: Typological data vs. language corpora. In Proceedings of the Workshop… ↗
  5. The COLING 2016 Organizing Committee. find
  6. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing, 4(1):3:1–3:34. DOI: https://doi.org/10.1145/1187415.1187418 ↗
  7. Mathieu Dehouck and Pascal Denis. 2018. A framework for understanding the role of morphology in universal dependency parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2864–2870, Brussels, Belgium. Association for Computational Linguistics. DOI:… find
  8. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. CoRR, cs.CL/2004.03720v1.
  9. Çaǧrı Çöltekin. 2010. A freely available morphological analyzer for Turkish. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). Valletta, Malta. European Language Resources Association (ELRA). find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Çaǧrı Çöltekin. 2014. A set of open source tools for Turkish natural language processing. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language Resources Association (ELRA). find
  12. Christos Christodoulopoulos and Mark Steedman. 2014. A massively parallel corpus: The Bible in 100 languages. Language Resources and Evaluation, 49:1–21. DOI: https://doi.org/10.1007/s10579-014-9287-y, PMID: 26321896, PMCID: PMC4551210 ↗
  13. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
  14. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  15. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,…
  16. Kimmo Kettunen. 2014. Can type-token ratio be used to show morphological complexity of languages? Journal of Quantitative Linguistics, 21(3):223–245. DOI: https://doi.org/10.1080/09296174.2014.911506 — [doi:10.1080/09296174.2014.911506](https://doi.org/10.1080/09296174.2014.911506) ↗
  17. Michael A. Covington and Joe D. McFall. 2010. Cutting the Gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. DOI: https://doi.org/10.1080/09296171003643098 — [doi:10.1080/09296171003643098](https://doi.org/10.1080/09296171003643098) ↗
  18. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sebastian Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal morphology. In Proceedings of the Eleventh Internatio…
  19. Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3158–3163, Reykjavik, Iceland. European Language Resources Association (ELRA).
  20. Judith L. Klavans. 2018. Computational challenges for polysynthetic languages. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  21. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. CoRR, cs.CL/1803.08240v1. find
  22. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit, pages 79–86. Phuket, Thailand. AAMT. find
  23. Sabrina J. Mielke. 2016. Language diversity in ACL 2004 - 2016. find
  24. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational…
  25. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic…
  26. Sabrina J. Mielke and Jason Eisner. 2019. Spell once, summon anywhere: A two-level open-vocabulary language model. Proceedings of the AAAI Conference on Artificial Intelligence, 33:68436850. DOI: https://doi.org/10.1609/aaai.v33i01.33016843 — [doi:10.1609/aaai.v33i01.33016843](https://doi.org/10.1609/aaai.v33i01.33016843) ↗
  27. Septina Dian Larasati, Vladislav Kuboň, and Daniel Zeman. 2011. Indonesian morphology tool (MorphInd): Towards an indonesian corpus. In Cerstin Mahlow and Michael Piotrowski, editors, Systems and Frameworks for Computational Morphology, Springer Berlin Heidelberg, Berlin, Heidelberg, pages 119–129.… — [doi:10.1007/978-3-642-23138-4](https://doi.org/10.1007/978-3-642-23138-4) find
  28. Tommi A. Pirinen. 2015. Omorfi — free and open source morphological lexical database for Finnish. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 313–315, Vilnius, Lithuania. Linköping University Electronic Press, Sweden. find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, cs.CL/1907.11692v1.
  30. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  31. Manuel Mager, Elisabeth Mager, Alfonso Medina-Urrea, Ivan Vladimir Meza Ruiz, and Katharina Kann. 2018. Lost in translation: Analysis of information loss during machine translation between polysynthetic and fusional languages. In Proceedings of the Workshop Benoı̂t Sagot. 2013. Comparing complexity… find
  32. Helmut Schmid, Arne Fitschen, and Ulrich Heid. 2004. SMOR: A German computational morphology covering derivation, composition and inflection. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), pages 1–263, Lisbon, Portugal. European Language Resourc…
  33. Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2016–2027, Vancouver, Canada. Association for Computational Linguistics. DOI:… find
  34. Lane Schwartz, Francis Tyers, Lori Levin, Christo Kirov, Patrick Littell, Chi-kiu Lo, Emily Prud’hommeaux, Hyunji Hayley Park, Kenneth Steimel, Rebecca Knowles, Jeffrey Micher, Lonny Strunk, Han Liu, Coleman Haley, Katherine J. Zhang, Robbie Jimerson, Vasilisa Andriyanets, Aldrian Obaja Muis, Naoki… find
  35. Hugo David Calderon Vilca, Flor Cagniy Cárdenas Mariñó, and Edwin Fredy Mamani Calderon. 2012. Analizador morfólogico de la lengua Quechua basado en software libre Helsinkifinite-statetransducer (HFST). find
  36. Sami Virpioja, Peter Smit, Stig-Arne Grönroos, and Mikko Kurimo. 2013. Morfessor 2.0: Python implementation and extensions for Morfessor baseline. Technical report, Aalto University; Aalto-yliopisto. find
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Roman Garnett, editors, Advances…
  39. Yusuke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching. Technical report, Department of Informatics, Kyushu University. find
  40. A Data We began with the data used in Mielke et al. (2019). This was originally a subset of a Bible corpus (Mayer and Cysouw, 2014), which is no longer publically available. We excluded constructed languages (epo, tlh) from the data, keeping a total of 104 verse-aligned Bibles in 60 languages 9 in 1… find
  41. Francis Tyers and Karina Mishchenkova. 2020. Dependency annotation of noun incorporation in polysynthetic languages. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 195–204, Barcelona, Spain (Online). Association for Computational Linguistics. 9 afr, aln, arb, arz,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2021-morphology-matters-multilingual-language-modeling/