Elamur Research⌕
☰
home / papers / 2025 / Confounding Factors in Relating Model Performance to Morphology

Confounding Factors in Relating Model Performance to Morphology

Paper · 2025 · Wessel Poelman, Thomas Bauwens and Miryam de Lhoneux

Summary

Studies disagree on whether morphologically rich languages are inherently harder for language models. The authors argue that much of this disagreement comes from uncontrolled variables in experimental design. They first list six confounds that an ideal study would keep fixed: the language set, morphological grouping, tokenization algorithm, the ratio of vocabulary size to data size, corpus domain, and the performance metric. They then revisit three explanations from Arnett and Bergen (2025) for why agglutinative languages get higher perplexity than fusional ones: poor morphological alignment of subword tokenizers, inefficient vocabulary use, and smaller training sets. They find confounds behind each conclusion. MorphScore checks only recall of a single stem-suffix boundary. Unigram tokenizer statistics explain little. Blaming data size ignores that writing systems themselves encode morphology. Finally, the paper proposes token bigram measures, notably accessor variety and entropic efficiency, computed on EuroParl and FineWeb with monolingual ULM tokenizers. These measures place languages on a gradient that follows morphological complexity and need no expert annotation. They also suggest a new hypothesis for why agglutination may be harder for causal LMs.

Key points

  • Six confounding factors are proposed as criteria for a sound experiment: languages, grouping, tokenization algorithm, vocabulary vs. data size, corpus domain, and performance indicator. The authors specifically caution against comparing monolingual PPL across different tokenizers and test texts.
  • The authors contrast MorphScore with full-alignment precision/recall/F1 over all morpheme boundaries, including suffix-suffix boundaries. They report the comparison for 13 languages, among them German, English, Finnish, Hungarian and Turkish.
  • The new metrics are accessor variety, total accessors, accessor uniqueness, and entropic efficiency (η). They are computed over neighbouring subwords in fixed-size windows, analogous to MATTR.
  • Data: EuroParl (21 languages, multi-parallel) and FineWeb (63 languages), with 19 languages shared, 65 in total, and about 200k lines per corpus. FLORES-200 proved too small for stable estimates.
  • Domain affects the metrics: EuroParl yields lower AV and η but higher MATTR than FineWeb. Agglutinative languages rank highest and isolating ones lowest, while the middle of the ranking shows that coarse typological groupings hide information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Morphology · 405 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Type–token ratio · 14 works

Cite

Cite as Wessel Poelman, Thomas Bauwens and Miryam de Lhoneux. 2025. Confounding Factors in Relating Model Performance to Morphology. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.369

@inproceedings{poelman2025confounding,
  title = {{Confounding Factors in Relating Model Performance to Morphology}},
  author = {Wessel Poelman, Thomas Bauwens and Miryam {de Lhoneux}},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.369},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.369}
}

References

60 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Farrell Ackerman and Robert Malouf. 2013. Morphological Organization: The Low Conditional Entropy Conjecture. Language, 89(3):429–464. find
  2. Chantal Amrhein and Rico Sennrich. 2021. How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 689–705. Association for Computational Linguistics. find
  3. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623. Association for Computational Linguistics.
  4. Catherine Arnett, Tyler A. Chang, and Benjamin Bergen. 2024. A Bit of a Problem: Measurement Disparities in Dataset Sizes across Languages. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 1–9. ELRA and ICCL. find
  5. Duygu Ataman, Matteo Negri, Marco Turchi, and Marcello Federico. 2017. Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to English. The Prague Bulletin of Mathematical Linguistics, 108(1):331–342. find
  6. Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia. 2021. MorphyNet: A Large Multilingual Database of Derivational and Inflectional Morphology. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 39–48. Association for Compu…
  7. Thomas Bauwens. 2024. Bits-per-character and its relation to perplexity. Blog post. find
  8. Thomas Bauwens and Pieter Delobelle. 2024. BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… find
  9. Christian Bentz, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardžić. 2016. A Comparison Between Morphological Complexity Measures: Typological Data vs. Language Corpora. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 142–153. find
  10. The COL-ING 2016 Organizing Committee. find
  11. Balthasar Bickel and Johanna Nichols. 2013. Fusion of selected inflectional formatives (v2020.4). In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Zenodo. find
  12. Kaj Bostrom and Greg Durrett. 2020. Byte Pair Encoding is Suboptimal for Language Model Pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624. Association for Computational Linguistics.
  13. Olivier J. Bousquet, Amit Daniely, Haim Kaplan, Yishay Mansour, Shay Moran, and Uri Stemmer. 2022. Monotone Learning. In Proceedings of Thirty Fifth Conference on Learning Theory, pages 842–866. PMLR. ISSN: 2640-3498. find
  14. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024a. When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096. Association for Computational…
  15. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024b. Goldfish: Monolingual Language Models for 350 Languages. ArXiv:2408.10441 [cs]. find
  16. Çağrı Çöltekin and Taraka Rama. 2023. What do complexity measures measure? Correlating and validating corpus-based measures of morphological complexity. Linguistics Vanguard, 9(s1):27–43. find
  17. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are All Languages Equally Hard to Language-Model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
  18. Michael A. Covington and Joe D. McFall. 2010. Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. find
  19. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 204–213, Dublin, Ireland. European Association for Machine Translation. find
  20. Haodi Feng, Kang Chen, Xiaotie Deng, and Weimin Zheng. 2004. Accessor Variety Criteria for Chinese Word Extraction. Computational Linguistics, 30(1):75–93.
  21. Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018a. Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction. Transactions of the Association for Computational Linguistics, 6:451–465.
  22. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018b. On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327. Associati…
  23. Kyle Gorman and Richard Sproat. 2023. Myths about Writing Systems in Speech & Language Technology. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 1–5. Association for Computational Linguistics. find
  24. Stig-Arne Grönroos, Sami Virpioja, Peter Smit, and Mikko Kurimo. 2014. Morfessor FlatCat: An HMM-Based Method for Unsupervised and Semi-Supervised Learning of Morphology. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1177–1185,… find
  25. Zellig S. Harris. 1955. From Phoneme to Morpheme. Language, 31(2):190–222. Publisher: Linguistic Society of America. find
  26. Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2):65–70. find
  27. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. ArXiv:2001.08361 [cs].
  28. Kimmo Kettunen. 2014. Can Type-Token Ratio be Used to Show Morphological Complexity of Languages? Journal of Quantitative Linguistics, 21(3):223–245. find
  29. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86. find
  30. Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75. Association for Computational Linguistics.
  31. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. Association for Computati…
  32. Mikko Kurimo, Mathias Creutz, Matti Varjokallio, Ebru Arisoy, and Murat Saraclar. 2006. Unsupervised segmentation of words into morphemes – Challenge 2005 An Introduction and Evaluation Report. In Proceedings of the PASCAL Challenge Workshop on Unsupervised segmentation of words into morphemes, page… find
  33. Mikko Kurimo, Sami Virpioja, Ville Turunen, and Krista Lagus. 2010. Morpho Challenge 2005-2010: Evaluations and Results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pages 87–95. Association for Computational Linguistics. find
  34. Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. 2024. MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  35. Paul McCann. 2020. Fugashi, a Tool for Tokenizing Japanese in Python. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 44–51. Association for Computational Linguistics. find
  36. Clara Meister and Ryan Cotterell. 2021. Language Model Evaluation Beyond Perplexity. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5328–5339. Associa… find
  37. Sabrina J. Mielke. 2019. Can you compare perplexity across different segmentations? Blog post. find
  38. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What Kind of Language Is Hard to Language-Model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989. Association for Computational Linguistics.
  39. Benjamin Minixhofer, Jonas Pfeiffer, and Ivan Vulić. 2023. Where‘s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation. find
  40. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7215–7235, Toronto, Canada. Association for Computational Linguistics. find
  41. Arturo Oncevay, Duygu Ataman, Niels Van Berkel, Barry Haddow, Alexandra Birch, and Johannes Bjerva. 2022. Quantifying Synthesis and Fusion and their Impact on Machine Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu… find
  42. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology Matters: A Multilingual Language Modeling Analysis. Transactions of the Association for Computational Linguistics, 9:261–276.
  43. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
  44. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. FineWeb2: One Pipeline to Scale Them All – Adapting Pre-Training Data Processing to Every Language. ArXiv:2506.20920 [cs].
  45. Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, and Can Udomcharoenchaikit. 2023. PyThaiNLP: Thai Natural Language Processing in Python. In Proceedings of the 3rd Works… find
  46. Varshini Reddy, Craig W. Schmidt, Yuval Pinter, and Chris Tanner. 2025. How Much is Enough? The Diminishing Returns of Tokenization Training Data. ArXiv:2502.20273 [cs]. — [arXiv:2502.20273](https://arxiv.org/abs/2502.20273) find
  47. Peter J. Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. WIREs Data Mining and Knowledge Discovery, 1(1):73–79. find
  48. Jessica Rumbelow and Matthew Watkins. 2023. Solid-GoldMagikarp (plus, prompt generation). Blog post. find
  49. Geoffrey Sampson, David Gil, and Peter Trudgill, editors. 2009. Language Complexity as an Evolving Variable. Oxford Studies in the Evolution of Language. Oxford University Press. find
  50. Edward Sapir. 1921. Language: An Introduction to the Study of Speech. Harcourt, Brace. find
  51. Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Compu…
  52. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the Noiseless Channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207. Association for Computational Lin… find
  53. George A. F. Seber and Alan J. Lee. 2003. Straight-Line Regression. In Linear Regression Analysis, pages 139–163. John Wiley & Sons, Ltd. find
  54. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics.
  55. Team NLLB, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  56. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC‘12), pages 2214–2218. European Language Resources Association (ELRA). find
  57. Ada Wan. 2022. Fairness in Representation for Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling. In International Conference on Learning Representations. find
  58. B. L. Welch. 1947. The Generalization of ‘Student’s’ Problem when Several Different Population Variances are Involved. Biometrika, 34(1/2):28–35. Publisher: [Oxford University Press, Biometrika Trust]. find
  59. Yingting Wu and Hai Zhao. 2018. Finding Better Subword Segmentation for Neural Machine Translation. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, Lecture Notes in Computer Science, pages 53–64, Cham. Springer International Publishing. find
  60. George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley Press, Cambridge, Massachusetts. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poelman-2025-confounding-factors-relating-model-performance/