Elamur Research⌕
☰
home / papers / 2025 / Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs

Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs

Paper · 2025 · Xiulin Yang, Tatsuya Aoyama, Yuekun Yao et al.

Summary

A common objection to using language models as models of human acquisition is that they supposedly learn any input pattern equally well, natural or not. This study tests that objection beyond English, the only language covered by earlier work (Kallini et al., 2024). The authors built two sentence-aligned parallel corpora from OPUS sources, OPUS12 and OPUS30, so that content is held constant across languages. They trained GPT-2 small on 12 attested languages from 4 families, on "impossible" versions of each (shuffled, reversed or otherwise deterministically reordered tokens), and on noun-phrase word orders that Greenberg's Universal 20 marks as typologically unattested.

Within each language, GPT-2 generally reaches lower perplexity on the real language than on its impossible variants, though some locally shuffled variants come close. Across languages, however, the two classes do not separate cleanly. For unattested NP orders, perplexity alone shows no preference for attested orders, but a targeted generalization test does. The authors conclude that LMs carry soft, human-like inductive biases that are weaker than those of human learners. This refutes the "anything goes" view while still showing limits.

Key points

  • Corpora:
    • OPUS12 covers 12 languages, with about 1M sentences and roughly 10.1M English words. That word count is comparable to a child's input over the first 2–5 years.
    • OPUS30 covers 30 languages, with 48K sentences and about 0.7M words, and serves as an extra test set.
    • Both corpora draw on NLLB, TED2020, the Bible, OpenSubtitles and CCAligned.
  • Setup:
    • GPT-2 small was trained per language with 3 seeds and 1200 training steps.
    • Models were scored by geometric mean perplexity on a parallel 10K-sentence test split.
    • Each language got its own tokenizer, rather than a shared multilingual vocabulary, to avoid favoring Latin-script languages.
  • Replication check: A scaled-down (10M-word) rerun of Kallini et al. (2024) reproduced their ranking exactly (Spearman's ρ = 1).
  • Separability: Perplexity separated attested from impossible languages with a score of 0.75. This is above chance but not perfect.
  • Confound: Normalizing NP order seems to lower entropy, which may explain why some unattested NP orders get lower perplexity. Code and data are released at github.com/picol-georgetown/multilingual-LM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🎓Georgetown UniversityWashington · United States88 works · 4,732 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Xiulin Yang, Tatsuya Aoyama, Yuekun Yao and Ethan Gotlieb Wilcox. 2025. Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1264

@inproceedings{yang2025anything,
  title = {{Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs}},
  author = {Xiulin Yang, Tatsuya Aoyama, Yuekun Yao and Ethan Gotlieb Wilcox},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1264},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1264}
}

References

86 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  2. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics.
  3. Florbela Barreto, António Branco, Eduardo Ferreira, Amália Mendes, Maria Fernanda Bacelar do Nascimento, Filipe Nunes, and João Ricardo Silva. 2006. Open resources and tools for the shallow processing of Portuguese: The TagShare project. In Proceedings of the Fifth International Conference on Langua… find
  4. Johan J Bolhuis, Stephen Crain, Sandiway Fong, and Andrea Moro. 2024. Three reasons why AI doesn’t model human language. Nature, 627(8004):489. find
  5. Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, and Ryan Cotterell. 2024. What languages are easy to language-model? a perspective from learning probabilistic regular languages. In Proceedings of the 62nd Annual Meeting of the Association… find
  6. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. Goldfish: Monolingual language models for 350 languages. Preprint, arXiv:2408.10441. find
  7. Noam Chomsky and Andrea Moro. 2022. The secrets of words. MIT Press. find
  8. Noam Chomsky, Lan Roberts, and Jeffrey Watumull. 2023. Noam Chomsky: The false promise of Chat-GPT. The New York Times. Accessed: 2024-12-16. find
  9. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the Bible in 100 languages. Language resources and evaluation, 49:375–395.
  10. Guglielmo Cinque. 2005. Deriving Greenberg’s Universal 20 and its exceptions. Linguistic inquiry, 36(3):315–332. find
  11. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages…
  12. Jennifer Culbertson, Julie Franck, Guillaume Braquet, Magda Barrera Navarro, and Inbal Arnon. 2020. A learning bias for word order harmony: Evidence from speakers of non-harmonic languages. Cognition, 204:104392. find
  13. Jennifer Culbertson and Elissa L Newport. 2015. Harmonic biases in child learners: In support of language universals. Cognition, 139:71–82. find
  14. Jennifer Culbertson and Elissa L Newport. 2017. Innovation of word order harmony across development. Open Mind, 1(2):91–100. find
  15. Jennifer Culbertson, Paul Smolensky, and Géraldine Legendre. 2012. Learning biases predict a word order universal. Cognition, 122(3):306–329. find
  16. Christine Cuskley, Rebecca Woods, and Molly Flaherty. 2024. The limitations of large language models for understanding human language and cognition. Open Mind, 8:1058–1083. find
  17. Forrest Davis and Marten van Schijndel. 2020. Recurrent neural network language models always learn English-like relative clause attachment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1979–1990, Online. Association for Computational Linguistics. find
  18. Iria de Dios-Flores, Juan Garcia Amboage, and Marcos Garcia. 2023. Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  19. Rodolfo Delmonte, Antonella Bristot, and Sara Tonelli. 2007. VIT-Venice Italian Treebank: Syntactic and quantitative features. In Sixth International Workshop on Treebanks and Linguistic Theories, volume 1, pages 43–54. Northern European Association for Language Technol. find
  20. Vittoria Dentella, Fritz Guenther, and Evelina Leivada. 2024. Language in vivo vs. in silico: Size matters but larger language models still do not comprehend language on a par with humans. arXiv preprint arXiv:2404.14883. — [arXiv:2404.14883](https://arxiv.org/abs/2404.14883) find
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  22. Matthew S Dryer. 1992. The Greenbergian word order correlations. Language, 68(1):81–138. find
  23. Stefan Dumitrescu. 2024. GPT-Neo Romanian 780M. Hugging Face. Available at: https://huggingface.co/dumitrescustefan/ gpt-neo-romanian-780m. ↗
  24. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 5960–5969, Online. Association for Computa…
  25. Danny Fox and Roni Katzir. 2024. Large language models and theoretical linguistics. Theoretical Linguistics, 50(1-2):71–76. find
  26. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. arXiv preprint arXiv:2501.17047. — [arXiv:2501.17047](https://arxiv.org/abs/2501.17047) find
  27. Dedre Gentner and Melissa Bowerman. 2009. Why some spatial semantic categories are harder to learn than others: The typological prevalence hypothesis. Crosslinguistic approaches to the psychology of language: Research in the tradition of Dan Isaac Slobin, 465:480. find
  28. Edward Gibson, Richard Futrell, Steven P Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in cognitive sciences, 23(5):389– 407. find
  29. David Gil. 2008. How complex are isolating languages. Language. find
  30. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2274–2286, Bangkok, Thailand. Assoc… find
  31. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput…
  32. Joseph H Greenberg et al. 1963. Some universals of grammar with particular reference to the order of meaningful elements. Universals of language, 2:73– 113. find
  33. Yeb Havinga. 2023. GPT Neo 1.3B pre-trained on cleaned Dutch mC4. Accessed: 2024-12-10. iGeniusAI. 2024. Italia-9b-instruct-v0.1. https://huggingface.co/iGeniusAI/ Italia-9B-Instruct-v0.1. Accessed: 2024-06-17. ↗
  34. Tamar Johnson, Kexin Gao, Kenny Smith, Hugh Rabagliati, and Jennifer Culbertson. 2021. Investigating the effects of i-complexity and e-complexity on the learnability of morphological systems. J. Lang. Model., 9. find
  35. John E Joseph and Frederick J Newmeyer. 2012. ‘All languages are equally complex’. Historiographia linguistica, 39. find
  36. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  37. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
  38. Roni Katzir. 2023. Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi. Biolinguistics, 17:1–12. find
  39. H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Ege-men Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, and M. Fatih Amasyali. 2024. Introducing cosmosGPT: Monolingual training for Turkish language models. In 2024 International Conference on INnovations in Intelligent SysTems and Applications (INISTA),… find
  40. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics. find
  41. Christo Kirov and Ryan Cotterell. 2018. Recurrent neural networks in linguistic theory: Revisiting Pinker and Prince (1988) and the past tense debate. Transactions of the Association for Computational Linguistics, 6:651–665.
  42. Alexander Koplenig and Sascha Wolfer. 2023. Languages with more speakers tend to be harder to (machine-) learn. Scientific Reports, 13(1):18521. find
  43. Alexander Koplenig, Sascha Wolfer, and Peter Meyer. 2023. A large quantitative analysis of written language challenges the idea that all languages are equally complex. Scientific Reports, 13(1):15351. find
  44. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. Psychometric predictive power of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1983–2005, Mexico City, Mexico. Association for Computational Linguistics. find
  45. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large language models are human-like internally. arXiv preprint arXiv:2502.01615.
  46. Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024. Large language models and the argument from the poverty of the stimulus. Linguistic Inquiry, pages 1–56. find
  47. Julien Launay, E.l. Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djamé Seddah. 2022. PAGnol: An extra-large French generative model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4275–4284, Marseille, F… find
  48. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p…
  49. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (EL…
  50. Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlórIA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Comp… find
  51. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.
  52. John H McWhorter. 2001. The worlds simplest grammars are creole grammars. Linguistic Typology, 5(2-3):125–166. find
  53. John H McWhorter. 2011. Linguistic simplicity and complexity: Why do languages undress?, volume 1. Walter de Gruyter. find
  54. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963– 980, Online and Punta Cana, Dominican Republic…
  55. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic…
  56. Raphaël Millière. 2024. Language models as models of language. arXiv preprint arXiv:2408.07144.
  57. Jeff Mitchell and Jeffrey Bowers. 2020. Priorless recurrent networks learn curiously. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5147–5158, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  58. Andrea Moro. 2023. Embodied syntax: impossible languages and the irreducible difference between humans and machines. Sistemi intelligenti, 35(2):321– 328. find
  59. Andrea Moro, Matteo Greco, and Stefano F Cappa. 2023. Large languages, impossible languages and human brains. Cortex, 167:82–85. find
  60. Frederick J Newmeyer. 2005. Possible and probable languages. Oxford: Oxford. find
  61. Frederick J Newmeyer. 2021. Complexity and relative complexity in generative grammar. Frontiers in Communication, 6:614352. find
  62. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… find
  63. Byung-Doh Oh and William Schuler. 2023b. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350.
  64. Malte Ostendorff. 2023. Gpt2-xl-wechsel-german. https://huggingface.co/malteos/ gpt2-xl-wechsel-german. ↗
  65. Joe Pater. 2019. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95(1):e41–e74. find
  66. Lisa Pearl and Benjamin Mis. 2011. How far can indirect evidence take us? Anaphoric one revisited. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33. find
  67. Steven T Piantadosi. 2023. Modern language models refute Chomsky’s approach to language. From fieldwork to linguistic theory: A tribute to Dan Everett, pages 353–414. find
  68. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.…
  69. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  70. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
  71. Jenny Saffran, Marc Hauser, Rebecca Seibel, Joshua Kapfhamer, Fritz Tsao, and Fiery Cushman. 2008. Grammatical pattern learning by human infants and cotton-top tamarin monkeys. Cognition, 107(2):479– 500. find
  72. Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Compu…
  73. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co…
  74. Jenny L Singleton and Elissa L Newport. 2004. When learners surpass their models: The acquisition of American sign language from inconsistent input. Cognitive psychology, 49(4):370–407. find
  75. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  76. Ianthi-Maria Tsimpli and Neil Smith. 1995. Mind of a Savant: Language, Learning and Modularity. Blackwell. find
  77. Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, and Alex Warstadt. 2024. Surprise! Uniform Information Density isn’t the whole story: Predicting surprisal contours in long-form discourse. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language… find
  78. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… find
  79. Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.
  80. Michał Wojczulis and Dariusz Kłeczek. 2021. papu-GaPT2 - Polish GPT2 language model. find
  81. Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. 2025. Can language models learn typologically implausible languages? arXiv preprint arXiv:2502.12317.
  82. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207–238. find
  83. Yuekun Yao and Alexander Koller. 2022. Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  84. Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. A family of pretrained transformer language models for R…
  85. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for… find
  86. A Experiment Results of Replicating Kallini et al. (2024) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2025-anything-goes-crosslinguistic-study-im/