Elamur Research⌕
☰
home / papers / 2026 / Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible

Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible

Paper · 2026 · Imry Ziv, Nur Lan and Emmanuel Chemla

Summary

The paper asks whether large language models, like human learners, find naturally occurring languages easier to acquire than "impossible" ones that break core linguistic constraints such as hierarchical, recursive structure. Kallini et al. (2024) had claimed that GPT-2 shows this preference. That claim was used to argue that human typology does not need strong innate biases. The authors reuse that ease-of-learning setup and widen it to more languages and more perturbation functions, mostly sentence-level shuffles and reversals. They train GPT-2 on each original dataset and on its perturbed variants, then compare validation perplexity curves. They examine the data from two angles. Intralinguistically, they compare each language with its own impossible versions. Interlinguistically, they test whether the full set of attested languages separates from the full set of impossible ones. In most cases GPT-2 learned the impossible variant as easily as the original, and sometimes more easily. Neither perspective produced a systematic boundary. This weakens the case for LLMs as evidence against the generative hypothesis and raises doubts about using them as models of human linguistic cognition.

Key points

  • The study replicates and extends the Kallini et al. (2024) perplexity-curve method. Model: GPT-2. Example perturbations: SHUFFLE global and SHUFFLE local, which scramble word order within sentences.

  • Two ease-of-learning metrics come from the learning curves: the minimum perplexity reached during training and the area under the perplexity curve (AUC).

  • In the interlinguistic test, the authors compare variance across languages under the same perturbation with variance within a language across its perturbations. Successful separation would predict higher within-language variance, but the results showed the reverse: each language grouped with its own impossible variants.

  • Setup differences from earlier work:

    • one whitespace tokenizer for all languages (vocabulary 50,257) rather than language-specific pretrained tokenizers;
    • no learning-rate warmup;
    • different training set sizes and hyperparameters.

  • Acknowledged limitations:

    • only GPT-2 was tested;
    • the sets of languages and perturbations are limited;
    • learning asymmetries alone cannot fully account for typology.

    Training ran on the Jean Zay supercomputer (GENCI/IDRIS).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tel Aviv UniversityTel Aviv · Israel82 works · 10,655 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🔬École Normale Supérieure - PSLParis · France23 works · 4,357 citations🎓Université Paris Sciences et LettresParis · France21 works · 3,290 citations🔬École des hautes études en sciences socialesParis · France15 works · 2,302 citations🔬Laboratoire de Sciences Cognitives et PsycholinguistiqueParis · France7 works · 907 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Imry Ziv, Nur Lan and Emmanuel Chemla. 2026. Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.249

@inproceedings{ziv2026biasless,
  title = {{Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible}},
  author = {Imry Ziv, Nur Lan and Emmanuel Chemla},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.249},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.249}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907–6919, Dublin, Ireland. Association f… find
  2. Catherine Arnett and Benjamin K. Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 6607– 6622, Mexico City, Mexico. Association for Computational Linguistics.
  3. Mark Baker. 2012. Formal generative typology. find
  4. Marco Baroni. 2022. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. In Shalom Lappin and Jürgen Bernardy, editors, Algebraic Structures in Natural Language, pages 1–16. CRC Press / Taylor Francis. Preprint available as arXiv:2106.08694. — [arXiv:2106.08694](https://arxiv.org/abs/2106.08694) find
  5. Thomas Brochhagen, Michael Franke, and Robert van Rooij. 2018. Coevolution of lexical meaning and pragmatic use. Cognitive Science, 42(8):2757– 2789. find
  6. Emmanuel Chemla and Ryan Nefdt. 2024. No such thing as a general learner: Language models and their dual optimization. Ms. find
  7. Noam Chomsky. 1957. Syntactic Structures. Mouton, The Hague. find
  8. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA. find
  9. Noam Chomsky. 1981. Lectures on Government and Binding. Foris Publications, Dordrecht, Netherlands. find
  10. Noam Chomsky. 2000. New Horizons in the Study of Language and Mind. Cambridge University Press, Cambridge; New York. Hardback edition, with foreword by Neil Smith. find
  11. Noam Chomsky, Ian Roberts, and Jeffrey Watumull. 2023. Noam chomsky: The false promise of chatgpt. The New York Times. Accessed: 2025-09-10. find
  12. Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, and Roger Levy. 2023. A cross-linguistic pressure for uniform information density in word order. Transactions of the Association for Computational Linguistics, 11:1048–1065.
  13. Jeremy Collins. 2016. Some language universals are historical accidents. In Karsten Schmidtke-Bode, Natalia Levshina, Susanne Maria Michaelis, and Ilja A. Serant, editors, Explanation in Typology: Diachronic Sources, Functional Motivations and the Nature of the Evidence, pages 51–65. Language Scienc… find
  14. Jennifer Culbertson and Elissa L. Newport. 2015. Harmonic biases in child learners: In support of language universals. Cognition, 139:71–82. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  16. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  17. Sandiway Fong and Robert C. Berwick. 2008. Treebank parsing and knowledge of language: A cognitive perspective. In Proceedings of the 30th Annual Conference of the Cognitive Science Society, page 539, Washington, DC. Cognitive Science Society. Paper No. 539. find
  18. Sandiway Fong, Igor Malioutov, Beracah Yankama, and Robert C. Berwick. 2013. Treebank parsing and knowledge of language. In Aline Villavicencio, Thierry Poibeau, Anna Korhonen, and Afra Alishahi, editors, Cognitive Aspects of Computational Language Acquisition, Theory and Applications of Natural Lan… find
  19. Danny Fox and Roni Katzir. 2024. Large language models and theoretical linguistics. Theoretical Linguistics, 50(1–2):71–76. find
  20. Richard Futrell and Michael Hahn. 2022. Information theory as a bridge between language function and language form. Frontiers in Communication, 7(657725). find
  21. Richard Futrell and Michael Hahn. 2025. Linguistic structure from a bottleneck on sequential information processing. Nature Human Behaviour. find
  22. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences. Accepted manuscript. find
  23. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL 2018, pages 1195–1205.
  24. Marc D. Hauser, Noam Chomsky, and W. Tecumseh Fitch. 2002. The faculty of language: What is it, who has it, and how did it evolve? Science, 298(5598):1569–1579. find
  25. Tim Hunter. 2025. Kallini et al. (2024) do not compare impossible languages with constituency-based ones. Computational Linguistics, 51(2):641–650. find
  26. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
  27. Simon Kirby, Mike Dowman, and Thomas L. Griffiths. 2007. Innateness and culture in the evolution of language. Proceedings of the National Academy of Sciences, 104(12):5241–5245. find
  28. Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024. Large language models and the argument from the poverty of the stimulus. Linguistic Inquiry, pages 1–56. find
  29. Evelina Leivada, Raquel Montero, Paolo Morosi, Natalia Moskvina, Tamara Serrano, Marcel Aguilar, and Fritz Guenther. 2025. Large language model probabilities cannot distinguish between possible and impossible language. arXiv preprint, arXiv:2509.15114. Submitted on 18 Sep 2025. — [arXiv:2509.15114](https://arxiv.org/abs/2509.15114) find
  30. Tal Linzen and Marco Baroni. 2020. Syntactic structure from deep learning. Annual Reviews of Linguistics. find
  31. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. find
  32. Daniel Milway. 2025. On modern language models, impossible languages and anti-science. Ling-Buzz preprint. https://ling.auf.net/lingbuzz/ 008314. ↗
  33. Jeff Mitchell and Jeffrey Bowers. 2020. Priorless recurrent networks learn curiously. In Proceedings of the 28th International Conference on Computational Linguistics. find
  34. Andrea Moro. 2016. Impossible Languages. Press, Cambridge, MA. find
  35. MIT Andrea Moro, Matteo Greco, and Stefano F. Cappa. 2023. Large languages, impossible languages and human brains. Cortex, 167:82–85. find
  36. Mariacristina Musso, Andrea Moro, Volkmar Glauche, Michel Rijntjes, Juergen Reichenbach, Christian Büchel, and Cornelius Weiller. 2003. Broca’s area and the language instinct. Nature Neuroscience, 6(7):774–781. find
  37. Partha Niyogi and Robert C. Berwick. 2009. The proper treatment of language acquisition and change in a population setting. Proceedings of the National Academy of Sciences, 106:10124–10129. find
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Accessed: 2024-11-15. find
  39. John R. Ross. 1967. Constraints on Variables in Syntax. Ph.D. thesis, MIT, Cambridge, MA. find
  40. H. Schmid. 1999. Improvements in Part-of-Speech Tagging with an Application to German, pages 13–25. Springer Netherlands, Dordrecht. find
  41. Avi Shmidman and Shaltiel Shmidman. 2025. Restoring missing spaces in scraped Hebrew social media. In Proceedings of the Tenth Workshop on Noisy and User-generated Text, pages 16–25, Albuquerque, New Mexico, USA. Association for Computational Linguistics.
  42. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 288…
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS 2017), volume 30, pages 5998–6008. Curran Associates, Inc.
  44. Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2023. Using computational models to test syntactic learnability. Linguistic Inquiry, pages 1–44. find
  45. Tianyang Xu, Tatsuki Kuribayashi, Yohei Oseki, Ryan Cotterell, and Alex Warstadt. 2025. Can language models learn typologically implausible languages? Preprint, arXiv:2502.12317.
  46. Xiulin Yang, Tatsuya Aoyama, Yuekun Yao, and Ethan Wilcox. 2025. Anything goes? a crosslinguistic study of (im)possible language learning in LMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26058–26077, Vienna, Austria. As…
  47. Victor H. Yngve. 1960. A model and an hypothesis for language structure. Proceedings of the American Philosophical Society, 104(5):444–466. find
  48. Imry Ziv, Nur Lan, Emmanuel Chemla, and Roni Katzir. 2025. Large Language Models as Proxies for Theories of Human Linguistic Cognition. Preprint, arXiv:2502.07687. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ziv-2026-biasless-language-models-learn-unnaturally/