Elamur Research⌕
☰
home / papers / 2026 / Language Acquisition Device in Large Language Models

Language Acquisition Device in Large Language Models

Paper · 2026 · Masato Mita, Taiga Someya, Ryo Yoshida et al.

Summary

LLMs need far more training data than children to learn language. This paper asks whether a short warm-up phase on synthetic data, known as pre-pretraining (PPT), works better when the data mimics structural properties of natural language rather than abstract formal complexity. The idea draws on the Language Acquisition Device hypothesis. The authors build MP-STRUCT, a generator based on the Minimalist Program. It produces lexically empty, bracketed tree serializations that encode three operations:

  • MERGE, for hierarchical composition
  • AGREE, for subject–verb number features
  • MOVE, for subject and wh-displacement with traces

They run 500 PPT steps on Pythia-1B before normal pretraining. The gains in token efficiency match strong baselines such as k-Shuffle Dyck. The models also become more robust to lexical perturbation and resist structurally implausible languages, which is a human-like trait.

Experiments with simplified variants show that MP-STRUCT CORE beats k-Shuffle Dyck even though it cannot be defined in C-RASP. This challenges the earlier expressivity hypothesis. The authors attribute the gain to "functional landmarks", explicit tokens that make dependencies easier to locate. The broader lesson is that how structure is organized matters for PPT, not only how expressive the language is.

Key points

  • Setup: Pythia-1B with a brief 500-step PPT phase. Baselines are Random sequences, 1-Dyck and k-Shuffle Dyck, all tokenized with the Pythia-1B tokenizer. Grammatical generalization is measured with BLiMP, which covers English only.
  • Core comparison: k-Shuffle Dyck scores MRS 15.6 and 29% efficiency. The analysis variant MP-STRUCT CORE reaches up to 37% efficiency.
  • C-RASP result: MP-STRUCT CORE requires functional head tokens to sit directly before their dependency brackets. C-RASP cannot express this kind of two-position constraint, yet the variant still performs better. The authors therefore argue that whether a language can be defined in C-RASP neither guarantees nor is required for PPT to be effective.
  • Limitations: Only roughly 1B-scale transformers were tested, and evaluation is English-only. "Dependency identification ambiguity" has no formal definition. Landmark tokens also enlarge the vocabulary, which may confound the comparison.
  • Code: github.com/osekilab/LAD-PPT

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Masato Mita, Taiga Someya, Ryo Yoshida and Yohei Oseki. 2026. Language Acquisition Device in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.895

@inproceedings{mita2026language,
  title = {{Language Acquisition Device in Large Language Models}},
  author = {Masato Mita, Taiga Someya, Ryo Yoshida and Yohei Oseki},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.895},
  url = {https://doi.org/10.18653/v1/2026.acl-long.895}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Stickland, Tomek Korbak, and Owain Evans. 2024. The reversal curse: Llms trained on "a is b" fail to learn "b is a". In International Conference on Representation Learning, volume 2024, pages 18623–18642. find
  2. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… find
  3. Lewis Carroll. 1871. Through the Looking-Glass, and What Alice Found There. Macmillan. find
  4. N. Chomsky. 1995. The Minimalist Program. Current studies in linguistics series. MIT Press. find
  5. Noam Chomsky. 1959. On Certain Formal Properties of Grammars. Information and Control, 2(2):137–167. find
  6. Noam Chomsky. 1965. Aspects of the Theory of Syntax. The MIT Press, Cambridge. find
  7. Noam Chomsky. 2000. Minimalist inquiries: The framework. In Roger Martin, David Michaels, and Juan Uriagereka, editors, Step by Step: Essays on Minimalist Syntax in Honor of Howard Lasnik, pages 89–155. MIT Press, Cambridge, MA. find
  8. Noam Chomsky. 2001. Derivation by phase. In Ken Hale: A Life in Language. The MIT Press. find
  9. Alexander Clark and Shalom Lappin. 2011. Linguistic Nativism and the Poverty of the Stimulus. Wiley-Blackwell. find
  10. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: limits of transformers on composition…
  11. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  12. Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. Between circuits and Chomsky: Pre-pretraining on formal languages imparts linguistic biases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9691–…
  13. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
  14. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  15. Isabel Papadimitriou and Dan Jurafsky. 2020. Learning Music Helps You Read: Using transfer to study linguistic structure in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6829–6839, Online. Association for Computational Lingu… find
  16. Isabel Papadimitriou and Dan Jurafsky. 2023. Injecting structural hints: Using language models to study inductive biases in language learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8402–8413, Singapore. Association for Computational Linguistics. find
  17. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21:140:1–140:67.
  18. Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowledge in language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7302– 7315, Dublin, Ireland. Association for Com… find
  19. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… find
  20. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  21. Andy Yang and David Chiang. 2024. Counting like transformers: Compiling temporal counting logic into softmax transformers. In First Conference on Language Modeling. find
  22. Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. 2020. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mita-2026-language-acquisition-device-large-language/