Elamur Research⌕
☰
home / papers / 2025 / Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models

Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models

Summary

The paper tests whether child-directed language (CDL) is truly better material for training language models than text written for adults. This idea was supported by the work of Huebner et al. (2021) on BabyBERTa, but it had been tested only on English and in a single setting. Researchers from the University of Groningen carry out a broader test. They train RoBERTa-based (masked LM) and GPT-2-based (causal LM) models from scratch on CHILDES transcripts. For comparison, the same models are trained on Wikipedia samples of the same size. The languages are English, French and German. The models' syntactic knowledge is evaluated on minimal-pair datasets. The authors also propose FIT-CLAMS, a variant of the CLAMS benchmark in which subjects and verbs are selected with their frequency in both corpora taken into account. The results are complemented by a regression analysis. The main conclusion: CDL does not provide a consistent advantage, and in most cases the Wikipedia-trained models perform better. Without controlling for word frequency, corpus comparisons can be misleading. Therefore, the widespread notion that child-directed speech is especially beneficial for LM training should be reconsidered.

Key points

  • Data: only adult utterances addressed to children were taken from CHILDES. For English, the AO-CHILDES split from Huebner et al. was used. Corpus sizes: ≈4.3M tokens (EN), 2.3M (FR) and 3.8M (DE). The Wikipedia corpora are of the same size. Sentences in CHILDES are shorter: about 5.6–6.5 words versus 22–39 in Wikipedia. The lower type/token ratio shows that child-directed speech is highly repetitive.
  • Models: both architectures have the same configuration: 8 layers, 8 attention heads, embedding 512, FFN 2048. A separate BPE tokenizer with a vocabulary of 8,192 was trained for each corpus. Training runs for 100,000 steps with AdamW, a warm-up of 40,000 steps and a learning rate of 1e-4. Results are averaged over three random seeds.
  • Evaluation: the models are tested on three existing minimal-pair benchmarks, including CLAMS. To these is added the new FIT-CLAMS, balanced for the frequency of subjects and verbs in the training data.
  • Limitations acknowledged by the authors: CHILDES is spoken language, while Wikipedia is written, so corpus type is confounded with modality. CDL contains grammatical inconsistencies. The regression covers only simple subject–verb agreement. FIT-CLAMS has no more than 10 lexemes per category (only 7 for French verbs), and these lexemes were selected manually.
  • Resource: the code and data are openly published at github.com/fpadovani/childes_vs_wiki.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Language models & pretraining Syntax & grammar formalisms Psycholinguistics & language acquisition

Cite

Cite as Francesca Padovani, Jaap Jumelet, Yevgen Matusevych and Arianna Bisazza. 2025. Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.999 · arXiv 2505.23689

@inproceedings{padovani2025child,
  title = {{Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models}},
  author = {Francesca Padovani, Jaap Jumelet, Yevgen Matusevych and Arianna Bisazza},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.999},
  eprint = {2505.23689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.999}
}

References

57 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Katrien Beuls and Paul Van Eecke. 2024. Humans learn language from situated communicative interactions. what about machines? Computational Linguistics, 50(3):1277–1311. find
  2. Andrew Biemiller. 2003. Vocabulary: Needed if more children are to read well. Reading psychology, 24(3-4):323–335. find
  3. Kathryn Bock and Carol A. Miller. 1991. Broken agreement. Cognitive Psychology, 23(1):45–93. find
  4. Bastian Bunzeck, Daniel Duran, and Sina Zarrieß. 2025. Do construction distributions shape formal language learning in German BabyLMs? In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 169–186, Vienna, Austria. Association for Computational Linguistics. find
  5. Bastian Bunzeck and Sina Zarrieß. 2023. GPT-wee: How small can a small language model really get? find
  6. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 35–46, Singapore. Association for Computational Linguistics. find
  7. Bastian Bunzeck and Sina Zarrieß. 2025. Subword models struggle with word learning, but surprisal hides it. arXiv preprint arXiv:2502.12835. find
  8. Ziling Cheng, Rahul Aralikatte, Ian Porada, Cesare Spinoso-Di Piano, and Jackie CK Cheung. 2023. Mcgill babylm shared task submission: The effects of data formatting and structural biases. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages… find
  9. Steven Y. Feng, Noah D. Goodman, and Michael C. Frank. 2024. Is child-directed speech effective training data for language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22055–22071, Miami, Florida, USA. Association for Computational Linguist… our card
  10. Charles A. Ferguson. 1964. Baby talk in six languages. American Anthropologist, 66(6_PART2):103–114. find
  11. Lieke Gelderloos, Grzegorz Chrupała, and Afra Alishahi. 2020. Learning to understand child-directed and adult-directed speech. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1–6, Online. Association for Computational Linguistics. find
  12. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. 2022. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369–380. find
  13. Akari Haga, Akiyo Fukatsu, Miyu Oba, Arianna Bisazza, and Yohei Oseki. 2024. BabyLM challenge: Exploring the effect of variation sets on language model training efficiency. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 252–261, Miami, FL, USA. A… find
  14. Matthew Honnibal. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  15. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics. our card
  16. Philip A Huebner and Jon A Willits. 2021. Using lexical context to discover the noun category: Younger children have it easier. In Psychology of learning and motivation, volume 75, pages 279–331. Elsevier. find
  17. Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics. find
  18. Vera Kempe, Mitsuhiko Ota, and Sonja Schaeffler. 2024. Does child-directed speech facilitate language development in all domains? a study space analysis of the existing evidence. Developmental Review, 72:101121. find
  19. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… our card
  20. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. our card
  21. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  22. Brian MacWhinney and Lawrence Erlbaum. 2000. The CHILDES project . Volume I: Tools for analyzing talk: Transcription format and programs. Mahwah, NJ: Lawrence Erlbaum. find
  23. Richard Diehl Martinez, Zebulon Goriely, Hope Mc-Govern, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. Climb: Curriculum learning for infant-inspired model building. arXiv preprint arXiv:2311.08886. find
  24. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  25. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. our card
  26. Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. CoRR, abs/2203.13112. find
  27. Aaron Mueller and Tal Linzen. 2023. How to plant trees in language models: Data and architectural effects on the emergence of syntactic inductive biases. In Annual Meeting of the Association for Computational Linguistics. find
  28. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput… find
  29. Mitja Nikolaus, Abhishek Agrawal, Petros Kaklamanis, Alex Warstadt, and Abdellzhah Fourtassi. 2024. Automatic annotation of grammaticality in child-caregiver conversations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC… our card
  30. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472, Miami, Florida, USA. Association for Comp… find
  31. Ludovica Pannitto and Aurelie Herbelot. 2022. Can recurrent neural networks validate usage-based theories of grammar acquisition? Frontiers in Psychology, Volume 13 - 2022. find
  32. Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy La-pastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkeld. 2024. Filtered corpus training (FiCT) shows that language models can generalize from indirect evidence. Transactions of the Association for Computational Linguistics, 12… find
  33. Colin Phillips, Matthew W. Wagers, and Ellen F. Lau. 2011. 5: Grammatical Illusions and Selective Fallibility in Real-Time Language Comprehension, pages 147 – 180. Brill, Leiden, The Netherlands. find
  34. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. find
  35. Eva Portelance and Masoud Jasbi. 2024. The roles of neural networks in language acquisition. Language and Linguistics Compass, 18(6):e70001. find
  36. Yulu Qin, Wentao Wang, and Brenden M Lake. 2024. A systematic investigation of learnability from single child linguistic input. arXiv preprint arXiv:2402.07899. find
  37. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Accessed: 2024-11-15. find
  38. Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, and Paula Buttery. 2024. Less is more: Pre-training cross-lingual small-scale language models with cognitively-plausible curriculum learning strategies. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learni… find
  39. Alessandro Sanchez, Stephan C Meylan, Mika Braginsky, Kyle E MacDonald, Daniel Yurovsky, and Michael C Frank. 2019. childes-db: A flexible and reproducible interface to the child language data exchange system. Behavior research methods, 51:1928– 1941. find
  40. Johanna Schick, Caroline Fryns, Franziska Wegdell, Marion Laporte, Klaus Zuberbühler, Carel van Schaik, Simon Townsend, and Sabine Stoll. 2022. The function and evolution of child-directed communication. PLoS Biology, 20(5):e3001630. find
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… our card
  42. Lennart Stöpler, Rufat Asadli, Mitja Nikolaus, Ryan Cotterell, and Alex Warstadt. 2025. Towards developmentally plausible rewards: Communicative success as a learning signal for interactive language models. arXiv preprint arXiv:2505.05970. find
  43. Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L. Frank. 2025. Blimp-nl: A corpus of Dutch minimal pairs and acceptability judgments for language model evaluation. Computational Linguistics, pages 1–35. find
  44. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. our card
  45. Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. find
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur… our card
  47. Héctor Javier Vázquez Martínez, Annika Heuser, Charles Yang, and Jordan Kodner. 2023. Evaluating neural language models as cognitive models of language acquisition. In Proceedings of the 1st Gen-Bench Workshop on (Benchmarking) Generalisation in NLP, pages 48–64, Singapore. Association for Computati… find
  48. Alex Warstadt and Samuel R. Bowman. 2022. What artificial neural networks can tell us about human language acquisition. In Algebraic Structures in Natural Language, pages 17–60. CRC Press. our card
  49. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023. Proceedings of the babylm challenge at the 27th conference on computational natural language learning. In Proceedings of the BabyLM Ch… find
  50. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. find
  51. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computation… find
  52. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati… our card
  53. Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. 2023. Towards more human-like language models based on contextualizer pretraining strategy. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 317–326, Singapore. Association for Compu… our card
  54. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. our card
  55. Aditya Yedetore, Tal Linzen, Robert Frank, and R. Thomas McCoy. 2023. How poor is the stimulus? evaluating hierarchical generalization in neural networks trained on child-directed speech. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… our card
  56. Guanghao You, Balthasar Bickel, Moritz M Daum, and Sabine Stoll. 2021. Child-directed speech is optimized for syntax-free semantic inference. Scientific Reports, 11(1):16527. find
  57. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/padovani-2025-child-directed-language-does-not/