Elamur Research⌕
☰
home / papers / 2024 / Is Child-Directed Speech Effective Training Data for Language Models?

Is Child-Directed Speech Effective Training Data for Language Models?

Summary

Children acquire language after hearing thousands of times fewer words than modern language models require. The authors test whether this advantage is explained by special properties of child-directed speech. To this end, they train GPT-2 and RoBERTa from scratch on approximately 29M words of English CHILDES. They compare it with TinyDialogues, a synthetic corpus of the same size created with GPT-4, as well as with OpenSubtitles, Wikipedia and the BabyLM data mixture. The models' syntactic knowledge is evaluated with the Zorro benchmark, and their semantic knowledge with word similarity tests. Two factors are examined separately: the age-based order in which conversations are presented (global) and the sequence of utterances within a dialogue (local). It turned out that an age-ordered curriculum has almost no effect on the outcome. In contrast, disrupting dialogue coherence impairs learning, and more diverse corpora yield better models. Thus, the efficiency of children's learning is most likely explained by the learning algorithm rather than by the uniqueness of the input data. This is important for research on training language models on small amounts of data.

Key points

  • Resource: TinyDialogues — approximately 29M words of synthetic dialogues centered on a child aged 2, 5, 10 or 15. The data and code are open: github.com/styfeng/TinyDialogues.
  • GPT-2 (124M): the highest scores were obtained on BabyLM (Zorro ≈82.9%, WS ≈0.42) and OpenSubtitles (≈81.0%, 0.38). CHILDES has the lowest WS score (≈0.24); TinyDialogues outperforms CHILDES on semantics (0.42).
  • Data presentation order: ordering from younger to older age, the reverse order and a random order yield practically identical results. For RoBERTa this experiment did not converge, so the authors do not interpret its results.
  • Shuffling utterances within a dialogue: worsens results on CHILDES, where the average utterance is about 4 words long, but has almost no effect on TinyDialogues, where utterances are longer (about 13 words).
  • Limitations: approximately 90% of CHILDES consists of data on children aged 2–5. Only small models were trained, and the set of benchmarks is limited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining · Psycholinguistics & language acquisition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.1231 · arXiv 2408.03617

@inproceedings{feng2024is,
  title = {{Is Child-Directed Speech Effective Training Data for Language Models?}},
  author = {Feng, Steven Y. and Goodman, Noah D. and Frank, Michael C.},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1231},
  eprint = {2408.03617},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1231}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. find our card
  2. Nasim Borazjanizadeh. 2023. Optimizing GPT-2 pre-training on BabyLM corpus with difficulty-based sentence reordering. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 356–365, Singapore. Association for Computational Linguistics. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find our card
  4. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 136–145, Jeju Island, Korea. Association for Computational Linguistics. find our card
  5. Aryaman Chobey, Oliver Smith, Anzi Wang, and Grusha Prasad. 2023. Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior? In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Lea… find
  6. Xudong Hong, Sharid Loáiciga, and Asad Sayeed. 2023. A surprisal oracle for active curriculum language modeling. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 259–268, Singapore. Association for Computational Linguistics. find
  7. Morten H Christiansen and Nick Chater. 1999. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23(2):157–205. find
  8. Philip A Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. Babyberta: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th conference on computational natural language learning, pages 624–646. find our card
  9. Rhodri Cusack, Marc’Aurelio Ranzato, and Christine J Charvet. 2024. Helpless infants are learning a foundation model. Trends in Cognitive Sciences. find
  10. Baxter S Eaves Jr, Naomi H Feldman, Thomas L Griffiths, and Patrick Shafto. 2016. Infant-directed speech is consistent with teaching. Psychological review, 123(6):758. find
  11. Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759. find our card
  12. Jeffrey L. Elman. 1993. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71–99. find
  13. Naja Ferjan Ramírez, Sarah Roseberry Lytle, and Patricia K Kuhl. 2020. Parent coaching increases conversational turns and advances infant language development. Proceedings of the National Academy of Sciences, 117(7):3484–3491. find
  14. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. ACM Transactions on Information Systems - TOIS, 20:406–414. find
  15. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. arXiv preprint arXiv:2401.06416. find
  16. Victor Kuperman, Hans Stadthagen-González, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior Research Methods, 44:978–990. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. find
  18. Bria Long, Violet Xiang, Stefan Stojanov, Robert Z. Sparks, Zi Yin, Grace E. Keene, Alvin W. M. Tan, Steven Y. Feng, Chengxu Zhuang, Virginia A. Marchman, Daniel L. K. Yamins, and Michael C. Frank. 2024. The babyview dataset: High-resolution egocentric videos of infants’ and young children’s everyda… — [arXiv:2406.10447](https://arxiv.org/abs/2406.10447) find
  19. Brian MacWhinney. 2014. The CHILDES project: Tools for analyzing talk, Volume I: Transcription format and programs. Psychology Press. find
  20. Michael C Frank. 2023b. Openly accessible llms can help us to understand human cognition. Nature Human Behaviour, 7(11):1825–1827. find
  21. Richard Diehl Martinez, Hope McGovern, Zebulon Goriely, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. CLIMB – curriculum learning for infant-inspired model building. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning,… find
  22. Michael C Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A Marchman. 2021. Variability and consistency in early language learning: The Wordbank project. MIT Press. find
  23. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing aanns. Preprint, arXiv:2403.19827. find
  24. Michael C Frank. 2023a. Bridging the data gap between children and large language models. Trends in Cognitive Sciences. find
  25. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182, Austin, Texas. Association for Computational Lingui… find our card
  26. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  27. Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, and Anna Rumshisky. 2024. Emergent abilities in reduced-scale generative language models. Preprint, arXiv:2404.02204. find
  28. Elissa L. Newport. 1988. Constraints on learning and their role in language acquisition: Studies of the acquisition of american sign language. Language Sciences, 10(1):147–172. find
  29. Elissa L Newport. 1990. Maturational constraints on language learning. Cognitive science, 14(1):11–28. find
  30. Yulu Qin, Wentao Wang, and Brenden M. Lake. 2024. A systematic investigation of learnability from single child linguistic input. In Proceedings of the 46th Annual Conference of the Cognitive Science Society. find
  31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  32. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. In Communications of the ACM (CACM) 8 (10). find
  33. Saber Sheybani, Himanshu Hansaria, Justin Wood, Linda Smith, and Zoran Tiganj. 2024. Curriculum learning with infant egocentric videos. Advances in Neural Information Processing Systems, 36. find
  34. Linda B Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. 2018. The developing infant creates a curriculum for statistical learning. Trends in Cognitive Sciences, 22(4):325–336. find
  35. Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C Frank. 2021. Saycam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective. Open mind, 5:20–29. find
  36. Alex Warstadt and Samuel R. Bowman. 2024. What artificial neural networks can tell us about human language acquisition. Preprint, arXiv:2208.07998. find our card
  37. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Chal… find
  38. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. find
  39. Guanghao You, Balthasar Bickel, Moritz M Daum, and Sabine Stoll. 2021. Child-directed speech is optimized for syntax-free semantic inference. Scientific Reports, 11(1):16527. find
  40. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… find
  41. Chengxu Zhuang, Evelina Fedorenko, and Jacob Andreas. 2023. Visual grounding helps learn word meanings in low-data regimes. arXiv preprint arXiv:2310.13257. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2024-is-child-directed-speech-effective/