Elamur Research⌕
☰
home / papers / 2021 / BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

Summary

The authors test whether a transformer language model can acquire English grammar when it receives the same amount and the same type of speech as a child before the age of six. The answer bears on a long-standing debate in cognitive science: whether linguistic experience alone suffices or whether innate linguistic biases are required. Standard models such as RoBERTa are trained on far larger amounts of text than a child hears, so they say little about child learning. The team trained a scaled-down version of RoBERTa, BabyBERTa, from scratch on the AO-CHILDES corpus, that is, transcripts of adult speech directed at children. Hyperparameters were tuned for small data, and prediction of unmasked tokens was disabled entirely. For comparison, the model was also trained on Wikipedia corpora and on simplified news, AO-Newsela. For evaluation, a new BLiMP-style set of minimal pairs with child-accessible vocabulary was created. In average accuracy, BabyBERTa nearly matched RoBERTa-base, although it has about 15 times fewer parameters and saw about 6000 times fewer words. The model and the test set have been released for psycholinguists, as well as for those developing more economical models.

Key points

  • Data: AO-CHILDES contains about 5M words of American English from the CHILDES database. As controls, three Wikipedia corpora (Wikipedia-1/2/3) and AO-Newsela, news articles simplified for children, were used. All corpora are of roughly the same size.
  • Model: about 8M parameters, 8 layers, 8 attention heads, 256 hidden units. A BPE vocabulary of 8192 units was built from lowercased corpora. The unmasking probability was reduced from 0.1 to 0.0, and each sequence was repeated 10 times with different masks.
  • Comparison with RoBERTa-base: 125M parameters and 30B words versus 5M words; 1024 V100 GPUs for a day versus a single GTX1080 for two hours. Average accuracy on the test set is 81.0 for RoBERTa-base and 80.5 for BabyBERTa.
  • Limitations: of 13 grammatical phenomena, the model performs noticeably above chance on most. However, its performance stays at chance on adjunct islands, ellipsis, gender agreement, NPI licensing, and superlative quantifiers.
  • Resource: a new test set of grammatical minimal pairs, compatible with the restricted vocabulary of child-directed speech, has been released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Psycholinguistics & language acquisition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.conll-1.49

@inproceedings{huebner2021babyberta,
  title = {{BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language}},
  author = {Huebner, Philip A. and Sulem, Elior and Cynthia, Fisher and Roth, Dan},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.49},
  url = {https://doi.org/10.18653/v1/2021.conll-1.49}
}

References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Geoff Bacon and Terry Regier. 2019. Does BERT agree? Evaluating knowledge of structure dependence through agreement relations. arXiv preprint arXiv:1908.09892. — [arXiv:1908.09892](https://arxiv.org/abs/1908.09892) find
  2. Ellen Gurman Bard and Anne H Anderson. 1994. The unintelligibility of speech to children: effects of referent availability. J. Child Lang., 21(3):623–648. find
  3. Andrew Biemiller. 2003. Vocabulary: Needed if more children are to read well. Reading psychology, 24(3-4):323–335. find
  4. Rebecca L Gómez and LouAnn Gerken. 2000. Infant artificial language learning and language acquisition. Trends in cognitive sciences, 4(5):178–186. find
  5. Noah Daniel Goodman, Joshua B Tenenbaum, and Michael C Frank. 2007. A bayesian framework for cross-situational word-learning. In Proc. of Advances in Neural Information Processing Systems. find
  6. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find our card
  7. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  8. Betty Hart and Todd R Risley. 1995. Meaningful differences in the everyday experience of young American children. Paul H Brookes Publishing. find
  9. Donald P Hayes and Margaret G Ahrens. 1988. Vocabulary simplification for children: A special case of ‘motherese’? Journal of child language, 15(2):395– 410. find
  10. Thea Cameron-Faulkner, Elena Lieven, and Michael Tomasello. 2003. A construction based analysis of child directed speech. Cogn. Sci., 27(6):843–873. find
  11. Alex De Carvalho, Isabelle Dautriche, Isabelle Lin, and Anne Christophe. 2017. Phrasal prosody constrains syntactic analysis in toddlers. Cognition, 163:67– 79. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  13. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find our card
  14. Ruthe Foushee, Tom Griffiths, and Mahesh Srinivasan. 2016. Lexical complexity of child-directed and overheard speech: Implications for learning. In CogSci. find
  15. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), page 650–655. find our card
  16. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA. Association for Computational Lin… find
  17. Philip A Huebner and Jon A Willits. 2021. Using lexical context to discover the noun category: Younger children have it easier. In Psychology of learning and motivation, volume 75. Elsevier. find
  18. Nenagh Kemp, Elena Lieven, and Michael Tomasello. 2005. Young children’s knowledge of the “determiner” and “adjective” categories. J. Speech Lang. Hear. Res., 48(3):592–609. find
  19. Katrin Kirchhoff and Steven Schimmel. 2005. Statistical properties of infant-directed versus adult-directed speech: Insights from speech recognition. The Journal of the Acoustical Society of America, 117(4):2238–2246. find
  20. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning. find
  21. Jeffrey Lidz and Laurel Perkins. 2018. Language acquisition. In Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience, pages 1– 49. Wiley. find
  22. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020a. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  23. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210– 5217, Online. Association for Computational Linguistics. find our card
  24. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020b. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. find our card
  25. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annu. Rev. Linguist., 7(1):195–212. find
  26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find our card
  27. Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. 2021. Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.07885. find
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  29. Brian MacWhinney. 2000. The CHILDES Project: Tools for analyzing talk. transcription format and programs, volume 1. Psychology Press. find
  30. Danielle Matthews and Colin Bannard. 2010. Children’s production of unfamiliar word sequences is predicted by positional variability and latent classes in a large sample of child-directed speech. Cognitive science, 34(3):465–488. find
  31. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140. find our card
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  33. Patricia A Reeder, Elissa L Newport, and Richard N Aslin. 2017. Distributional learning of subcategories in an artificial grammar: Category generalization and subcategory restrictions. Journal of memory and language, 97:17–29. find
  34. Douglas Roland, Frederic Dick, and Jeffrey L Elman. 2007. Frequency of basic english grammatical structures: A corpus analysis. J. Mem. Lang., 57(3):348– 379. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 371–376, Berlin, Germany. Association for Computational Linguistics. find our card
  36. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736. find
  37. Virginia Valian. 1986. Syntactic categories in the speech of young children. Dev. Psychol., 22(4):562– 579. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. find our card
  39. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Proceedings of NeurIPS. find our card
  40. Alex Warstadt and Samuel R Bowman. 2020. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society. find
  41. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo-nananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020a. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8(0):377–392. find
  42. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217–235… find
  43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card
  44. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. find our card
  45. Karolina Zaczynska, Nils Feldhus, Robert Schwarzen-berg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating German transformer language models with syntactic agreement tests. In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing, S… find
  46. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. 2021. When do you need billions of words of pretraining data? In "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huebner-2021-babyberta-learning-more-grammar-small/