Elamur Research⌕
☰
home / papers / 2023 / BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition

BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition

Summary

The BabyLM Challenge is a shared task in which language models are trained on a small corpus. Its size approximates the amount of language a child hears at an early age. The authors built on the fact that adults first speak to young children in simple, short phrases, and the complexity of speech increases as the child grows older. They tested whether curriculum learning, i.e., presenting examples from simple to complex, would help the model. The strict-small corpus (~10M words) was split into sentences, filtered, and case-normalized using the Moses truecaser. A RoBERTa-base model was trained from scratch. Sentences were ordered by three complexity measures: the number of subword tokens, the number of constituents, and the maximum depth of the dependency tree (max-dependency). The sorted data were divided into blocks and fed to the model sequentially. Of the three measures, max-dependency worked best. However, none of the curriculum learning variants outperformed the baseline model trained on shuffled data. The conclusion is negative but useful: mechanically transferring the idea of child-directed speech to model training does not automatically improve quality.

Key points

  • Evaluation was performed on BLiMP (zero-shot, accuracy), (Super)GLUE (F1), and MSGS (MCC). Syntactic complexity measures were computed with spaCy parsers.
  • Data preprocessing alone noticeably helped the baseline model: the average score rose from 44.80 to 47.91, and BLiMP from 69.23 to 70.46.
  • Average scores of the curriculum models: Ntoken — 42.59, Nconst. — 42.63, Max-dep. — 45.81. All of them are lower than the baseline model with data cleaning (47.91).
  • For Max-dep., splits into 3, 4, 6, and 8 blocks were compared. The highest average score (47.46) came from the 6-block split, but the authors consider the differences between the variants insignificant. The 4-block variant was chosen as the main model; it is published on Hugging Face as akari000/roberta-dependency-max-4split.
  • The authors acknowledge a limitation: each model was trained with only one random seed. For future work, they recommend examining the effect of the seed, as well as other training settings, in particular warmup and optimizers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Psycholinguistics & language acquisition

Cite

Cite as Miyu Oba, Akari Haga, Akiyo Fukatsu and Yohei Oseki. 2023. BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition. Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.conll-babylm.25

@inproceedings{oba2023babylm,
  title = {{BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition}},
  author = {Miyu Oba, Akari Haga, Akiyo Fukatsu and Yohei Oseki},
  booktitle = {Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning},
  year = {2023},
  doi = {10.18653/v1/2023.conll-babylm.25},
  url = {https://doi.org/10.18653/v1/2023.conll-babylm.25}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery. our card
  2. Anne Fernald, Traute Taeschner, Judy Dunn, Mechthild Papousek, Bénédicte de Boysson-Bardies, and Ikuko Fukui. 1989. A cross-language study of prosodic modifications in mothers’ and fathers’ speech to preverbal infants. Journal of Child Language. find
  3. Federmeier and Lili Sahakyan, editors, The Context of Cognition: Emerging Perspectives, volume 75 of Psychology of Learning and Motivation, pages 279– 331. Academic Press. find
  4. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of EACL: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. find
  5. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of ACL (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics. our card
  6. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  8. Elissa. Newport, Henry Gleitman, and Lila Gleitman. 1977. Mother, id rather do it myself: Some effects and non-effects of maternal speech style. In Catherine E. Snow and Charles A. Ferguson, editors, Talking to Children, pages 109–149. Cambridge University Press. find
  9. Joakim Nivre and Jens Nilsson. 2005. Pseudo-projective dependency parsing. In Proceedings of ACL, pages 99–106, Ann Arbor, Michigan. Association for Computational Linguistics. our card
  10. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. find
  11. Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kim-brough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American Journal of Speech-Language Pathology… find
  12. Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of EMNLP, pages 1373–1378, Lisbon, Portugal. Association for Computational Linguistics. our card
  13. Philip A. Huebner and Jon A. Willits. 2021. Chapter eight - using lexical context to discover the noun category: Younger children have it easier. In Kara D. find
  14. Catherine E. Snow. 1972. Mothers’ speech to children learning language. Child Development, 43(2):549– 565. find
  15. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… our card
  16. Alex Warstadt, Leshem Choshen, Ryan Cotterell, Tal Linzen, Aaron Mueller, Ethan Wilcox, Williams Adina, and Chengxu Zhuang. 2023. Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge. Association for Computational… find
  17. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020a. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377– 392. find
  18. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the EMNLP, pages 217–235, Online. Association for Computational Linguistics. find
  19. Lukas Wertz, Katsiaryna Mirylenka, Jonas Kuhn, and Jasmina Bogojeska. 2022. Investigating active learning sampling strategies for extreme multi label text classification. In Proceedings of LREC, pages 4597– 4605, Marseille, France. European Language Resources Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oba-2023-babylm-challenge-curriculum-learning-based/