Elamur Research⌕
☰
home / papers / 2023 / How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

Paper · 2023 · Aaron Mueller and Tal Linzen

Summary

Language models that read text as flat word sequences may rely on positional shortcuts, such as "the first noun is the subject," instead of syntactic structure, which limits robust generalization. Mueller and Linzen ask which parts of pre-training give encoder-decoder Transformers a preference for hierarchical rules. They fine-tune T5-style models, using both public "efficient" checkpoints from Tay et al. (2022) and models they pre-train themselves on corpora of different genres and sizes. The fine-tuning tasks are two English syntactic transformations: question formation and passivization. The training examples fit both a structural rule and a linear heuristic. Test sentences with an inserted relative clause or prepositional phrase then reveal which rule the model actually learned. Raw parameter count does not predict hierarchical generalization, and adding layers helps more than widening them. Pre-training on child-directed speech produces the hierarchical bias with roughly an order of magnitude less text than Wikipedia or web data. These results inform data curation for pre-training and suggest Transformers can be credible models of human language acquisition.

Key points

  • Paradigm: The study uses a poverty-of-the-stimulus design.

    • In question formation, MOVE-MAIN (the correct rule) is pitted against MOVE-FIRST.
    • In passivization, MOVE-MAIN is pitted against MOVE-SECOND.
    • The data are taken from the English-language datasets released by Mueller et al. (2022), which themselves extend McCoy et al. (2020).

  • Metrics: Models are scored with exact sequence accuracy plus a targeted measure for each task: main auxiliary accuracy for questions and object accuracy for passives.

  • Architectures: The T5 variants from Tay et al. (2022) are:

    VariantParametersLayers per encoder/decoder
    Tiny16M4
    Mini31M4
    Small60M6
    Base220M12

    Across these variants, depth outweighs width for hierarchical generalization.

  • Data efficiency: Simpler input, such as child-directed speech, yields the hierarchical bias with far less data than Wikipedia or C4-type web text. The authors propose a two-phase "starting small" pre-training scheme based on this finding.

  • Limitations and resources: The study covers only T5-like models with a span denoising objective, only English, and only two transformations. Code is available at github.com/aaronmueller/emergent-syntax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States258 works · 108,081 citations🎓New York UniversityNew York · United States233 works · 104,005 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Aaron Mueller and Tal Linzen. 2023. How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases. arXiv:2305.19905.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.19905 · arXiv 2305.19905

@misc{mueller2023how,
  title = {{How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases}},
  author = {Aaron Mueller and Tal Linzen},
  year = {2023},
  doi = {10.48550/arxiv.2305.19905},
  eprint = {2305.19905},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.19905}
}

References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
  2. Andrew Biemiller. 2003. Vocabulary: Needed if more children are to read well. Reading Psychology, 24(3-4):323–335. find
  3. Daniel Campos. 2021. Curriculum learning for language modeling. Computing Research Repository, arXiv:2108.02170. — [arXiv:2108.02170](https://arxiv.org/abs/2108.02170) find
  4. Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41–75. find
  5. Jeffrey L. Elman. 1993. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71–99. find
  6. Robert Frank and Donald Mathis. 2007. Transformational networks. In Proceedings of the Workshop on Psychocomputational Models of Human Language Acquisition. Cognitive Science Society. find
  7. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for…
  8. Lieke Gelderloos, Grzegorz Chrupała, and Afra Alishahi. 2020. Learning to understand child-directed and adult-directed speech. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1–6, Online. Association for Computational Linguistics. find
  9. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. Computing Research Repository, arXiv:1901.05287.
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  11. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366. find
  12. Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. 2022. Artificial neural network language models align neurally and behaviorally with humans even after a developmentally realistic amount of training. bioRxiv. find
  13. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  14. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics.
  15. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  16. Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699. Special Issue in Honour of Jacques Mehler, Cognition’s founding editor. find
  17. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  18. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210– 5217, Online. Association for Computational Linguistics.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692.
  20. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre-trained models. In International Conference on learning representations. find
  21. Brian MacWhinney. 2000. The CHILDES project: The database, volume 2. Psychology Press. find
  22. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  23. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, pages 2096–2101, Madison, Wisconsin. Cognitiv… find
  24. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140.
  25. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
  26. Tom M Mitchell. 1980. The need for biases in learning generalizations (Rutgers Computer Science Tech. Rept. CBM-TR-117). Rutgers University. find
  27. Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, and Sebastian Schuster. 2022. Coloring the blank slate: Pre-training imparts a hierarchical inductive bias to sequence-to-sequence models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1352–1368, Dublin, Ireland. A… find
  28. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput…
  29. Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. 2023. Characterizing intrinsic compositionality in transformers with tree projections. In International Conference on Learning Representations. find
  30. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710–3723, Online. As… find
  31. Jackson Petty and Robert Frank. 2021. Transformers generalize linearly. Computing Research Repository, arXiv:2109.12036. — [arXiv:2109.12036](https://arxiv.org/abs/2109.12036) find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  34. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon…
  35. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of Basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. find
  36. Melanie Soderstrom. 2007. Beyond babytalk: Re-evaluating the nature and content of speech input to preverbal infants. Developmental Review, 27(4):501– 532. find
  37. Maxim Surkov, Vladislav Mosin, and Ivan Yamshchikov. 2022. Do data-based curricula work? In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 119– 128, Dublin, Ireland. Association for Computational Linguistics. find
  38. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2022. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations. find
  39. Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn’t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… find
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  41. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  42. Alex Warstadt and Samuel R. Bowman. 2020. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, Online. Cognitive Science Society. find
  43. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020a. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  44. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217–23… find
  45. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computation…
  46. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati…
  47. Colin Wilson. 2006. Learning phonology with substantive bias: An experimental and computational study of velar palatalization. Cognitive Science, 30(5):945– 982. find
  48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mueller-2023-plant-trees-language-models-data/