Elamur Research⌕
☰
home / papers / 2022 / What Artificial Neural Networks Can Tell Us About Human Language Acquisition

What Artificial Neural Networks Can Tell Us About Human Language Acquisition

Summary

The authors examine how far artificial neural networks can help clarify how humans acquire language. The central problem is that modern language models learn under conditions very different from those of children. In particular, they receive roughly a thousand times more text than a typical child. This is a conceptual rather than an experimental work, written as a chapter in an edited volume. The authors describe a learning scenario as the combination of two components: the learner's innate biases and the learning environment. As their main tool, they propose ablation experiments, in which a model is trained with a hypothesized advantage deliberately removed. If a model without that advantage still acquires the target knowledge, and has no other advantages over a human, the result shows that the knowledge is learnable without it. Positive results therefore carry more weight than negative ones. Such studies are impossible to conduct on humans for ethical and practical reasons. So far, models trained on reduced amounts of data judge sentence acceptability worse than humans do. Before concluding that humans have innate linguistic knowledge, then, models should be given non-linguistic signals: images, video, and interaction with other agents.

Key points

  • Amount of linguistic input (Fig. 1): by age 10, a child hears about 100M words. ELMo was trained on roughly 1B words, BERT on 3B, RoBERTa on 30B, and GPT-3 on 200B.
  • Table 1 breaks the learning scenario down into variables. On the learner side, these are the architecture, the learning algorithm, the hyperparameters, and the order in which data are presented. On the environment side, they are the input modality, the amount and distribution of data, and memory.
  • Following the procedure of Warstadt et al. (2020b), a full simulation of human language development takes about a week on a single computer.
  • 2024 correction: the English-language CHILDES corpora, including children's own speech, contain more than 20 million words. The paper had previously estimated about 5 million.
  • The authors propose benchmarks and competitions with strict limits on the amount and type of pretraining data, in order to obtain more plausible cognitive models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Psycholinguistics & language acquisition

Cite

Cite as Alex Warstadt and Samuel R. Bowman. 2022. What Artificial Neural Networks Can Tell Us About Human Language Acquisition. arXiv:2208.07998.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2208.07998 · arXiv 2208.07998

@misc{warstadt2022what,
  title = {{What Artificial Neural Networks Can Tell Us About Human Language Acquisition}},
  author = {Alex Warstadt and Samuel R. Bowman},
  year = {2022},
  doi = {10.48550/arxiv.2208.07998},
  eprint = {2208.07998},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.07998}
}

References

163 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omri Abend, Tom Kwiatkowski, Nathaniel J. Smith, Sharon Goldwater, and Mark Steedman. Bootstrapping language acquisition. Cognition, 164:116–143, July 2017. ISSN 00100277. doi: 10.1016/j.cognition.2017.02.009. URL https://linkinghub.elsevier.com/retrieve/pii/S0010027717300495. — [doi:10.1016/j.cognition.2017.02.009](https://doi.org/10.1016/j.cognition.2017.02.009) ↗
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track. Toulon, France., 2017. our card
  3. JL Austin. How to Do Things With Words. Oxford University Press, 1962. find
  4. Carl Lee Baker. Introduction to Generative-Transformational Synta. Prentice-Hall, Englewood Cliffs, NJ, 1978. find
  5. Marco Baroni. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv:2106.08694 [cs], June 2021. URL http://arxiv.org/abs/2106.08694. arXiv: 2106.08694. — [arXiv:2106.08694](https://arxiv.org/abs/2106.08694) ↗
  6. Yonatan Belinkov and James R. Glass. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49–72, 2019. URL https://www.aclweb.org/anthology/Q19-1004.pdf. ↗ our card
  7. Robert C Berwick, Paul Pietroski, Beracah Yankama, and Noam Chomsky. Poverty of the stimulus revisited. Cognitive Science, 35(7):1207–1242, 2011. find
  8. Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emergent language game. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 981–985, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.186… find
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  10. Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 32, 2019. find
  11. Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International conference on learning representations, 2022. find
  12. Tyler A. Chang and Benjamin K. Bergen. Word Acquisition in Neural Language Models. Transactions of the Association for Computational Linguistics, 10: URL 1–16, January 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00444. https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00444/109271/Word-Acquisitio… ↗
  13. Rui P. Chaves. What Don’t RNN Language Models Learn About Filler-Gap Dependencies? In Proceedings of the third meeting of the Society for Computation in Linguistics (SCiL), 2020. find
  14. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. In European conference on computer vision, pages 740–755. Springer, April 2015. URL http://arxiv.org/abs/1504.00325. arXiv:… ↗ our card
  15. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer vision – ECCV 2020, pages 104–120, Cham, 2020. Spring… our card
  16. Noam Chomsky. Aspects of the Theory of Syntax. MIT Press, 1965. find
  17. Noam Chomsky. Problems of knowledge and freedom: The Russell lectures. 1971. find
  18. Noam Chomsky. Biolinguistic Explorations: Design, Development, Evolution. International Journal of Philosophical Studies, 15(1):1–21, January 2007. ISSN 0967-2559, 1466-4542. doi: 10.1080/09672550601143078. URL http://www.tandfonline.com/doi/abs/10.1080/09672550601143078. — [doi:10.1080/09672550601143078](https://doi.org/10.1080/09672550601143078) ↗
  19. Noam Chomsky and Howard Lasnik. The theory of principles and parameters. In The minimalist program. MIT Press, 1993. find
  20. Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-learning trajectories of neural language models. arXiv preprint arXiv:2109.06096, 2021. find
  21. Michelle M Chouinard and Eve V Clark. Adult reformulations of child errors as negative evidence. Journal of child language, 30(3):637–669, 2003. Publisher: Cambridge University Press. find
  22. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and others. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. our card
  23. Shammur Absar Chowdhury and Roberto Zamparelli. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th international conference on computational linguistics, pages 133–144, 2018. find
  24. Morten H Christiansen and Nick Chater. Creating language: Integrating evolution, acquisition, and processing. MIT Press, 2016. find
  25. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs], December 2014. URL http://arxiv.org/abs/1412.3555. arXiv: 1412.3555. ↗
  26. Alexander Clark and Shalom Lappin. Linguistic Nativism and the Poverty of the Stimulus. John Wiley & Sons, 2011. find
  27. G. G. Coulton. The Princes of the World. In From St. Francis to Dante, Translations from the Chronicle of the Franciscan Salimbene, 1221-1288, pages 239–256. University of Pennsylvania Press, 2 edition, 1972. ISBN 978-0-8122-7672-5. URL https://www.jstor.org/stable/j.ctv4t8279.25. ↗
  28. Stephen Crain and Mineharu Nakayama. Structure Dependence in Grammar Formation. Language, 63(3):522–543, 1987. URL https://www.jstor.org/stable/415004. ↗
  29. Alejandrina Cristia, Emmanuel Dupoux, Michael Gurven, and Jonathan Stieglitz. Child-Directed Speech Is Infrequent in a Forager-Farmer Population: A Time Allocation Study. Child Development, 90(3):759–773, May 2019. ISSN 0009-3920, 1467-8624. doi: 10.1111/cdev.12974. URL https://onlinelibrary.wiley.c… — [doi:10.1111/cdev.12974](https://doi.org/10.1111/cdev.12974) ↗
  30. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in neural information processing systems, volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/7137debd45ae4d0ab9aa… ↗
  31. Mark Davies. The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights. International journal of corpus linguistics, 14(2):159–190, 2009. Publisher: John Benjamins. find
  32. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… our card
  33. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Neural Models for Reasoning over Multiple Mentions Using Coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… our card
  34. Emmanuel Dupoux. Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43–59, April 2018. ISSN 00100277. doi: 10.1016/j.cognition.2017.11.008. URL http://arxiv.org/abs/1607.08723. arXiv: 1607.08723. — [doi:10.1016/j.cognition.2017.11.008](https://doi.org/10.1016/j.cognition.2017.11.008) ↗
  35. Gabe Dupre. (What) Can Deep Learning Contribute to Theoretical Linguistics? Minds and Machines, September 2021. ISSN 0924-6495, 1572-8641. doi: 10.1007/s11023-021-09571-w. URL https://link.springer.com/10.1007/s11023-021-09571-w. — [doi:10.1007/s11023-021-09571-w](https://doi.org/10.1007/s11023-021-09571-w) ↗
  36. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9: 1012–1031, 2021a. find
  37. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021b. our card
  38. Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990. Publisher: Wiley Online Library. find
  39. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 134–139, 2016. URL https://www.aclweb.org/anthology/W16-2524. ↗ our card
  40. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International J… find
  41. Janet Dean Fodor and Carrie Crowther. Understanding stimulus poverty arguments. The Linguistic Review, 18(1-2), January 2002. ISSN 0167-6318, 1613-3676. doi: 10.1515/tlir.19.1-2.105. URL https://www.degruyter.com/document/doi/10.1515/tlir.19.1-2.105/html. — [doi:10.1515/tlir.19.1-2.105](https://doi.org/10.1515/tlir.19.1-2.105) ↗
  42. Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71, 1988. Publisher: Elsevier. find
  43. Michael C. Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A. Marchman. Wordbank: An open repository for developmental vocabulary data. Journal of Child Language, 44(3):677–694, May 2017. ISSN 0305-0009, 1469-7602. doi: 10.1017/S0305000916000209. URL https://www.cambridge.org/core/product/ident… — [doi:10.1017/s0305000916000209](https://doi.org/10.1017/s0305000916000209) ↗
  44. Robert Frank and Donald Mathis. Transformational networks. Models of Human Language Acquisition, page 22, 2007. find
  45. Victoria Fromkin, Stephen Krashen, Susan Curtiss, David Rigler, and Marilyn Rigler. The Development of Language in Genie: a Case of Language Acquisition beyond the ”Critical Period”. Brain and Language, 1:81–107, 1974. find
  46. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko. The Natural Stories corpus: a reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55(1):63–77, March 2021. ISSN… — [doi:10.1007/s10579-020-09503-7](https://doi.org/10.1007/s10579-020-09503-7) find
  47. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. SyntaxGym: An Online Platform for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online, July 2020. Association… our card
  48. Dedre Gentner. Why nouns are learned before verbs: Linguistic relativity versus natural partitioning. In S Kuczaj, editor, Language development: Language cognition and culture., page 48. 1982. find
  49. Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL Hansen, and Terrance D Paul. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):248–265,… find
  50. Lila Gleitman and Eric Wanner. Language Acquisition: The State of the Art. In Lila Gleitman and Eric Wanner, editors, Language Acquisition: The State of the Art. Cambridge University Press, 1982. find
  51. E Mark Gold. Language identification in the limit. Information and control, 10(5):447– 474, 1967. Publisher: Elsevier. find
  52. Peter Gordon. Level-ordering in lexical development. Cognition, pages 73–93, 1985. find
  53. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream hierarchically. Proceedings of the Society for Computation in Linguistics, 2(1):363–364, 2019. our card
  54. Rebecca L Gómez and LouAnn Gerken. Infant artificial language learning and language acquisition. Trends in cognitive sciences, 4(5):178–186, 2000. Publisher: Elsevier. find
  55. John Hale. A Probabilistic Earley Parser as a Psycholinguistic Model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2001. URL https://aclanthology.org/N01-1021. ↗ our card
  56. Chung-hye Han, Julien Musolino, and Jeffrey Lidz. Endogenous sources of variation in language acquisition. Proceedings of the National Academy of Sciences, 113(4):942– 947, January 2016. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1517094113. URL https://pnas.org/doi/full/10.1073/pnas.1517094113. — [doi:10.1073/pnas.1517094113](https://doi.org/10.1073/pnas.1517094113) ↗
  57. Betty Hart and Todd R. Risley. American parenting of language-learning children: Persisting differences in family-child interactions observed in natural home environments. Developmental Psychology, 28(6):1096, 1992. URL https://psycnet.apa.org/fulltext/1993-09151-001.pdf. Publisher: American Psychol… ↗
  58. David Haussler. Probably approximately correct learning. In Proceedings of the eighth national conference on artificial intelligence, pages 1101–1108. AAAI Press, 1990. find
  59. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In International conference on learning representations, 2020. our card
  60. John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https://www.aclweb.org/anthology/D19-1275. event-place: Hong Kong. ↗
  61. John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138,… our card
  62. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. Publisher: MIT Press. find
  63. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, 2018. find
  64. Steve R. Howell, Damian Jankowicz, and Suzanna Becker. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. Journal of Memory and Language, 53(2):258–276, August 2005. ISSN 0749596X. doi: 10.1016/j.jml.2005.03.002. URL https://linkinghub.elsevier.… — [doi:10.1016/j.jml.2005.03.002](https://doi.org/10.1016/j.jml.2005.03.002) ↗
  65. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online, July 2020. Association for Computati… find
  66. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language pro… our card
  67. Philip A. Huebner and Jon A. Willits. Using lexical context to discover the noun category: Younger children have it easier. In Kara D. Federmeier and Lili Sahakyan, editors, The context of cognition: Emerging perspectives, volume 75 of Psychology of learning and motivation, pages 279–331. Academic P… — [doi:10.1016/bs.plm.2021.08.002](https://doi.org/10.1016/bs.plm.2021.08.002) find
  68. Taichi Iki and Akiko Aizawa. Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2189–2196, Online and Punta Cana, Dominican Republic, 2021. Association for Compu… find
  69. Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021. find
  70. Slava Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE transactions on acoustics, speech, and signal processing, 35(3):400–401, 1987. find
  71. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring Compositional Generalization: A Comprehensive Method on Reali… find
  72. Najoung Kim and Tal Linzen. COGS: A Compositional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online, November 2020. Association for Computational Linguistics. doi: 10.1… find
  73. John P. Kimball. The Formal Theory of Grammar. Prentice-Hall, Englewood Cliffs, NJ, 1973. find
  74. Simon Kirby. Function, selection, and innateness: The emergence of language universals. OUP Oxford, 1999. find
  75. Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge ‘naturally’ in multi-agent dialog. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2962–2967, Copenhagen, Denmark, September 2017. Association for Computational… find
  76. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journal of… our card
  77. Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2879–2888, 2018. find
  78. Brenden M Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instructions. In Proceedings of the 41st Annual Conference of the Cognitive Science Society, May 2019. find
  79. Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. Generative Spoken Language Modeling from Raw Audio. arXiv:2102.01192 [cs], September 2021. URL http://arxiv.org/abs/2102.01192… — [arXiv:2102.01192](https://arxiv.org/abs/2102.01192) ↗
  80. Thomas K Landauer and Susan T Dutnais. A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. Psychological Review, 104(2):211–240, 1997. find
  81. Jey Han Lau, Alexander Clark, and Shalom Lappin. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5): 1202–1241, 2017. Publisher: Wiley Online Library. find
  82. Marvin Lavechin, Maureen de Seyssel, Marianne Métais, Florian Metze, Abdelrahman Mohamed, Hervé BREDIN, Emmanuel Dupoux, and Alejandrina Cristia. Early phonetic learning from ecological audio: domain-general versus domain-specific mechanisms. 2022. Publisher: PsyArXiv. find
  83. Angeliki Lazaridou and Marco Baroni. Emergent Multi-Agent Communication in the Deep Learning Era. arXiv:2006.02419 [cs], July 2020. URL http://arxiv.org/abs/2006.02419. arXiv: 2006.02419. — [arXiv:2006.02419](https://arxiv.org/abs/2006.02419) ↗
  84. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. Combining Language and Vision with a Multimodal Skip-gram Model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 153–163, Denver, Colorado, M… our card
  85. Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations, March 2017. URL http://arxiv.org/abs/1612.07182. arXiv: 1612.07182. — [arXiv:1612.07182](https://arxiv.org/abs/1612.07182) ↗
  86. Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International conference on learning representations, 2018. find
  87. Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663–7674, Online, July 2020. A… find
  88. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): 436, 2015. find
  89. Julie Anne Legate and Charles D Yang. Empirical re-assessment of stimulus poverty arguments. The Linguistic Review, 18(1-2):151–162, 2002. Publisher: Walter de Gruyter. find
  90. Fred Lerdahl, Ray S Jackendoff, and Ray Jackendoff. A Generative Theory of Tonal Music. MIT Press, 1983. find
  91. Roger Levy. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177, March 2008. ISSN 00100277. doi: 10.1016/j.cognition.2007.05.006. URL https://linkinghub.elsevier.com/retrieve/pii/S0010027707001436. — [doi:10.1016/j.cognition.2007.05.006](https://doi.org/10.1016/j.cognition.2007.05.006) ↗
  92. Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. Advances in neural information processing systems, 32, 2019. find
  93. Jeffrey Lidz, Sandra Waxman, and Jennifer Freedman. What infants know about syntax but couldn’t have learned: Experimental evidence for syntactic structure at 18 months. Cognition, 89(3):295–303, October 2003. ISSN 00100277. doi: 10.1016/S0010-0277(03)00116-1. URL https://linkinghub.elsevier.com/ret… — [doi:10.1016/s0010-0277(03)00116-1](https://doi.org/10.1016/s0010-0277%2803%2900116-1) ↗
  94. Tal Linzen. What can linguistics and deep learning contribute to each other? Response to Pater. Language, 95(1):e99–e108, 2019. Publisher: Linguistic Society of America. find
  95. Tal Linzen and Marco Baroni. Syntactic Structure from Deep Learning. Annual Reviews of Linguistics, 2021. find
  96. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535, 2016. our card
  97. Pierre Lison and Jorg Tiedemann. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), page 7, 2016. our card
  98. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. URL http://arxiv.org/abs/1907.11692. ↗
  99. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting Inductive Biases of Fine-tuned Models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=mNtmhaDkAr. ↗
  100. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/c7… ↗ our card
  101. Brian MacWhinney. The CHILDES project: Tools for analyzing talk, Volume II: The database. Psychology Press, 2014. find
  102. Jessica Mankewitz, Veronica Boyce, Brandon Waldon, Georgia Loukatou, Dhara Yu, Jesse Mu, Noah D Goodman, and Michael C Frank. Multi-party referential communication in complex strategic games. Publisher: PsyArXiv. Christopher D. Manning. Computational Linguistics and Deep Learning. Computational Ling… — [doi:10.1162/coli](https://doi.org/10.1162/coli) find
  103. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054, December 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/p… — [doi:10.1073/pnas.1907367117](https://doi.org/10.1073/pnas.1907367117) find
  104. Gary F Marcus. Negative evidence in language acquisition. Cognition, 46(1):53–85, 1993. Publisher: Elsevier. find
  105. Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, 2018. find
  106. R Thomas McCoy, Robert Frank, and Tal Linzen. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society., 2018. find
  107. R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transactions of the Association for Computational Linguistics, 8:125–140, December 2020. ISSN 2307-387X. doi: 10.1162/tacl a 00304. URL https://d… our card
  108. Tom M Mitchell. The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research . . . , 1980. find
  109. Richard Montague. The proper treatment of quantification in ordinary English. In Approaches to natural language, pages 221–242. Springer, 1973. find
  110. Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. arXiv:2011.11588 [cs, eess], December 2020. URL http://a… — [arXiv:2011.11588](https://arxiv.org/abs/2011.11588) find
  111. A Emin Orhan, Vaibhav V Gupta, and Brenden M Lake. Self-supervised learning through the eyes of a child. page 12. Ludovica Pannitto and Aurélie Herbelot. Recurrent babbling: evaluating the acquisition of grammar from limited input data. In Proceedings of the 24th Conference on Computational Natural… find
  112. Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald. Deep subjecthood: Higher-order grammatical features in multilingual BERT. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: Main volume, pages 2522–2532, Online, Apri… find
  113. Joe Pater. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95(1):e41–e74, 2019. Publisher: Linguistic Society of America. find
  114. Andy Perfors, Joshua B Tenenbaum, and Terry Regier. The learnability of abstract syntactic principles. Cognition, 118(3):306–338, 2011. Publisher: Elsevier. find
  115. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V… our card
  116. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… our card
  117. Jackson Petty and Robert Frank. Transformers Generalize Linearly. arXiv:2109.12036 [cs], September 2021. URL http://arxiv.org/abs/2109.12036. arXiv: 2109.12036. — [arXiv:2109.12036](https://arxiv.org/abs/2109.12036) ↗
  118. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-Theoretic Probing for Linguistic Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609–4622, Online, July 2020. Association for… our card
  119. Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner. How much pretraining data do language models need to learn syntax? September 2021. URL https://arxiv.org/abs/2109.03160v2. ↗
  120. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI, 2018. find
  121. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. our card
  122. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and others. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. find
  123. Ezer Rasin and Athulya Aravind. The nature of the semantic stimulus: the acquisition of every as a case study. Natural Language Semantics, 29(2):339–375, June 2021. ISSN 0925-854X, 1572-865X. doi: 10.1007/s11050-020-09168-6. URL https://link.springer.com/10.1007/s11050-020-09168-6. — [doi:10.1007/s11050-020-09168-6](https://doi.org/10.1007/s11050-020-09168-6) ↗
  124. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages. In Proceedings of NAACL-HLT, pages 3532–3542, 2019. find
  125. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256, Online, 2020. Association for Compu… our card
  126. Florencia Reali and Morten H Christiansen. Uncovering the richness of the stimulus: Structure dependence and indirect statistical evidence. Cognitive Science, 29(6):1007–1028, 2005. Publisher: Wiley Online Library. find
  127. Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International conference on learning representations, 2019. find
  128. Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. In Proceedings of the 19th international conference on autonomous agents and MultiAgent systems, pages 1125–1133, 2020. find
  129. Eric Reuland. Grammar of binding in the languages of the world: Unity versus diversity. Cognition, 168:370–379, November 2017. ISSN 00100277. doi: 10.1016/j.cognition.2016.01.020. URL https://linkinghub.elsevier.com/retrieve/pii/S0010027716300208. — [doi:10.1016/j.cognition.2016.01.020](https://doi.org/10.1016/j.cognition.2016.01.020) ↗
  130. Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. “LazImpa”: Lazy and Impatient neural agents learn to communicate efficiently. In Proceedings of the 24th conference on computational natural language learning, pages 335–343, Online, November 2020. Association for Computational Linguistics. URL htt… find
  131. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What we know about how BERT works. In Findings of EMNLP, 2020. URL https://www.aclweb.org/anthology/2020.tacl-1.54. ↗ our card
  132. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(05), pages 8732–8740, 2020. doi: https://doi.org/10.1609/aaai.v34i05.6399. ↗
  133. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.… our card
  134. Carson T. Schütze. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press, 1996. find
  135. John R. Searle. Speech acts: An essay in the philosophy of language, volume 626. Cambridge University Press, 1969. find
  136. Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, 2016. our card
  137. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2112.04482, 2021. — [arXiv:2112.04482](https://arxiv.org/abs/2112.04482) find
  138. Melanie Soderstrom, Amanda Seidl, Deborah G Kemler Nelson, and Peter W Jusczyk. The prosodic bootstrapping of phrases: Evidence from prelinguistic infants. Journal of Memory and Language, 49(2):249–267, 2003. Publisher: Elsevier. find
  139. Jon Sprouse, Carson T. Schütze, and Diogo Almeida. A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010. Lingua, 134:219–248, 2013. find
  140. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SygXPaEYvH. ↗ our card
  141. Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C. Frank. SAYCam: A Large, Longitudinal Audiovisual Dataset Recorded From the Infant’s Perspective. Open Mind, 5:20–29, May 2021. ISSN URL https://direct.2470-2986.doi: 10.1162/opmi a 00039. mit.edu/opmi/article/doi/10.1162/o… — [doi:10.1162/opmi](https://doi.org/10.1162/opmi) ↗
  142. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464–7473, 2019. our card
  143. Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, Ho… our card
  144. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In Proceedings of ICLR, 2019. our card
  145. L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134– 1142, November 1984. ISSN 0001-0782, 1557-7317. doi: 10.1145/1968.1972. URL https://dl.acm.org/doi/10.1145/1968.1972. — [doi:10.1145/1968.1972](https://doi.org/10.1145/1968.1972) ↗
  146. Annmarie Van Dooren, Anouk Dieuleveut, Ailı́s Cournane, and Valentine Hacquard. Figuring out root and epistemic uses for modals: The role of the input. Journal of Semantics. Marten van Schijndel, Aaron Mueller, and Tal Linzen. Quantity doesn’t buy quality syntax with neural language models. In Proce… find
  147. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S… our card
  148. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in neural information proce… find
  149. Elena Voita and Ivan Titov. Information-Theoretic Probing with Minimum Description Length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, November 2020. Association for Computational Linguistics. URL https://www.aclweb.org/a… ↗ our card
  150. Alex Wang and Kyunghyun Cho. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the workshop on methods for optimizing and evaluating neural language generation, pages 30–36, Minneapolis, Minnesota, June 2019. Association for Computational Linguistic… our card
  151. Alex Warstadt and Samuel R. Bowman. Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments. arXiv preprint arXiv:1901.03438, 2019. — [arXiv:1901.03438](https://arxiv.org/abs/1901.03438) find
  152. Alex Warstadt and Samuel R Bowman. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Conference of the Cognitive Science Society., 2020. find
  153. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. find
  154. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392, 2020a. doi: 10.1162/tacl a 00321. URL https://doi.org/10.… find
  155. Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman. Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republ… find
  156. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do RNN Language Models Learn about Filler–Gap Dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, 2018. our card
  157. Ethan Wilcox, Pranali Vani, and Roger Levy. A Targeted Assessment of Incremental Processing in Neural Language Models and Humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol… find
  158. Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697– 4708, 2020. find
  159. Colin Wilson. Learning phonology with substantive bias: An experimental and computational study of velar palatalization. Cognitive science, 30(5):945–982, 2006. Publisher: Wiley Online Library. find
  160. Yuan Yang and Steven T. Piantadosi. One model for the learning of language. Proceedings of the National Academy of Sciences, 119(5):e2021865119, February 2022. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.2021865119. URL https://pnas.org/doi/full/10.1073/pnas.2021865119. — [doi:10.1073/pnas.2021865119](https://doi.org/10.1073/pnas.2021865119) ↗
  161. Tian Yun, Chen Sun, and Ellie Pavlick. Does Vision-and-Language Pretraining Improve Lexical Grounding? In Proceedings of EMNLP, September 2021. arXiv: 2109.10246. find
  162. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 93–104, Brussels, Belgium, 2018. Association for Computational Lingu… find
  163. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. When Do You Need Billions of Words of Pretraining Data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/warstadt-2022-artificial-neural-networks-tell-us/