Elamur Research⌕
☰
home / papers / 2024 / Language Models as Models of Language

Language Models as Models of Language

Paper · 2024 · Raphael Milliere

Summary

This chapter is a review and position piece, not an empirical study. It asks whether neural language models belong in theoretical linguistics as accounts of how humans acquire and represent grammar, or whether they are only engineering tools. The paper releases no corpus, treebank or model and surveys existing research instead.

It first traces statistical NLP from Shannon's information theory, through distributional semantics, to Transformer models. It then sorts the evidence on syntax into three groups: behavioural tests on targeted constructions, probing of internal representations, and causal interventions. Millière reads this literature as showing real sensitivity to hierarchical structure, even in models trained on child-scale input.

He argues against dismissing models via the competence/performance distinction. Carefully controlled training setups can turn models into idealized learners that offer how-possibly explanations relevant to poverty-of-stimulus and nativism debates. He also flags a limit: reproducing correct outputs is not the same as explaining cross-linguistic constraints.

For NLP practitioners, the chapter clarifies when experiments with language models can count as linguistic evidence.

Key points

  • Publication details: forthcoming chapter in an Oxford handbook volume devoted to philosophical questions about linguistics (eds. Nefdt, Dupre, Stanton; Oxford University Press). The author is at Macquarie University; arXiv v1 is dated 13 Aug 2024.
  • Historical survey: covers Shannon's next-letter prediction experiments (1951), Weaver's machine-translation memo (1949), and the rule-based systems SHRDLU and LUNAR. It also discusses Harris's distributional hypothesis and Osgood's semantic differential, whose dimensions were evaluation, potency and activity.
  • Taxonomy of methods:
    • behavioural studies: targeted syntactic tasks, compositionality and recursion;
    • probing: diagnostic and parameter-free, with a discussion of methodological pitfalls;
    • interventional studies: counterfactual interventions and mechanistic interpretability.
  • Theoretical analysis (Section 4): considers language models as models of performance and of competence, examines in-principle learnability claims, and treats models as model learners and as scientific models. The how-possibly explanation framing draws on Bokulich (2014).
  • Caveats: the expletive-subject example ("It's raining") is explained in generative theory through case theory and the EPP, and a model that merely produces such sentences explains nothing. The supplied text reports no quantitative benchmarks and no UD/LAS-style resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Raphael Milliere. 2024. Language Models as Models of Language. arXiv:2408.07144.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.07144 · arXiv 2408.07144

@misc{milliere2024language,
  title = {{Language Models as Models of Language}},
  author = {Raphael Milliere},
  year = {2024},
  doi = {10.48550/arxiv.2408.07144},
  eprint = {2408.07144},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.07144}
}

References

242 references, 50 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In International Conference on Learning Representations, November 2016.
  2. Ekin Akyurek and Jacob Andreas. LexSym: Compositionality as Lexical Symmetry. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 639–657, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/202… find
  3. Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. Learning to Recombine and Resample Data For Compositional Generalization. In International Conference on Learning Representations, October 2020. find
  4. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, November 2018. 16 I am grateful to Emma Borg, Gabe Dupre, Chris Hill, Barry C. Smith, and Charles Rathkopf for comments on previous versions of this chapter. find
  5. Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Context-Free Grammar, May 2023. find
  6. Ben Ambridge and Liam Blything. Large language models are better than theoretical linguists at theoretical linguistics. Theoretical Linguistics, 50(1-2):33–48, June 2024. ISSN 1613-4060. doi: 10.1515/tl-2024-2002. — [doi:10.1515/tl-2024-2002](https://doi.org/10.1515/tl-2024-2002) find
  7. Jacob Andreas. Good-Enough Compositional Data Augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556–7566, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.676.
  8. David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for Constituency Structure in Neural Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6738–6757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis… find
  9. Marco Baroni. On the Proper Role of Linguistically Oriented Deep Net Analysis in Linguistic Theorising. In Algebraic Structures in Natural Language. CRC Press, 2022. ISBN 978-1-00-320538-8. find
  10. Marco Baroni, Raffaella Bernardi, and Roberto Zamparelli. Frege in Space: A Program for Composition Distributional Semantics. Linguistic Issues in Language Technology, 9, 2014. find
  11. Gašper Beguš, Maksymilian Dąbkowski, and Ryan Rhodes. Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs, May 2023. find
  12. Gašper Beguš, Thomas Lu, and Zili Wang. Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks, July 2024. find
  13. Lisa Beinborn and Nora Hollenstein. Cognitive Plausibility in Natural Language Processing. Synthesis Lectures on Human Language Technologies. Springer International Publishing, Cham, 2024. ISBN 978-3-031-43259-0 978-3-031-43260-6. doi: 10.1007/978-3-031-43260-6.
  14. Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1):207–219, April 2022. ISSN 0891-2017. doi: 10.1162/coli_a_00422.
  15. Islam Beltagy, Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. Montague Meets Markov: Deep Semantics with Probabilistic Logical Form. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Se… find
  16. Emily M. Bender and Alexander Koller. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online, July 2020. Association for Computational Linguistics. doi: 10.18653…
  17. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000. find
  18. Elika Bergelson, Marisa Casillas, Melanie Soderstrom, Amanda Seidl, Anne S. Warlaumont, and Andrei Amatuni. What Do North American Babies Hear? A large-scale cross-corpus analysis. Developmental Science, 22(1):e12724, 2019. ISSN 1467-7687. doi: 10.1111/desc.12724. — [doi:10.1111/desc.12724](https://doi.org/10.1111/desc.12724) find
  19. Robert C. Berwick, Paul Pietroski, Beracah Yankama, and Noam Chomsky. Poverty of the Stimulus Revisited. Cognitive Science, 35(7):1207–1242, 2011. ISSN 1551-6709. doi: 10.1111/j.1551-6709.2011.01189.x. — [doi:10.1111/j.1551-6709.2011.01189.x](https://doi.org/10.1111/j.1551-6709.2011.01189.x) find
  20. Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions, July 2023. find
  21. Idan A. Blank. What are large language models supposed to model? Trends in Cognitive Sciences, 27(11):987–989, November 2023. ISSN 1364-6613, 1879-307X. doi: 10.1016/j.tics.2023.08.006. — [doi:10.1016/j.tics.2023.08.006](https://doi.org/10.1016/j.tics.2023.08.006) find
  22. Alisa Bokulich. How the Tiger Bush Got its Stripes: ‘How Possibly’ vs. ‘How Actually’ Model Explanations. The Monist, 97(3):321–338, July 2014. ISSN 0026-9662. doi: 10.5840/monist201497 321. — [doi:10.5840/monist201497](https://doi.org/10.5840/monist201497) find
  23. Gemma Boleda. Distributional Semantics and Linguistic Theory. Annual Review of Linguistics, 6(1): 213–234, 2020. doi: 10.1146/annurev-linguistics-011619-030303. find
  24. Gemma Boleda and Aurélie Herbelot. Formal distributional semantics: Introduction to the special issue. Computational Linguistics, 42(4):619–635, December 2016. ISSN 0891-2017. doi: 10.1162/ COLI_a_00261. find
  25. Gemma Boleda, Marco Baroni, The Nghia Pham, and Louise McNally. Intensionality was only alleged: On adjective-noun composition in distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, pages 35–46, Potsdam, Germany, March 2… find
  26. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  27. Cameron Buckner. Morgan’s Canon, meet Hume’s Dictum: Avoiding anthropofabulation in cross-species comparisons. Biology & Philosophy, 28(5):853–871, September 2013. ISSN 1572-8404. doi: 10.1007/s10539-013-9376-0. — [doi:10.1007/s10539-013-9376-0](https://doi.org/10.1007/s10539-013-9376-0) find
  28. Cameron Buckner. Deep learning: A philosophical introduction. Philosophy Compass, 14(10):e12625, 2019. ISSN 1747-9991. doi: 10.1111/phc3.12625. — [doi:10.1111/phc3.12625](https://doi.org/10.1111/phc3.12625) find
  29. Cameron Buckner. Black Boxes or Unflattering Mirrors? Comparative Bias in the Science of Machine Behaviour. The British Journal for the Philosophy of Science, pages 000–000, April 2021. ISSN 0007-0882. doi: 10.1086/714960. — [doi:10.1086/714960](https://doi.org/10.1086/714960) find
  30. Andrew Carnie. Syntax: A Generative Introduction. John Wiley & Sons, April 2021. ISBN 978-1-119-56923-7. find
  31. Lucas Georges Gabriel Charpentier and David Samuel. Not all layers are equally as important: Every Layer Counts BERT. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell, editors… find
  32. Nick Chater, Alexander Clark, John A. Goldsmith, and Amy Perfors. Empiricism and Language Learnability. OUP Oxford, July 2015. ISBN 978-0-19-105358-0. find
  33. Rui Chaves. What Don’t RNN Language Models Learn About Filler-Gap Dependencies? In Proceedings of the Society for Computation in Linguistics 2020, pages 1–11, New York, New York, January 2020. Association for Computational Linguistics. find
  34. Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs, September 2023. find
  35. Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov, Irina Proskurina, Laida Kushnareva, Ekaterina Artemova, Serguei Barannikov, Irina Piontkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. Acceptability Judgements via Examining the Topology of Attention Maps, October 2022. find
  36. Noam Chomsky. Syntactic Structures. Mouton, 1957. find
  37. Noam Chomsky. On certain formal properties of grammars. Information and Control, 2(2):137–167, June 1959. ISSN 0019-9958. doi: 10.1016/S0019-9958(59)90362-6. — [doi:10.1016/s0019-9958(59)90362-6](https://doi.org/10.1016/s0019-9958%2859%2990362-6) find
  38. Noam Chomsky. Aspects of the Theory of Syntax. Cambridge, MA, USA: MIT Press, 1965. find
  39. Noam Chomsky. The Minimalist Program. MIT Press, 1995. find
  40. Noam Chomsky, Ian Roberts, and Jeffrey Watumull. Noam Chomsky: The False Promise of ChatGPT. The New York Times, March 2023. ISSN 0362-4331. find
  41. Shammur Absar Chowdhury and Roberto Zamparelli. RNN Simulations of Grammaticality Judgments on Long-distance Dependencies. In Proceedings of the 27th International Conference on Computational Linguistics, pages 133–144, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistic… find
  42. Morten H. Christiansen and Nick Chater. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science, 23(2):157–205, April 1999. ISSN 0364-0213. doi: 10.1016/S0364-0213(99)00003-8. — [doi:10.1016/s0364-0213(99)00003-8](https://doi.org/10.1016/s0364-0213%2899%2900003-8) find
  43. Morten H. Christiansen and Nick Chater. Creating Language: Integrating Evolution, Acquisition, and Processing. MIT Press, March 2016. ISBN 978-0-262-03431-9. find
  44. Grzegorz Chrupała and Afra Alishahi. Correlating Neural and Symbolic Representations of Language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1… find
  45. Radoslaw M. Cichy and Daniel Kaiser. Deep Neural Networks as Scientific Models. Trends in Cognitive Sciences, 23(4):305–317, April 2019. ISSN 1364-6613, 1879-307X. doi: 10.1016/j.tics.2019.01.009. — [doi:10.1016/j.tics.2019.01.009](https://doi.org/10.1016/j.tics.2019.01.009) find
  46. Alexander Clark. Learnability. In The Handbook of Language Emergence, chapter 17, pages 377–395. John Wiley & Sons, Ltd, 2015. ISBN 978-1-118-34613-6. doi: 10.1002/9781118346136.ch17. — [doi:10.1002/9781118346136.ch17](https://doi.org/10.1002/9781118346136.ch17) find
  47. Alexander Clark and Shalom Lappin. Linguistic Nativism and the Poverty of the Stimulus. John Wiley & Sons, December 2010. ISBN 978-1-4443-9055-1. find
  48. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What Does BERT Look at? An Analysis of BERT’s Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy, August 2019. Association for Compu… find
  49. Joseph Coffey, Okko Räsänen, Camila Scaff, and Alejandrina Cristia. The difficulty and importance of estimating the lower and upper bounds of infant speech exposure, June 2024. find
  50. Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-Learning to Compositionally Generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3… find
  51. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers… — What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties
  52. Louise Connell and Dermot Lynott. What Can Language Models Tell Us About Human Cognition? Current Directions in Psychological Science, 33(3):181–189, June 2024. ISSN 0963-7214. doi: 10.1177/09637214241242746. — [doi:10.1177/09637214241242746](https://doi.org/10.1177/09637214241242746) find
  53. Pablo Contreras Kallens, Ross Deans Kristensen-McLachlan, and Morten H. Christiansen. Large Language Models Demonstrate the Potential of Statistical Learning in Language. Cognitive Science, 47(3):e13256, 2023. ISSN 1551-6709. doi: 10.1111/cogs.13256. — [doi:10.1111/cogs.13256](https://doi.org/10.1111/cogs.13256) find
  54. Hannah P. Cowley, Mandy Natter, Karla Gray-Roncal, Rebecca E. Rhodes, Erik C. Johnson, Nathan Drenkow, Timothy M. Shead, Frances S. Chance, Brock Wester, and William Gray-Roncal. A framework for rigorous evaluation of human performance in human and machine learning comparison studies. Scientific Rep… — [doi:10.1038/s41598-022-08078-3](https://doi.org/10.1038/s41598-022-08078-3) find
  55. Alejandrina Cristia, Emmanuel Dupoux, Michael Gurven, and Jonathan Stieglitz. Child-Directed Speech Is Infrequent in a Forager-Farmer Population: A Time Allocation Study. Child Development, 90(3):759–773, May 2019. ISSN 0009-3920. doi: 10.1111/cdev.12974. — [doi:10.1111/cdev.12974](https://doi.org/10.1111/cdev.12974) find
  56. Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 619–634, Online and Punta Cana, Dominican Republic, Nove… find
  57. Peter W. Culicover and Ray Jackendoff. Simpler Syntax. OUP Oxford, July 2005. ISBN 978-0-19-153380-8. find
  58. Maksymilian Dąbkowski and Gašper Beguš. Large language models and (non-)linguistic recursion, June 2023. find
  59. Iria de-Dios-Flores and Marcos Garcia. A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time, June 2022. find
  60. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990. ISSN 1097-4571. doi: 10.1002/(SICI)1097-4571(199009)41:6\<391::AID-ASI1\>3.0.CO;2-9. — [doi:10.1002/(sici)1097-4571(199009)41:6](https://doi.org/10.1002/%28sici%291097-4571%28199009%2941:6) find
  61. Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural Networks and the Chomsky Hierarchy. In The Eleventh International Conference on Learning Representations, September 2022.
  62. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, December 2023. doi: 10.1073/pnas.2309583120. — [doi:10.1073/pnas.2309583120](https://doi.org/10.1073/pnas.2309583120) find
  63. Vittoria Dentella, Fritz Guenther, and Evelina Leivada. Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans, April 2024. find
  64. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], October 2018.
  65. Adrien Doerig, Rowan P. Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W. Lindsay, Konrad P. Kording, Talia Konkle, Marcel A. J. van Gerven, Nikolaus Kriegeskorte, and Tim C. Kietzmann. The neuroconnectionist research programme. Nature Reviews Neuroscience, 24(7): 431–450, July 2023.… — [doi:10.1038/s41583-023-00705-w](https://doi.org/10.1038/s41583-023-00705-w) find
  66. Lucia Donatelli and Alexander Koller. Compositionality in Computational Linguistics. Annual Review of Linguistics, 9(1):463–481, 2023. doi: 10.1146/annurev-linguistics-030521-044439. — [doi:10.1146/annurev-linguistics-030521-044439](https://doi.org/10.1146/annurev-linguistics-030521-044439) find
  67. Gabe Dupre. Empiricism, syntax, and ontogeny. Philosophical Psychology, 34(7):1011–1046, October 2021a. ISSN 0951-5089. doi: 10.1080/09515089.2021.1937591. — [doi:10.1080/09515089.2021.1937591](https://doi.org/10.1080/09515089.2021.1937591) find
  68. Gabe Dupre. (What) Can Deep Learning Contribute to Theoretical Linguistics? Minds and Machines, 31(4):617–635, December 2021b. ISSN 1572-8641. doi: 10.1007/s11023-021-09571-w. — [doi:10.1007/s11023-021-09571-w](https://doi.org/10.1007/s11023-021-09571-w) find
  69. Paul Egré. Explanation in Linguistics. Philosophy Compass, 10(7):451–462, 2015. ISSN 1747-9991. doi: 10.1111/phc3.12225. — [doi:10.1111/phc3.12225](https://doi.org/10.1111/phc3.12225) find
  70. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, March 2021. ISSN 2307-387X. doi: 10.1162/tacl_a_00359.
  71. Ronen Eldan and Yuanzhi Li. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, May 2023.
  72. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… find
  73. Jeffrey L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2):195–225, September 1991. ISSN 1573-0565. doi: 10.1007/BF00114844. — [doi:10.1007/bf00114844](https://doi.org/10.1007/bf00114844) find
  74. Jeffrey L. Elman, Elizabeth A. Bates, Mark H. Johnson, and Annette Karmiloff-Smith. Rethinking Innateness: A Connectionist Perspective on Development. Rethinking Innateness: A Connectionist Perspective on Development. The MIT Press, Cambridge, MA, US, 1996. ISBN 978-0-262-05052-4. find
  75. Katrin Erk. Towards a semantics for distributional representations. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, pages 95–106, Potsdam, Germany, March 2013. Association for Computational Linguistics. find
  76. Katrin Erk. The Probabilistic Turn in Semantics and Pragmatics. Annual Review of Linguistics, 8 (1):101–121, 2022. doi: 10.1146/annurev-linguistics-031120-015515. — [doi:10.1146/annurev-linguistics-031120-015515](https://doi.org/10.1146/annurev-linguistics-031120-015515) find
  77. Linnea Evanson, Yair Lakretz, and Jean-Rémi King. Language acquisition: Do children and language models follow similar learning stages?, June 2023.
  78. Martin B. H. Everaert, Marinus A. C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. Structures, Not Strings: Linguistics as Part of the Cognitive Sciences. Trends in Cognitive Sciences, 19(12):729–743, December 2015. ISSN 1364-6613. doi: 10.1016/j.tics.2015.09.008. — [doi:10.1016/j.tics.2015.09.008](https://doi.org/10.1016/j.tics.2015.09.008) find
  79. Chaz Firestone. Performance vs. competence in human–machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562–26571, October 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1905334117. — [doi:10.1073/pnas.1905334117](https://doi.org/10.1073/pnas.1905334117) find
  80. John R. Firth. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis, 1957. find
  81. Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3–71, March 1988. ISSN 0010-0277. doi: 10.1016/0010-0277(88)90031-5. — [doi:10.1016/0010-0277(88)90031-5](https://doi.org/10.1016/0010-0277%2888%2990031-5) find
  82. Birte U. Forstmann and Eric-Jan Wagenmakers. Model-Based Cognitive Neuroscience: A Conceptual Introduction. In Birte U. Forstmann and Eric-Jan Wagenmakers, editors, An Introduction to Model-Based Cognitive Neuroscience, pages 139–156. Springer, New York, NY, 2015. ISBN 978-1-4939-2236-9. doi: 10.100… — [doi:10.1007/978-1-4939-2236-9_7](https://doi.org/10.1007/978-1-4939-2236-9_7) find
  83. Michael C. Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452, August 2023a. ISSN 2731-0574. doi: 10.1038/s44159-023-00211-x. — [doi:10.1038/s44159-023-00211-x](https://doi.org/10.1038/s44159-023-00211-x) find
  84. Michael C. Frank. Bridging the data gap between children and large language models, June 2023b. find
  85. Michael C. Frank. Large language models as models of human cognition, July 2023c. find
  86. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  87. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. SyntaxGym: An Online Platform for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online, July 2020. Association…
  88. David Gil. Riau Indonesian as a Pivotless Language. In Tipologija i Teorija Jazyka, Ot Opisanija k Objasneniju, K 60-Letiju Aleksandra Evgen’evicha Kibrika (Typology and Linguistic Theory, From Description to Explanation, For the 60th Birthday of Aleksandr E. Kibrik), pages 187–211. Jazyki russkoj k… find
  89. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Netw… find
  90. E Mark Gold. Language identification in the limit. Information and Control, 10(5):447–474, May 1967. ISSN 0019-9958. doi: 10.1016/S0019-9958(67)91165-5. find
  91. Yoav Goldberg. Assessing BERT’s Syntactic Abilities, January 2019. — Assessing BERT's Syntactic Abilities
  92. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless Green Recurrent Networks Dream Hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon…
  93. Michael Hahn. Theoretical Limitations of Self-Attention in Neural Sequence Models. Transactions of the Association for Computational Linguistics, 8:156–171, January 2020. ISSN 2307-387X. doi: 10.1162/tacl_a_00306. find
  94. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany, August 2016. Association…
  95. Sophie Hao and Tal Linzen. Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number, October 2023. find
  96. Jacqueline Harding. Operationalising Representation in Natural Language Processing, June 2023. find
  97. Zellig S. Harris. Distributional Structure. Word, 10:146–162, 1954. doi: 10.1080/00437956.1954.1165 9520. find
  98. Marc D. Hauser, Noam Chomsky, and W. Tecumseh Fitch. The Faculty of Language: What Is It, Who Has It, and How Did It Evolve? Science, 298(5598):1569–1579, November 2002. doi: 10.1126/science.298.5598.1569. — [doi:10.1126/science.298.5598.1569](https://doi.org/10.1126/science.298.5598.1569) find
  99. Aurélie Herbelot and Eva Maria Vecchi. Building a shared world: Mapping distributional to model-theoretic semantic spaces. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 22–32, Lisbon, Portugal, September 2015. Association for Computational Linguisti… find
  100. John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China, No… find
  101. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  102. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. Do Attention Heads in BERT Track Syntactic Dependencies?, November 2019. find
  103. Jennifer Hu and Michael C. Frank. Auxiliary task demands mask the capabilities of smaller language models, April 2024.
  104. Jennifer Hu and Roger Levy. Prompt-based methods may underestimate large language models’ linguistic generalizations, May 2023. find
  105. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A Systematic Assessment of Syntactic Generalization in Neural Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online, July 2020. Association for Computati… find
  106. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions, January 2024. find
  107. Yi Ting Huang, Kathryn Leech, and Meredith L. Rowe. Exploring socioeconomic differences in syntactic development through the lens of real-time processing. Cognition, 159:61–75, February 2017. ISSN 0010-0277. doi: 10.1016/j.cognition.2016.11.004. — [doi:10.1016/j.cognition.2016.11.004](https://doi.org/10.1016/j.cognition.2016.11.004) find
  108. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and ’Diagnostic Classifiers’ Reveal How Recurrent and Recursive Neural Networks Process Hierarchical Structure. Journal of Artificial Intelligence Research, 61:907–926, April 2018. ISSN 1076-9757. doi: 10.1613/jair.1.11196. — [doi:10.1613/jair.1.11196](https://doi.org/10.1613/jair.1.11196) find
  109. Janellen Huttenlocher, Marina Vasilyeva, Elina Cymerman, and Susan Levine. Language input and child syntax. Cognitive Psychology, 45(3):337–374, November 2002. ISSN 0010-0285. doi: 10.1016/S0010-0285(02)00500-5. — [doi:10.1016/s0010-0285(02)00500-5](https://doi.org/10.1016/s0010-0285%2802%2900500-5) find
  110. Anna A. Ivanova, John Hewitt, and Noga Zaslavsky. Probing artificial neural networks: Insights from neuroscience, April 2021. find
  111. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P…
  112. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible Language Models, January 2024.
  113. Jerrold J. Katz and Jerry A. Fodor. The Structure of a Semantic Theory. Language, 39(2):170–210, 1963. ISSN 0097-8507. doi: 10.2307/411200. — [doi:10.2307/411200](https://doi.org/10.2307/411200) find
  114. Roni Katzir. Why large language models are poor theories of human linguistic cognition. A reply to Piantadosi (2023)., April 2023. find
  115. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring Compositional Generalization: A Comprehensive Method on Realis… find
  116. Najoung Kim and Tal Linzen. COGS: A Compositional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online, November 2020. Association for Computational Linguistics. doi: 10.1… find
  117. Najoung Kim, Tal Linzen, and Paul Smolensky. Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models, December 2022. find
  118. Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang-goo Lee. Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction. In International Conference on Learning Representations, September 2019. find
  119. Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 61–65, Baltimore, MD, USA, June 2014. Association for Computatio…
  120. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. find
  121. Brenden Lake and Marco Baroni. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In Proceedings of the 35th International Conference on Machine Learning, pages 2873–2882. PMLR, July 2018. find
  122. Brenden M Lake. Compositional generalization through meta sequence-to-sequence learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. find
  123. Brenden M. Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, pages 1–7, October 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06668-3. find
  124. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics, 9:1336–…
  125. Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699, August 2021. ISSN 0010-0277. doi: 10.1016/j.cognition.2021.104 699. — [doi:10.1016/j.cognition.2021.104](https://doi.org/10.1016/j.cognition.2021.104) find
  126. Yair Lakretz, Théo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. Can Transformers Process Recursive Nested Constructions, Like Humans? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3226–3232, Gyeongju, Republic of Korea, October 2022. International Commi… find
  127. Andrew Kyle Lampinen. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans, February 2023.
  128. Thomas K. Landauer and Susan T. Dumais. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240, 1997. ISSN 1939-1471. doi: 10.1037/0033-295X.104.2.211. — [doi:10.1037/0033-295x.104.2.211](https://doi.org/10.1037/0033-295x.104.2.211) find
  129. Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. Probing for the Usage of Grammatical Number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8818–8831, 2022. doi: 10.18653/v 1/2022.acl-long.… find
  130. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, May 2015. ISSN 1476-4687. doi: 10.1038/nature14539. — [doi:10.1038/nature14539](https://doi.org/10.1038/nature14539) find
  131. Soo-Hwan Lee and Sebastian Schuster. Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions. In Proceedings of the Society for Computation in Linguistics 2022, pages 206–211, online, February 2022. Associa…
  132. Evelina Leivada, Vittoria Dentella, and Fritz Günther. Evaluating the Language Abilities of Large Language Models vs. Humans: Three Caveats. Biolinguistics, 18:1–12, April 2024. ISSN 1450-3417. doi: 10.5964/bioling.14391. — [doi:10.5964/bioling.14391](https://doi.org/10.5964/bioling.14391) find
  133. Alessandro Lenci. Distributional Models of Word Meaning. Annual Review of Linguistics, 4(1): 151–171, 2018. doi: 10.1146/annurev-linguistics-030514-125254. — [doi:10.1146/annurev-linguistics-030514-125254](https://doi.org/10.1146/annurev-linguistics-030514-125254) find
  134. Cara Su-Yi Leong and Tal Linzen. Testing learning hypotheses using neural networks by manipulating learning data, July 2024. find
  135. Michael Lepori and R. Thomas McCoy. Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3637–3651, Barcelona, Spain (Online), Decembe… find
  136. Roger Levy. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177, March 2008. ISSN 0010-0277. doi: 10.1016/j.cognition.2007.05.006. — [doi:10.1016/j.cognition.2007.05.006](https://doi.org/10.1016/j.cognition.2007.05.006) find
  137. Bingzhi Li, Guillaume Wisniewski, and Benoit Crabbé. Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4599–4610, Online and Punta Cana, Dominican Republic, Novemb… find
  138. Yongjie Lin, Yi Chern Tan, and Robert Frank. Open Sesame: Getting inside BERT’s Linguistic Knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253, Florence, Italy, August 2019. Association for Computational Linguistics. doi:…
  139. Tal Linzen. What can linguistics and deep learning contribute to each other? Response to Pater. Language, 95(1):e99–e108, 2019. ISSN 1535-0665. doi: 10.1353/lan.2019.0015. — [doi:10.1353/lan.2019.0015](https://doi.org/10.1353/lan.2019.0015) find
  140. Tal Linzen and Marco Baroni. Syntactic Structure from Deep Learning. Annual Review of Linguistics, 7(1):195–212, 2021. doi: 10.1146/annurev-linguistics-032020-051035. find
  141. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4: 521–535, 2016. doi: 10.1162/tacl_a_00115.
  142. Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers Learn Shortcuts to Automata. In The Eleventh International Conference on Learning Representations, September 2022. find
  143. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi…
  144. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692, 2019b.
  145. Gary F. Marcus. Rethinking Eliminative Connectionism. Cognitive Psychology, 37(3):243–282, December 1998. ISSN 0010-0285. doi: 10.1006/cogp.1998.0694. — [doi:10.1006/cogp.1998.0694](https://doi.org/10.1006/cogp.1998.0694) find
  146. David Mareček and Rudolf Rosa. From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 263–275, Florence, Italy, August 2019. Association for Computational Lin… find
  147. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2024. find
  148. Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1151. find
  149. R. Thomas McCoy and Thomas L. Griffiths. Modeling rapid language learning by distilling Bayesian priors into artificial neural networks, May 2023. find
  150. Louise McNally and Gemma Boleda. Conceptual Versus Referential Affordance in Concept Composition. In James A. Hampton and Yoad Winter, editors, Compositionality and Concepts in Linguistics and Psychology, Language, Cognition, and Mind, pages 245–267. Springer International Publishing, Cham, 2017. IS… — [doi:10.1007/978-3-319-45977-6_10](https://doi.org/10.1007/978-3-319-45977-6_10) find
  151. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs], September 2013.
  152. Raphaël Millière. Philosophy of cognitive science in the age of deep learning. WIREs Cognitive Science, n/a(n/a):e1684, 2024. ISSN 1939-5086. doi: 10.1002/wcs.1684. — [doi:10.1002/wcs.1684](https://doi.org/10.1002/wcs.1684) find
  153. Raphaël Millière and Cameron Buckner. A Philosophical Introduction to Language Models – Part I: Continuity With Classic Debates, January 2024. find
  154. Raphaël Millière and Charles Rathkopf. Anthropocentric bias and the possibility of artificial cognition, July 2024. find
  155. Kanishka Misra and Kyle Mahowald. Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs, March 2024.
  156. Jeff Mitchell and Jeffrey Bowers. Priorless Recurrent Networks Learn Curiously. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5147–5158, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.col… find
  157. Dimitri Coelho Mollo and Raphaël Millière. The Vector Grounding Problem, April 2023.
  158. Andrea Moro. Impossible Languages. MIT Press, August 2016. ISBN 978-0-262-33561-4. find
  159. Andrea Moro, Matteo Greco, and Stefano F. Cappa. Large languages, impossible languages and human brains. Cortex, 167:82–85, October 2023. ISSN 0010-9452. doi: 10.1016/j.cortex.2023.07.003. — [doi:10.1016/j.cortex.2023.07.003](https://doi.org/10.1016/j.cortex.2023.07.003) find
  160. Aaron Mueller and Tal Linzen. How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases, May 2023. find
  161. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. Cross-Linguistic Syntactic Evaluation of Word Prediction Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online, July 2020. Association for C…
  162. Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, and Sebastian Schuster. Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1352–1368, Dublin, Ireland, May 202… find
  163. Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Characterizing Intrinsic Compositionality in Transformers with Tree Projections, November 2022. find
  164. Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Grokking of Hierarchical Structure in Vanilla Transformers, May 2023. find
  165. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, September 2022.
  166. Ryan M. Nefdt. The Philosophy of Theoretical Linguistics: A Contemporary Outlook. Cambridge University Press, Cambridge, 2024. ISBN 978-1-316-51425-2. doi: 10.1017/9781009082853. — [doi:10.1017/9781009082853](https://doi.org/10.1017/9781009082853) find
  167. James Nguyen and Roman Frigg. Scientific Representation. Elements in the Philosophy of Science, August 2022. doi: 10.1017/9781009003575. — [doi:10.1017/9781009003575](https://doi.org/10.1017/9781009003575) find
  168. Jingcheng Niu, Wenjie Lu, and Gerald Penn. Does BERT Rediscover a Classical NLP Pipeline? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3143– 3153, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. find
  169. Peter Norvig. On Chomsky and the Two Cultures of Statistical Learning. In Wolfgang Pietsch, Jörg Wernecke, and Maximilian Ott, editors, Berechenbarkeit der Welt? Philosophie und Wissenschaft im Zeitalter von Big Data, pages 61–83. Springer Fachmedien, Wiesbaden, 2017. ISBN 978-3-658-12153-2. doi: 10… — [doi:10.1007/978-3-658-12153-2_3](https://doi.org/10.1007/978-3-658-12153-2_3) find
  170. Byung-Doh Oh and William Schuler. Transformer-Based LM Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens, April 2023. find
  171. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024.001, March 2020. ISSN 2476-0757. doi: 10.23915/distill.00024.001. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) find
  172. Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. Making Transformers Solve Compositional Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3591–3607, Dublin, Ireland, May 2022. Association for Computat… find
  173. OpenAI. GPT-4 Technical Report, March 2023.
  174. Charles E. Osgood. The nature and measurement of meaning. Psychological bulletin, 49(3):197–237, May 1952. ISSN 1939-1455. doi: 10.1037/h0055737. — [doi:10.1037/h0055737](https://doi.org/10.1037/h0055737) find
  175. Isabel Papadimitriou and Dan Jurafsky. Pretrain on just structure: Understanding linguistic inductive biases using transfer learning, April 2023. find
  176. Barbara Hall Partee. Montague Grammar, Mental Representations, and Reality. In Stig Kanger and Sven Ōhman, editors, Philosophy and Grammar: Papers on the Occasion of the Quincentennial of Uppsala University, Synthese Library, pages 59–78. Springer Netherlands, Dordrecht, 1981. ISBN 978-94-009-9012-… — [doi:10.1007/978-94-009-9012-8_5](https://doi.org/10.1007/978-94-009-9012-8_5) find
  177. Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkeld. Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence, May 2024.
  178. Ellie Pavlick. Semantic Structure in Deep Learning. Annual Review of Linguistics, 8(1):447–471, 2022. doi: 10.1146/annurev-linguistics-031120-122924. — [doi:10.1146/annurev-linguistics-031120-122924](https://doi.org/10.1146/annurev-linguistics-031120-122924) find
  179. Lisa Pearl. How Statistical Learning Can Play Well with Universal Grammar. In A Companion to Chomsky, chapter 17, pages 267–286. John Wiley & Sons, Ltd, 2021. ISBN 978-1-119-59873-2. doi: 10.1002/9781119598732.ch17. — [doi:10.1002/9781119598732.ch17](https://doi.org/10.1002/9781119598732.ch17) find
  180. Lisa Pearl. Poverty of the Stimulus Without Tears. Language Learning and Development, 18(4): 415–454, October 2022. ISSN 1547-5441, 1547-3341. doi: 10.1080/15475441.2021.1981908. — [doi:10.1080/15475441.2021.1981908](https://doi.org/10.1080/15475441.2021.1981908) find
  181. Lisa S. Pearl. Modeling syntactic acquisition. In Jon Sprouse, editor, The Oxford Handbook of Experimental Syntax, page 0. Oxford University Press, March 2023. ISBN 978-0-19-879772-2. doi: 10.1093/oxfordhb/9780198797722.013.8. — [doi:10.1093/oxfordhb/9780198797722.013.8](https://doi.org/10.1093/oxfordhb/9780198797722.013.8) find
  182. Steven Piantadosi. Modern language models refute Chomsky’s approach to language, March 2023. find
  183. Steven Piantadosi and Felix Hill. Meaning without reference in large language models, August 2022. find
  184. Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. Pareto Probing: Trading Off Accuracy for Complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3138–3153, Online, November 2020. Association for Computational Linguistics.… find
  185. Steven Pinker and Ray Jackendoff. The faculty of language: What’s special about it? Cognition, 95 (2):201–236, March 2005. ISSN 0010-0277. doi: 10.1016/j.cognition.2004.08.004. — [doi:10.1016/j.cognition.2004.08.004](https://doi.org/10.1016/j.cognition.2004.08.004) find
  186. Steven Pinker and Alan Prince. On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28(1):73–193, March 1988. ISSN 0010-0277. doi: 10.1016/0010-0277(88)90032-7. — [doi:10.1016/0010-0277(88)90032-7](https://doi.org/10.1016/0010-0277%2888%2990032-7) find
  187. Christopher Potts. Characterizing English Preposing in PP constructions, August 2023. find
  188. Geoffrey K. Pullum and Barbara C. Scholz. Empirical assessment of stimulus poverty arguments. The Linguistic Review, 19(1-2):9–50, June 2002. ISSN 1613-3676. doi: 10.1515/tlir.19.1-2.9. — [doi:10.1515/tlir.19.1-2.9](https://doi.org/10.1515/tlir.19.1-2.9) find
  189. Yulu Qin, Wentao Wang, and Brenden Lake. A systematic investigation of learnability from single child linguistic input. Proceedings of the Annual Meeting of the Cognitive Science Society, 46(0), 2024.
  190. Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. Improving Compositional Generalization with Latent Structure and Data Augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:…
  191. Jake Quilty-Dunn, Nicolas Porot, and Eric Mandelbaum. The Best Game in Town: The Re-Emergence of the Language of Thought Hypothesis Across the Cognitive Sciences. Behavioral and Brain Sciences, pages 1–55, January 2023. ISSN 0140-525X, 1469-1825. doi: 10.1017/S0140525X22002849. — [doi:10.1017/s0140525x22002849](https://doi.org/10.1017/s0140525x22002849) find
  192. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Preprint, 2019. find
  193. Alessandro Raganato and Jörg Tiedemann. An Analysis of Encoder Representations in Transformer-Based Machine Translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium, November 2018. Association for Computa…
  194. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. Can LSTM Learn to Capture Agreement? The Case of Basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium, November 2018. Association for Computational Linguis… find
  195. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194–209, Online, November 2021. Ass… find
  196. Tim Räz and Claus Beisbart. The Importance of Understanding Deep Learning. Erkenntnis, August 2022. ISSN 1572-8420. doi: 10.1007/s10670-022-00605-y. — [doi:10.1007/s10670-022-00605-y](https://doi.org/10.1007/s10670-022-00605-y) find
  197. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 8:842–866, 2020. doi: 10.1162/tacl_a_00349.
  198. Rudolf Rosa and David Mareček. Inducing Syntactic Trees from BERT Representations, June 2019. find
  199. John Robert Ross. Constraints on variables in syntax. Technical report, Massachusetts Institute of Technology, September 1967. find
  200. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can AI-Generated Text be Reliably Detected?, June 2023.
  201. G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Communications of the ACM, 18(11):613–620, November 1975. ISSN 0001-0782. doi: 10.1145/361219.361220. find
  202. David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. Trained on 100 million words and still in shape: BERT meets British National Corpus. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1954–1974, Dubrovnik, Croatia, May 2023. Association for Computational Li… find
  203. Carson T. Schütze. Linguistic evidence and grammatical theory. WIREs Cognitive Science, 2(2): 206–221, 2011. ISSN 1939-5086. doi: 10.1002/wcs.102. — [doi:10.1002/wcs.102](https://doi.org/10.1002/wcs.102) find
  204. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Philip Levy. Large-Scale Evidence for Logarithmic Effects of Word Predictability on Reading Time, November 2022. find
  205. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27 (3):379–423, July 1948. ISSN 0005-8580. doi: 10.1002/j.1538-7305.1948.tb01338.x. — [doi:10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) find
  206. C. E. Shannon. Prediction and Entropy of Printed English. Bell System Technical Journal, 30(1): 50–64, January 1951. ISSN 1538-7305. doi: 10.1002/j.1538-7305.1951.tb01366.x. — [doi:10.1002/j.1538-7305.1951.tb01366.x](https://doi.org/10.1002/j.1538-7305.1951.tb01366.x) find
  207. Nicholas Shea. Representation in Cognitive Science. OUP Oxford, October 2018. ISBN 978-0-19-881288-3. find
  208. Xing Shi, Inkit Padhi, and Kevin Knight. Does String-Based Neural MT Learn Source Syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1159.
  209. Anders Søgaard. Understanding models understanding language. Synthese, 200(6):443, October 2022. ISSN 1573-0964. doi: 10.1007/s11229-022-03931-4. — [doi:10.1007/s11229-022-03931-4](https://doi.org/10.1007/s11229-022-03931-4) find
  210. Jon Sprouse. Acceptability judgments and grammaticality, prospects and challenges. In Syntactic Structures after 60 Years, chapter Syntactic Structures after 60 Years, pages 195–224. De Gruyter Mouton, January 2018. ISBN 978-1-5015-0692-5. find
  211. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  212. Emily Sullivan. Understanding from Machine Learning Models. The British Journal for the Philosophy of Science, March 2022. doi: 10.1093/bjps/axz035. — [doi:10.1093/bjps/axz035](https://doi.org/10.1093/bjps/axz035) find
  213. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452.
  214. Michael Tomasello. Constructing a Language. Harvard University Press, June 2009. ISBN 978-0-674-04439-5. find
  215. Greta Tuckute, Nancy Kanwisher, and Evelina Fedorenko. Language in brains, minds, and machines. Annual Review of Neuroscience, 47(Volume 47, 2024):277–301, 2024. ISSN 1545-4126. doi: 10.1146/annurev-neuro-120623-101142. — [doi:10.1146/annurev-neuro-120623-101142](https://doi.org/10.1146/annurev-neuro-120623-101142) find
  216. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sy…
  217. Noortje J. Venhuizen, Petra Hendriks, Matthew W. Crocker, and Harm Brouwer. Distributional formal semantics. Information and Computation, 287:104763, September 2022. ISSN 0890-5401. doi: 10.1016/j.ic.2021.104763. — [doi:10.1016/j.ic.2021.104763](https://doi.org/10.1016/j.ic.2021.104763) find
  218. Philippe Verreault-Julien. How could models possibly provide how-possibly explanations? Studies in History and Philosophy of Science Part A, 73:22–33, February 2019. ISSN 0039-3681. doi: 10.1016/j.shpsa.2018.06.008. — [doi:10.1016/j.shpsa.2018.06.008](https://doi.org/10.1016/j.shpsa.2018.06.008) find
  219. David Vilares, Michalina Strzyz, Anders Søgaard, and Carlos Gómez-Rodríguez. Parsing as Pretraining. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9114–9121, April 2020. ISSN 2374-3468. doi: 10.1609/aaai.v34i05.6446. — [doi:10.1609/aaai.v34i05.6446](https://doi.org/10.1609/aaai.v34i05.6446) find
  220. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 small. In NeurIPS ML Safety Workshop, November 2022. find
  221. Wentao Wang, Wai Keen Vong, Najoung Kim, and Brenden M. Lake. Finding Structure in One Child’s Linguistic Experience. Cognitive Science, 47(6):e13305, 2023. ISSN 1551-6709. doi: 10.1111/cogs.13305. — [doi:10.1111/cogs.13305](https://doi.org/10.1111/cogs.13305) find
  222. Alex Warstadt and Samuel R. Bowman. Can neural networks acquire a structural bias from raw linguistic data? Proceedings of the Annual Meeting of the Cognitive Science Society, January 2020. find
  223. Alex Warstadt and Samuel R. Bowman. What Artificial Neural Networks Can Tell Us about Human Language Acquisition. In Algebraic Structures in Natural Language. CRC Press, 2022. ISBN 978-1-00-320538-8.
  224. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392, July 2020. ISSN 2307-387X. doi: 10.1162/tacl_a_00321.
  225. Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Gotlieb Wilcox, and Chengxu Zhuang. Call for Papers – The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus, January 2023. find
  226. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang, Adina Williams, Ryan Cotterell, and Tal Linzen. Insights from the first BabyLM Challenge: Training sample-efficient language models on a developmentally plausible corpus. Proceedings of the Annual Meeting of the Cogn… find
  227. Warren Weaver. Translation. In William N. Locke and Donald A. Booth, editors, Machine Translation of Languages. MIT Press, Boston, MA, May 1955. ISBN 978-0-262-12002-9. find
  228. Felix A. Wichmann and Robert Geirhos. Are Deep Neural Networks Adequate Behavioral Models of Human Visual Perception? Annual Review of Vision Science, 9(1):null, 2023. doi: 10.1146/annurev-vision-120522-031739. — [doi:10.1146/annurev-vision-120522-031739](https://doi.org/10.1146/annurev-vision-120522-031739) find
  229. Ethan Gotlieb Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do RNN Language Models Learn about Filler–Gap Dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium, November 2018. Associat…
  230. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior, June 2020. find
  231. Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. Using Computational Models to Test Syntactic Learnability. Linguistic Inquiry, pages 1–44, April 2023. ISSN 0024-3892. doi: 10.1162/ ling_a_00491. find
  232. Ethan Gotlieb Wilcox, Michael Hu, Aaron Mueller, Tal Linzen, Alex Warstadt, Leshem Choshen, Chengxu Zhuang, Ryan Cotterell, and Adina Williams. Bigger is not always better: The importance of human-scale language modeling for psycholinguistics, July 2024. find
  233. Terry Winograd. Procedures as a Representation for Data in a Computer Program for Understanding Natural Language. PhD thesis, Massachusetts Institute of Technology, January 1971. find
  234. W. A. Woods. Progress in natural language understanding: An application to lunar geology. In Proceedings of the June 4-8, 1973, National Computer Conference and Exposition, AFIPS ’73, pages 441–450, New York, NY, USA, June 1973. Association for Computing Machinery. ISBN 978-1-4503-7916-8. doi: 10.11… — [doi:10.1145/1499586.1499695](https://doi.org/10.1145/1499586.1499695) find
  235. Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, Online, July 2020. Association for Computational Linguistics. doi: 1…
  236. Takateru Yamakoshi, James L. McClelland, Adele E. Goldberg, and Robert D. Hawkins. Causal interventions expose implicit situation models for commonsense language understanding, June 2023. find
  237. Charles D. Yang. Universal Grammar, statistics or both? Trends in Cognitive Sciences, 8(10): 451–456, October 2004. ISSN 1364-6613, 1879-307X. doi: 10.1016/j.tics.2004.08.006. — [doi:10.1016/j.tics.2004.08.006](https://doi.org/10.1016/j.tics.2004.08.006) find
  238. Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-Attention Networks Can Process Bounded Hierarchical Languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro… find
  239. Aditya Yedetore, Tal Linzen, Robert Frank, and R. Thomas McCoy. How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech, June 2023.
  240. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. When Do You Need Billions of Words of Pretraining Data?, November 2020. find
  241. Haoyu Zhao, Abhishek Panigrahi, Rong Ge, and Sanjeev Arora. Do Transformers Parse while Predicting the Masked Word?, March 2023. find
  242. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks, June 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/milliere-2024-language-models-models-language/