Elamur Research⌕
☰
home / papers / 2022 / Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations

Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations

Paper · 2022 · Arabella Sinclair, Jaap Jumelet, Willem Zuidema et al.

Summary

Do neural language models represent sentence structure abstractly, apart from the specific words used? The authors test this with structural priming, a method from psycholinguistics: after a model reads a sentence with a given construction, does it assign higher probability to a follow-up sentence built the same way? They propose a new measure, the Priming Effect, computed from in-context probabilities with no weight updates. This differs from the adaptation-based approach of Prasad et al. (2019). For the experiments they build PRIME-LM, a corpus of roughly 1.3M prime–target pairs that controls for lexical overlap, semantic similarity and other confounds. Most Transformer LMs tested show priming for most constructions. Follow-up analyses on GPT2-LARGE show four patterns. The effect weakens with distance and grows with repeated exposure. Shared words and related meaning amplify it. It shrinks when primes make no semantic sense, so the learned structure is not fully independent of meaning. Priming also persists when phrase complexity differs between prime and target, which points to hierarchical rather than purely linear knowledge. The study positions priming as a complement to probing and targeted syntactic evaluation.

Key points

  • Resource: PRIME-LM, about 1.3M prime–target sentence pairs (e.g., prepositional-object vs. double-object datives). Code and data are released at github.com/dmg-illc/prime-lm.
  • Method: The new Priming Effect metric measures structural persistence from preceding context at inference time, without fine-tuning the model between prime and target.
  • Factors studied on GPT2-LARGE:
    • Priming decreases as more material separates prime and target.
    • Priming increases with more exposures to the structure.
    • Lexical overlap and semantic similarity strongly boost it.
    • Implausible primes partly cancel it, arguably more than in humans.
  • Hierarchy: Positive priming survives when phrases differ in complexity across prime and target. This suggests the models encode some hierarchical syntax, not only sequential order.
  • Masked LMs: A preliminary test on BERT, RoBERTa and ALBERT used pseudo-log-likelihood scoring (Salazar et al., 2020) and found biased priming patterns. The authors suggest bidirectional context may reduce reliance on the prime.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓University of AberdeenAberdeen · United Kingdom20 works · 1,385 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Arabella Sinclair, Jaap Jumelet, Willem Zuidema and Raquel Fernández. 2022. Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00504

@article{sinclair2022structural,
  title = {{Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations}},
  author = {Arabella Sinclair, Jaap Jumelet, Willem Zuidema and Raquel Fernández},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00504},
  url = {https://doi.org/10.1162/tacl_a_00504}
}

References

87 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Alishahi, Grzegorz Chrupala, and Tal Linzen. 2019. Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP Workshop. Natural Language Engineering, 25(4):543–557. https://doi.org/10.1017/S135132491900024X Marco Baroni. 2022. On the proper role of linguistically-ori… — [doi:10.1017/s135132491900024x](https://doi.org/10.1017/s135132491900024x) ↗
  2. Sarah Bernolet and Robert J. Hartsuiker. 2010. Does verb bias modulate syntactic priming? Cognition, 114(3):455–461. https://doi.org/10.1016/j.cognition.2009.11.005 — [doi:10.1016/j.cognition.2009.11.005](https://doi.org/10.1016/j.cognition.2009.11.005) ↗
  3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. GitHub repository. https://doi.org/10.18653/v1/2022.bigscience-1.9 ↗
  4. Kathryn Bock. 1986. Syntactic persistence in language production. Cognitive Psychology, 18(3):355–387. https://doi.org/10.1016 /0010-0285(86)90004-6 find
  5. Kathryn Bock. 1989. Closed-class immanence in sentence production. Cognition, 31(2):163–186. https://doi.org/10.1016/0010-0277(89)90 022-X Kathryn Bock, Gary S. Dell, Franklin Chang, and Kristine H. Onishi. 2007. Persistent structural priming from language comprehension to language production. Cogni… — [doi:10.1016/0010-0277(89)90](https://doi.org/10.1016/0010-0277%2889%2990) ↗
  6. Kathryn Bock and Zenzi M. Griffin. 2000. The persistence of structural priming: Transient activation or implicit learning? Journal of Experimental Psychology: General, 129(2):177. https://doi.org/10.1037/0096-3445.129.2 .177 — [doi:10.1037/0096-3445.129.2](https://doi.org/10.1037/0096-3445.129.2) ↗
  7. Gemma Boleda. 2020. Distributional semantics and linguistic theory. Annual Review of Linguistics, 6(1):213–234. https://doi.org/10.1146/annurev-linguistics-011619 -030303 ↗
  8. Holly P. Branigan, Martin J. Pickering, and Alexandra A. Cleland. 1999. Syntactic priming in written production: Evidence for rapid decay. Psychonomic Bulletin & Review, 6(4):635–640. https://doi.org/10.3758/BF03212972 — [doi:10.3758/bf03212972](https://doi.org/10.3758/bf03212972) ↗
  9. Holly P. Branigan, Martin J. Pickering, Janet F. McLean, and Andrew J. Stewart. 2006. The role of local and global syntactic structure in language production: Evidence from syntactic priming. Language and Cognitive Processes, 21(7–8):974–1010. https://doi.org/10 .1080/016909600824609 find
  10. Joan Bresnan, Anna Cueni, Tatiana Nikitina, and R. Harald Baayen. 2007. Predicting the dative alternation. In Cognitive Foundations of Interpretation, pages 69–94. KNAW. find
  11. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, SandhiniAgarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, C… — Language Models are Few-Shot Learners
  12. Franklin Chang, Gary S. Dell, Kathryn Bock, and Zenzi M. Griffin. 2000. Structural priming as implicit learning: A comparison of models of sentence production. Journal of Psycholinguistic Research, 29(2):217–230. https://doi.org/10.1023/A:1005101313330 — [doi:10.1023/a:1005101313330](https://doi.org/10.1023/a:1005101313330) ↗
  13. Noam Chomsky. 1957. Syntactic Structures. De Gruyter Mouton. https://doi.org/10 .1515/9783112316009 find
  14. Alexandra A. Cleland and Martin J. Pickering. 2003. The use of lexical and syntactic information in language production: Evidence from the priming of noun-phrase structure. Journal of Memory and Language, 49(2):214–230. https://doi.org/10.1016/S0749 -596X(03)00060-3 — [doi:10.1016/s0749](https://doi.org/10.1016/s0749) ↗
  15. William G. Cochran. 1977. Sampling Techniques, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York; London. find
  16. Mark Davies. 2009. The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights. International Journal of Corpus Linguistics, 14(2):159–190. https://doi.org/10.1075/ijcl.14.2.02dav — [doi:10.1075/ijcl.14.2.02dav](https://doi.org/10.1075/ijcl.14.2.02dav) ↗
  17. Workshop on Natural Language Generation (ENLG 07), pages 33–40, Saarbrücken, Germany. DFKI GmbH. https://doi.org/10 .3115/1610163.1610170 find
  18. Forrest Davis and Marten van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 396–407, Online. Association for Computational Linguistics. https://doi.org /10… ↗
  19. Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A theoretical analysis of the repetition problem in text generation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh… find
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  21. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui…
  22. Amit Dubey, Frank Keller, and Patrick Sturt. 2008. A probabilistic corpus-based model of syntacticparallelism. Cognition, 109(3):326–344. https://doi.org/10.1016/j.cognition .2008.09.006 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) ↗
  23. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. Syntaxgym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. https://doi.org/10 .1865…
  24. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. https://doi.org/10 .1162/tacl_a_00298
  25. Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271–276, Gothenburg, Sweden. Association for Computati… find
  26. Alex B. Fine and T. Florian Jaeger. 2013. Evidence for implicit learning in syntactic comprehension. Cognitive Science, 37(3):578–591. https://doi.org/10.1111/cogs.12022 — [doi:10.1111/cogs.12022](https://doi.org/10.1111/cogs.12022) ↗
  27. Mary Ellen Foster and Michael White. 2007. Avoiding repetition in generated text. In Proceedings of the Eleventh European Micah B. Goldwater, Marc T. Tomlinson, Catharine H. Echols, and Bradley C. Love. 2011. Structural priming as structure-mapping: Children use analogies from previous utterances to… find
  28. Gries. 2005. Syntactic priming: A corpus-based approach. Journal of Psycholinguistic Research, 34(4):365–399. https://doi.org/10.1007/s10936-005-6139-3 — [doi:10.1007/s10936-005-6139-3](https://doi.org/10.1007/s10936-005-6139-3) ↗
  29. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-H…
  30. Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? Implications for language understanding and data privacy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… find
  31. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4129–4138. Association for Computation…
  32. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744. find
  33. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. https://doi.org/10.1613/jair.1.11196 — [doi:10.1613/jair.1.11196](https://doi.org/10.1613/jair.1.11196) ↗
  34. Iva Ivanova, Holly P. Branigan, Janet F. McLean, Albert Costa, and Martin J. Pickering. 2017. Do you what I say? People reconstruct the syntax of anomalous utterances. Language, Cognition and Neuroscience, 32(2):175–189. https://doi.org/10.1080/23273798 .2016.1236976 — [doi:10.1080/23273798](https://doi.org/10.1080/23273798) ↗
  35. Iva Ivanova, Martin J. Pickering, Holly P. Branigan, Janet F. McLean, and Albert Costa. 2012. The comprehension of anomalous sentences: Evidence from structural priming. Cognition, 122(2):193–209. https://doi.org/10 .1016/j.cognition.2011.10.013 find
  36. T. Florian Jaeger and Neal E. Snider. 2013. Alignment as a consequence of expectation adaptation: Syntactic priming is affected by the prime’s prediction error given both prior and recent experience. Cognition, 127(1):57–83. https://doi.org/10.1016/j.cognition .2012.10.013 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) ↗
  37. Harold Jeffreys. 1961. Theory of Probability, 3rd edition. International series of monographs on physics. Clarendon Press. find
  38. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. CoRR, abs/1602.02410. find
  39. Jaap Jumelet. 2020. diagNNose: A library for neural activation analysis. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 342–350, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.blackboxnlp-1.32 ↗
  40. Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021. Language models use monotonicity to assess NPI licensing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4958–4969, Online. Association for Computational Linguistics.… find
  41. Michael P. Kaschak, Timothy J. Kutta, and John L. Jones. 2011. Structural priming as implicit learning: Cumulative priming effects and individual differences. Psychonomic Bulletin & Review, 18(6):1133–1139. https://doi.org/10.3758/s13423-011-0157-y Robert E. Kass and Adrian E. Raftery. 1995. Bayes… — [doi:10.3758/s13423-011-0157-y](https://doi.org/10.3758/s13423-011-0157-y) ↗
  42. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818. Association for Computational Linguistics. https://doi.or… ↗
  43. Jordan Kodner and Nitish Gupta. 2020. Overestimation of syntactic representation in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1757–1762, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /202… find
  44. Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699. https://doi.org/10.1016/j .cognition.2021.104699 — [doi:10.1016/j](https://doi.org/10.1016/j) find
  45. Yair Lakretz, Germán Kruszewski, Théo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
  46. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the 8th International Conference on Learning Representations (ICLR).
  47. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computatio… find
  48. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. https://doi.org/10.1162/tacl_a_00115 ↗
  49. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  50. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre-trained models. In Proceedings of the 9th International Conference on Learning Representations (ICLR). find
  51. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxG-BERT: BERT meets construction grammar. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 4020–4032. International Committee on Computational Linguistics. https://doi.org… ↗
  52. Kyle Mahowald, Ariel James, Richard Futrell, and Edward Gibson. 2016. A meta-analysis of syntactic priming in language production. Journal of Memory and Language, 91:5–27. https://doi.org/10.1016/j.jml.2016.03 .009 — [doi:10.1016/j.jml.2016.03](https://doi.org/10.1016/j.jml.2016.03) ↗
  53. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.1016/j .jml.2016.03.009 find
  54. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
  55. Inc. Merriam-Webster. 1989. Webster’s Dictionary of English Usage. Springfield, MA: Merriam-Webster. find
  56. Douglas L. Nelson, Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers, 36(3):402–407. https://doi.org/10.3758/BF03195588 ↗
  57. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710–3723, Online. As… find
  58. Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1145–1160, Online. Association for Computati…
  59. Martin J. Pickering and Holly P. Branigan. 1998. The representation of verbs: Evidence from syntactic priming in language production. Journal of Memory and Language, 39(4):633–651. https://doi.org/10.1006/jmla.1998.2592 — [doi:10.1006/jmla.1998.2592](https://doi.org/10.1006/jmla.1998.2592) ↗
  60. Martin J. Pickering and Victor S. Ferreira. 2008. Structural priming: A critical review. Psychological Bulletin, 134(3):427. https://doi.org/10.1037/0033-2909.134.3.427 — [doi:10.1037/0033-2909.134.3.427](https://doi.org/10.1037/0033-2909.134.3.427) ↗
  61. Martin J. Pickering, Janet F. McLean, and Holly P. Branigan. 2013. Persistent structural priming and frequency effects during comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(3):890. https://doi.org/10.1037/a0029181 — [doi:10.1037/a0029181](https://doi.org/10.1037/a0029181) ↗
  62. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Comput… find
  63. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  64. David Reitter and Frank Keller. 2007. Against sequence priming: Evidence from constituents and distituents in corpus data. In Proceedings of the 29th Annual Meeting of the Cognitive Science Society. find
  65. David Reitter, Frank Keller, and Johanna D. Moore. 2011. A computational cognitive model of syntactic priming. Cognitive Science, 35(4):587–637. https://doi.org/10.1111/j.1551-6709.2010.01165.x David Reitter and Johanna D. Moore. 2014. Alignment and task success in spoken dialogue. Journal of Memor… ↗
  66. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl_a_00349 ↗
  67. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /2020.ac…
  68. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (NeurIPS). find
  69. Christoph Scheepers. 2003. Syntactic priming of relative clause attachments: Persistence of structural configuration in sentence production. Cognition, 89(3):179–205. https://doi .org/10.1016/S0010-0277(03)00119-7 — [doi:10.1016/s0010-0277(03)00119-7](https://doi.org/10.1016/s0010-0277%2803%2900119-7) find
  70. Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. 2020. Artificial neural networks accurately predict language processing in the brain. bioRxiv. find
  71. Katrien Segaert, Linda Wheeldon, and Peter Hagoort. 2016. Unifying structural priming effects on syntactic choices and timing of sentence generation. Journal of Memory and Language, 91:59–80. https://doi.org/10 .1016/j.jml.2016.03.011 find
  72. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 28…
  73. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)…
  74. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. https://doi.org/10.18653/v1/P19-1452 ↗
  75. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In Proceedings of the 7th I…
  76. Kristen M. Tooley and Kathryn Bock. 2014. On the parity of structural persistence in language production and comprehension. Cognition, 132(2):101–136. https://doi.org/10 .1016/j.cognition.2014.04.002 find
  77. Jean E. Fox Tree and Paul J. A. Meijer. 1999. Building syntactic structure in speaking. Journal of Psycholinguistic Research, 28(1):71–90. https://doi.org/10.1023/A:1023239604158 — [doi:10.1023/a:1023239604158](https://doi.org/10.1023/a:1023239604158) ↗
  78. Jannis Vamvas and Rico Sennrich. 2021. On the limits of minimal pairs in contrastive evaluation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 58–68, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://do… find
  79. Marten van Schijndel and Tal Linzen. 2018. A neural model of adaptation in reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4704–4710, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1499 ↗
  80. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, pages 5998–6008.
  81. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183–196, Online. Association for Computational Linguistics. https://doi.org /10.18653/v1/2020.emnl… ↗
  82. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, et al. 2019. Investigating BERT’s knowledge of language: Five analysis methods with NPIs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces…
  83. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020, pages 351–352. https://doi.org/10.1162/tacla00321 ↗
  84. Linda Wheeldon and Mark Smith. 2003. Phrase structure priming: A short-lived effect. Language and Cognitive Processes, 18(4):431–442. https://doi.org/10.1080/01690960244000063 — [doi:10.1080/01690960244000063](https://doi.org/10.1080/01690960244000063) ↗
  85. Jennifer C. White, Tiago Pimentel, Naomi Saphra, and Ryan Cotterell. 2021. A non-linear structural probe. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 132–138, Online.Association for Computati… find
  86. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  87. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DI-ALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Ling… — [doi:10.18653/v1](https://doi.org/10.18653/v1) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sinclair-2022-structural-persistence-language-models-priming/