Elamur Research⌕
☰
home / papers / 2025 / Constructions are Revealed in Word Distributions

Constructions are Revealed in Word Distributions

Paper · 2025 · Joshua Rozner, Leonie Weissweiler, Kyle Mahowald et al.

Summary

Construction Grammar holds that form-meaning pairings are learned from exposure to language. This paper asks how much constructional information the distribution over word strings actually carries. Plain corpus counts cannot answer causal "what made this word appear here" questions. The authors therefore use RoBERTa as a computable stand-in for the distribution itself. They do not treat it as a model of the learner.

Building on perturbed masking, they introduce two affinity measures:

  • Global affinity: how strongly the whole context predicts a word.
  • Local affinity: how much masking a second word shifts that prediction, measured with Jensen–Shannon divergence.

The measures recover many English constructions. These include superficially similar "so ADJ that S" patterns that earlier prompting and probing studies reported LLMs could not separate. They also include schematic constructions whose slots take abstract word classes. Qualitative analysis shows the limits: affinity is neither necessary nor sufficient to identify every construction. The authors conclude that distributional statistics are a strong but partial cue for learners. The method also offers linguists a counterfactual extension of collostructional analysis.

Key points

  • Model: RoBERTa, a bidirectional masked LM with open weights, known training data and no instruction tuning. It was deliberately chosen as a weaker, conservative test. Analysis is restricted to single-token words, which the 50k vocabulary makes workable.
  • Hard case: the Epistemic Adjective Phrase, Affective Adjective Phrase and Causal Excess Construction share one surface frame but differ in causal direction. Zhou et al. (2024) found GPT-3.5/4 and Llama2 struggled with them, while affinity methods distinguish them.
  • Schematic constructions: results on NPN and the comparative correlative suggest the distribution encodes the semantic categories that fill abstract slots.
  • Context sensitivity: in examples like "Green Day," affinities between words appear or vanish depending on the surrounding context.
  • Limitations and resources: one model, English only, and no complete inventory of constructions to test against. Code and data are at github.com/jsrozner/cxs_are_revealed (EMNLP 2025, pp. 2116–2138).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations🎓Uppsala UniversityUppsala · Sweden64 works · 13,164 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Joshua Rozner, Leonie Weissweiler, Kyle Mahowald and Cory Shain. 2025. Constructions are Revealed in Word Distributions. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.108

@inproceedings{rozner2025constructions,
  title = {{Constructions are Revealed in Word Distributions}},
  author = {Joshua Rozner, Leonie Weissweiler, Kyle Mahowald and Cory Shain},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.108},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.108}
}

References

88 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations.
  2. Gerry TM Altmann and Jelena Mirković. 2009. Incrementality and prediction in human sentence processing. Cognitive science, 33(4):583–609. find
  3. Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2024. Simple linguistic inferences of large language models (llms): Blind spots and blinds. Preprint, arXiv:2305.14785. — [arXiv:2305.14785](https://arxiv.org/abs/2305.14785) find
  4. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  5. Claire Bonial and Harish Tayyar Madabushi. 2024. A construction grammar corpus of varying schematicity: A dataset for the evaluation of abstractions in language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL…
  6. Joan Bresnan, Ash Asudeh, Ida Toivonen, and Stephen Wechsler. 2015. Lexical-functional syntax. John Wiley & Sons. find
  7. Joan Bybee. 2006. From usage to grammar: The mind’s response to repetition. Language, pages 711–733. find
  8. Bert Cappelle. 2024. Can construction grammar be proven wrong? Elements in Construction Grammar. find
  9. Devin Casenhiser and Adele E Goldberg. 2005. Fast mapping between a phrasal form and meaning. Developmental science, 8(6):500–508. find
  10. Noam Chomsky. 1995. The minimalist program. Cambridge, MA. find
  11. Gabriella Chronis, Kyle Mahowald, and Katrin Erk. 2023. A method for studying semantic construal in grammatical constructions with interpretable contextual embedding spaces. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 242–… find
  12. William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective. Oxford University Press, USA. find
  13. William Croft and D Alan Cruse. 2004. Cognitive linguistics. Cambridge University Press. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  15. Holger Diessel. 2004. The acquisition of complex sentences. Cambridge University Press. find
  16. Holger Diessel. 2019. The grammar network. Cambridge University Press. find
  17. Jonathan Dunn. 2017. Computational learning of construction grammars. Language and cognition, 9(2):254–292. find
  18. Jonathan Dunn. 2019. Frequency vs. association for constraint selection in usage-based construction grammar. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 117–128, Minneapolis, Minnesota. Association for Computational Linguistics. find
  19. Jonathan Dunn. 2024. Computational construction grammar: A usage-based approach. Elements in Cognitive Linguistics. find
  20. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  21. M. Teresa Espinal and Jaume Mateu. 2019. Idioms and phraseology. In Oxford research encyclopedia of linguistics. Oxford University Press. find
  22. Tom Fawcett. 2006. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874. find
  23. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386.
  24. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. find
  25. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. As…
  26. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7324–7338. PMLR. find
  27. Adele E Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press. find
  28. Adele E Goldberg. 2003. Constructions: A new theoretical approach to language. Trends in cognitive sciences, 7(5):219–224. find
  29. Adele E Goldberg. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford University Press, USA. find
  30. Rui Feng, Congcong Yang, and Yunhua Qu. 2022. A word embedding model for analyzing patterns and their distributional semantics. Journal of Quantitative Linguistics, 29(1):80–105. find
  31. Adele E. Goldberg. 2013. Constructionist approaches. In Thomas Hoffmann and Graeme Trousdale, editors, The Oxford handbook of construction grammar, chapter 2. Oxford University Press. find
  32. Charles J Fillmore. 1986. Varieties of conditional sentences. In Eastern States Conference on Linguistics, volume 3, pages 163–182. find
  33. Adele E. Goldberg. 2024. Usage-based constructionist approaches and large language models. Constructions and Frames, 16(2):220–254. find
  34. Charles J Fillmore. 1988. The mechanisms of “construction grammar". In Annual Meeting of the Berkeley Linguistics Society, volume 14, pages 35–55. find
  35. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association.
  36. Michael C Frank. 2023. Bridging the data gap between children and large language models. Trends in Cognitive Sciences. find
  37. Lyn Frazier and Janet Dean Fodor. 1978. The sausage machine: A new two-stage parsing model. Cognition, 6(4):291–325. find
  38. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, page 1–98. find
  39. Richard Futrell, Peng Qian, Edward Gibson, Evelina Fedorenko, and Idan Blank. 2019. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the fifth international conference on dependency linguistics (depling, syntaxfest 2019), pages 3–13. find
  40. Robert Hawkins, Takateru Yamakoshi, Thomas Griffiths, and Adele Goldberg. 2020. Investigating representations of verb bias in neural language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4653–4663, Online. Association for Computatio… find
  41. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2025. Investigating idiomaticity in word representations. Computational Linguistics, 51:505–555.
  42. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… find
  43. Martin Hilpert. 2014. Collostructional analysis. Corpus methods for semantics: Quantitative studies in polysemy and synonymy, 43:391. find
  44. Thomas Hoffmann. 2022. Construction grammar. Cambridge University Press. find
  45. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python. find
  46. Jacob Louis Hoover, Wenyu Du, Alessandro Sordoni, and Timothy J. O’Donnell. 2021. Linguistic dependencies and statistical dependence. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2941–2963, Online and Punta Cana, Dominican Republic. Association for… find
  47. Ray Jackendoff. 2008. ’construction after construction’and its theoretical challenges. Language, pages 8–28. find
  48. Paul Kay and Ivan A Sag. 2012. Cleaning up the big mess: Discontinuous dependencies and complex determiners. In Sign-based construction grammar, chapter 5, pages 229–256. Citeseer. find
  49. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computatio… find
  50. Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  51. Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling.
  52. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  53. Benjamin Lyngfelt, Linnéa Bäckström, Lars Borin, Anna Ehrlemark, and Rudolf Rydstedt. 2018. Constructicography at work: Theory meets practice in the swedish constructicon. In Constructicography, pages 41–106. John Benjamins. find
  54. Kyle Mahowald. 2023. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 265– 273, Dubrovnik, Croatia. Association for Computa… find
  55. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computationa…
  56. R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. 2024. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121. find
  57. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29). find
  58. Raphaël Millière. 2024. Language models as models of language. Preprint, arXiv:2408.07144.
  59. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929, Miami, Florida, USA. Association for Computational Linguisti…
  60. Geoffrey Nunberg, Ivan A Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491–538. find
  61. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  62. Carl Pollard and Ivan A Sag. 1994. Head-driven phrase structure grammar. University of Chicago Press. find
  63. Christopher Potts. 2023. Characterizing english preposing in pp constructions. Journal of Linguistics, pages 1–39. find
  64. Giulia Rambelli, Emmanuele Chersoni, Marco S. G. Senaldi, Philippe Blache, and Alessandro Lenci. 2023. Are frequent phrases directly retrieved like idioms? an investigation with self-paced reading and language models. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 87–…
  65. Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, and Harish Tayyar Madabushi. 2025. Assessing language comprehension in large language models using construction grammar. CoRR, abs/2501.04661. — [arXiv:2501.04661](https://arxiv.org/abs/2501.04661) find
  66. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  67. Nathaniel J Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  68. Michaela Socolof, Jackie Cheung, Michael Wagner, and Timothy O’Donnell. 2022. Characterizing idioms: Conventionality and contingency. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4024–4037, Dublin, Ireland. Association for… find
  69. Lotte Sommerer and Andreas Baumann. 2021. Of absent mothers, strong sisters and peculiar daughters: The constructional network of English NPN constructions. Cognitive Linguistics, 32(1):97–131. find
  70. Mark Steedman. 2001. The syntactic process. MIT press. find
  71. Anatol Stefanowitsch and Stefan Th. Gries. 2003. Collostructions: Investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8(2):209–243. find
  72. Anatol Stefanowitsch and Stefan Th. Gries. 2005. Co-varying collexemes. Corpus Linguistics and Linguistic Theory, 1(1):1–43. find
  73. Michael K Tanenhaus, Michael J Spivey-Knowlton, Kathleen M Eberhard, and Julie C Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217):1632– 1634. find
  74. John R Taylor. 2004. Why construction grammar is radical. Annual Review of Cognitive Linguistics, 2(1):321–348. find
  75. Michael Tomasello. 2005. Constructing a language: A usage-based theory of language acquisition. Harvard university press. find
  76. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  77. Tim Veenboer and Jelke Bloem. 2023. Using collostructional analysis to evaluate BERT’s representation of linguistic constructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12937–12951, Toronto, Canada. Association for Computational Linguistics. find
  78. Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, and Hinrich Schütze. 2023. Construction grammar provides unique insight into neural language models. In Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023), pages…
  79. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing…
  80. Leonie Weissweiler, Abdullatif Köksal, and Hinrich Schütze. 2024. Hybrid human-LLM corpus construction and LLM evaluation for rare linguistic phenomena. Preprint, arXiv:2403.06965.
  81. Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, Online. Association for Computational Linguistics.
  82. Stefanie Wulff. 2008. Rethinking idiomaticity. Rethinking Idiomaticity, pages 20–34. find
  83. Stefanie Wulff. 2013. 274 words and idioms. In The Oxford Handbook of Construction Grammar. Oxford University Press. find
  84. Lvxiaowei Xu, Zhilin Gong, Jianhua Dai, Tianxiang Wang, Ming Cai, and Jiawei Peng. 2024. CoELM: Construction-enhanced language modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10061–10081, Bangkok, Thailand. Associati… find
  85. Ziheng Zeng and Suma Bhat. 2021. Idiomatic expression identification using semantic compatibility. Transactions of the Association for Computational Linguistics, 9:1546–1562.
  86. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2025. A survey of large language models. Prepri…
  87. Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, and Lori Levin. 2024. Constructions are so difficult that Even large language models get them right for the wrong reasons. find
  88. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3804–3811, Torino, Italia. ELRA and ICCL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rozner-2025-constructions-are-revealed-word-distributions/