Elamur Research⌕
☰
home / papers / 2023 / Lexical Semantics with Large Language Models: A Case Study of English “break”

Lexical Semantics with Large Language Models: A Case Study of English “break”

Paper · 2023 · Erika Petersen and Christopher Potts

Summary

The paper asks whether large language models can serve as research instruments for lexical semantics, using the English verb break as a test case. Break has a very wide spread of senses, and those senses interact with argument structure, especially the causative alternation. The authors build on a richly annotated dataset of break uses from Petersen (2020). They train lightweight probe models on contextual representations, mainly from RoBERTa-large, with parallel analyses on BERT and DeBERTa. They also restate existing linguistic proposals as a sentence-level feature space, so that theory and model representations can be compared directly.

The probes show that LLMs recover many known sense distinctions. They also surface new sense combinations that deserve linguistic attention. The authors frame the comparison through three tenets of lexical theory:

  • High dimensionality: LLMs share it with linguistics.
  • Contextual modulation: LLMs share this one too.
  • Discreteness: LLMs reject it.

They argue that this mismatch should push linguists toward a more fluid, usage-based, token-oriented theory of word meaning.

Key points

  • Main model: RoBERTa-large. Appendices repeat the analyses with BERT and DeBERTa variants. Data and code are released at github.com/epetsen/break-llms.
  • Table 2 lists 26 break frames:
    • 13 without particles (shatter, reveal, surpass, tame, make change, etc.).
    • 13 with particles or predicates (break out, break into, break down, break even, etc.).
    • Senses 1–4 alternate causatively, while senses 5–11 are strictly transitive.
  • Taking part in the causative alternation depends on the verb together with its theme argument, not on the verb alone. Attested cases such as "the code broke" suggest that some senses vary gradually on this property.
  • Static word vectors match linguistic theory only on high dimensionality. LLMs match it on both high dimensionality and contextual modulation.
  • The authors name three limitations:
    • The study covers only one English verb.
    • The models are trained on largely uncontrolled data.
    • The last step, linguists inspecting probe-selected examples, is less reproducible than the rest.

Interesting ideas

  • I1. Hand-written linguistic analyses can be turned into binary feature vectors per sentence (transitive, agentive, unaccusative, metaphorical, plus sense flags), giving theory and neural representations a shared format for comparison.
  • I2. Whether a verb alternates should be decided by the verb plus its object noun, because break the vase alternates while break the law or break her fall does not.
  • I3. Particle combinations change argument structure unpredictably: the war broke out behaves as unaccusative, while we broke into the building has an agent.
  • I4. Probes are used to retrieve interesting examples for human inspection, not as final judges, so the models act as discovery tools.
  • I5. A cheap, released pipeline lets the same analysis be rerun whenever a new model appears, which helps track how model bias shifts the conclusions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Verb-class entries could store such feature rows per verb+argument pattern, making sense choice an explicit, inspectable lookup rather than a fixed per-lemma label.
  • I2 (frames): VerbNet-based role assignment for intransitive break-type clauses could check the theme noun before mapping the subject to Patient rather than Agent.
  • I3 (frames): Treat verb+particle as its own frame key with its own subject-role rule, instead of inheriting roles from the bare verb.
  • I4 (learn): Cases where probe predictions and gold annotations disagree could be queued as candidates for self-induced repair rules or for LLM-teacher review.
  • I5 (learn): Keep sense and role probes as fixed regression checks run against each new witness parser or teacher model, alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksLexical semantics & word embeddings · 63 works

Cite

Cite as Erika Petersen and Christopher Potts. 2023. Lexical Semantics with Large Language Models: A Case Study of English “break”. Findings of the Association for Computational Linguistics: EACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-eacl.36

@inproceedings{petersen2023lexical,
  title = {{Lexical Semantics with Large Language Models: A Case Study of English “break”}},
  author = {Erika Petersen and Christopher Potts},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-eacl.36},
  url = {https://doi.org/10.18653/v1/2023.findings-eacl.36}
}

References

56 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Víctor Acedo-Matellán and Jaume Mateu. 2014. From syntax to roots: A syntactic approach to root interpretation. In Artemis Alexiadou, Hagit Borer, and Florian Schäfer, editors, The Syntax of Roots and the Roots of Syntax, pages 14–32. Oxford University Press, Oxford. find
  2. Herbert H. Clark. 1997. Dogmas of understanding. Discourse Processes, 23(3):567–59. ↗
  3. Artemis Alexiadou, Elena Anagnostopoulou, and Florian Schäfer. 2006. The properties of anticausatives crosslinguistically. In Mara Frascarelli, editor, Phases of Interpretation, pages 187–211. Mouton de Gruyter, Berlin. find
  4. Ryan Cotterell, Adam Poliak, Benjamin Van Durme, and Jason Eisner. 2017. Explaining and generalizing skip-gram through exponential family principal component analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 175–181, Valencia, Spain. Association for Computational Linguistics. find
  5. Marianna Apidianaki. 2022. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Computational Linguistics, pages 1–60. ↗
  6. Mark Davies. 2008. The Corpus of Contemporary American English: 450 million words, 1990-present. Available online at http://corpus.byu.edu/coca/. ↗
  7. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86–90, Montreal, Quebec, Canada. Association for Computational Linguistics. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Collin F. Baker and Hiroaki Sato. 2003. The FrameNet data and software. In The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pages 161–164, Sapporo, Japan. Association for Computational Linguistics. find
  10. David Dowty. 1976. Montague grammar and the lexical decomposition of causative verbs. In Barbara H. Partee, editor, Montague Grammar, pages 201–245. Academic Press, New York. ↗
  11. David Dowty. 1979. Word Meaning and Montague Grammar. D. Reidel, Dordrecht. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv:1907.11692. ↗
  12. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics. find
  13. Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021. Analysis and evaluation of language models for word sense disambiguation. Computational Linguistics, 47(2):387–443. ↗
  14. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
  15. Charles Fillmore. 1970. The grammar of hitting and breaking. In R.A. Jacobs and P.S. Rosenbaum, editors, Readings in English Transformational Grammar, pages 120–133. Ginn, Waltham, MA. find
  16. Aina Garí Soler, Marianna Apidianaki, and Alexandre Allauzen. 2019. Word usage similarity estimation with sentence representations and automatic substitutes. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 9–21, Minneapolis, Minnesota. Association for Computational Linguistics. find
  17. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics. ↗
  18. Eric Kellerman. 1978. Giving learners a break: Native language intuitions as a source of predictions about transferability. Working Papers on Bilingualism, 15:60–92. find
  19. Angelika Kratzer. 1996. Severing the external argument from its verb. In Johan Rooryck and Laurie Zaring, editors, Phrase Structure and the Lexicon, pages 109– 137. Kluwer, Dordrecht. ↗
  20. Beth Levin. 2017. The elasticity of verb meaning revisited. In Proceedings of SALT 27, pages 571–599. ↗
  21. Beth Levin and Malka Rappaport Hovav. 1995. Unaccusativity: At the Syntax–Lexical Semantics Interface. MIT Press, Cambridge, MA. find
  22. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems. find
  23. Asifa Majid, James S. Boster, and Melissa Bowerman. 2008. The cross-lingusitic categorization of everyday events: A study of cutting and breaking. Cognition, 109:235–250. find
  24. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems 30, pages 6294–6305. ↗
  25. John P. McCrae, Theodorus Fransen, Sina Ahmadi, Paul Buitelaar, and Koustava Goswami. 2022. Toward an integrative approach for making sense distinctions. Frontiers in Artificial Intelligence, 5:1–18. ↗
  26. Louise McNally and Alexandra Anna Spalek. 2017. ‘Figurative’ uses of verbs and grammar. Unpublished manuscript. find
  27. Louise McNally and Alexandra Anna Spalek. 2022. Grammatically relevant aspects of meaning and verbal polysemy. Linguistics, pages 1–45. ↗
  28. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Leon Bottou, Max Welling, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. ↗
  30. Sathvik Nair, Mahesh Srinivasan, and Stephen Meylan. 2020. Contextualized word embeddings encode aspects of human-like word sense knowledge. In Proceedings of the Workshop on the Cognitive Aspects of the Lexicon, pages 129–141. Association for Computational Linguistics. ↗
  31. Jiangtian Li and Marc F. Joanisse. 2021. Word senses as clusters of meaning modulations: A computational model of polysemy. Cognitive Science: A Multidisciplinary Journal, 45(4):1–30. ↗
  32. Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald. 2021. Deep subjecthood: Higher-order grammatical features in multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2522–2532, Online. Association for Computational Linguistics. find
  33. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. find
  34. Joe Pater. 2019. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95(1):e41–e74. ↗
  35. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  36. David M. Perlmutter. 1978. Impersonal passives and the Unaccusative Hypothesis. In Proceedings of the Annual Meeting of the Berkeley Linguistics Society, 38, pages 157–189. Berkeley Linguistics Society, Linguistic Society of America. ↗
  37. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  38. Erika Petersen. 2020. Break + NP constraints and the causative alternation. Ms., Stanford University. find
  39. Christopher Piñón. 2001. A finer look at the causative-inchoative alternation. In Proceedings of SALT 11, pages 346–364. ↗
  40. James Pustejovsky. 1991. The generative lexicon. Computational Linguistics, 17(4):409–441. find
  41. James Pustejovsky. 1995. The Generative Lexicon. The MIT Press, Cambridge, MA. ↗
  42. James Pustejovsky. 2006. Introduction to Generative Lexicon. Ms., Brandeis. find
  43. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Ms, OpenAI. find
  44. Malka Rappaport Hovav and Beth Levin. 2012. Lexicon uniformity and the causative alternation. In Martin Everaert, Marijana Marelj, and Tal Siloni, editors, The Theta System: Argument Structure at the Interface, pages 150–176. Oxford University Press, Oxford. ↗
  45. John R. Searle. 1980. The background of meaning. In John R. Searle, Ferenc Kiefer, and Manfred Bierwisch, editors, Speech Act Theory and Pragmatics, pages 221–232. D. Reidel, Dordrecht. ↗
  46. Alexandra Anna Spalek. 2012. Putting order into literal and figurative uses of verbs: romper as a case study. Borealis, 1(2):140–167. ↗
  47. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. ↗
  48. Sean Trott and Benjamin Bergen. 2021. RAW-C: relatedness of ambiguous words-in context (A new lexical resource for English). CoRR, abs/2105.13266. ↗
  49. Peter D Turney. 2001. Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In European Conference on Machine Learning, pages 491–502. Springer. find
  50. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86):2579–2605. find
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
  52. Martin Wattenberg, Fernanda Viégas, and Ian Johnson. 2016. How to use t-SNE effectively. Distill, 1(10):e2. ↗
  53. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. ArXiv, abs/1909.10430. ↗
  54. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  55. Josef Ruppenhofer, Michael Ellsworth, Miriam R. L. Petruck, Christopher R. Johnson, and Jan Scheffczyk. 2006. FrameNet II: Extended Theory and Practice. International Computer Science Institute, Berkeley, CA. find
  56. Florian Schäfer. 2008. The Syntax of (Anti-)Causatives: External Arguments in Change-of-State Contexts. John Benjamins, Amsterdam. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/petersen-2023-lexical-semantics-large-language-models/