Elamur Research⌕
☰
home / papers / 2023 / Are Frequent Phrases Directly Retrieved like Idioms? An Investigation with Self-Paced Reading and Language Models

Are Frequent Phrases Directly Retrieved like Idioms? An Investigation with Self-Paced Reading and Language Models

Summary

The paper asks whether frequent but fully compositional word sequences are handled like idioms in comprehension, meaning retrieved as stored units rather than assembled word by word. The authors built 48 English verb-determiner-noun items. Each item appears in three versions sharing the same verb: idiomatic (ID, spill the beans), high-frequency compositional (HF, spill the milk) and low-frequency compositional (LF, spill the rice). HF and LF variants were chosen with frequency and LogDice association queries on enTenTen18. Nouns were balanced for frequency and length. Each stimulus followed a context sentence.

Human reading times were collected in a self-paced reading study. The same materials were then scored with Surprisal from several RNN and Transformer language models. Humans read ID and HF phrases at similar speeds, and both faster than LF phrases. The models mirrored this by giving LF items higher Surprisal. The closest match to human behaviour came from the smallest models, consistent with reports of inverse scaling.

The results support usage-based accounts in which frequent multiword sequences are processed more holistically than compositionally.

Key points

  • Stimuli: 48 triplets (ID / HF / LF). The idioms were sourced from Kyriacou et al. (2020) and from two normative collections, those of Bulkes and Tanner (2017) and of Libben and Titone (2008). They were then screened using an Idiom Familiarity survey and a Typical Objects Production study.
  • Matched compositional phrases were drawn from enTenTen18 (about 21.9 billion words) using SketchEngine. The LogDice score controlled verb-noun association.
  • Models tested:
    • GPT2 in four sizes (base, medium, large, xl)
    • BERT-base-uncased
    • T5-base
    • tinyLSTM
    • GRNN
  • The small GPT2 and a 2-layer LSTM reproduced the human pattern exactly: no significant ID–HF difference, and significantly higher Surprisal for LF.
  • Removing the preceding context sentence raised GPT2 Surprisal overall, but the ordering of conditions stayed the same. A Context × Condition interaction appeared only for GPT2-xl (F = 106.49). Data and code are shared on OSF.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.mwe-1.13

@inproceedings{rambelli2023are,
  title = {{Are Frequent Phrases Directly Retrieved like Idioms? An Investigation with Self-Paced Reading and Language Models}},
  author = {Rambelli, Giulia and Chersoni, Emmanuele and Senaldi, Marco S. G. and Blache, Philippe and Lenci, Alessandro},
  booktitle = {Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.mwe-1.13},
  url = {https://doi.org/10.18653/v1/2023.mwe-1.13}
}

References

72 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Verna Dankers, Christopher G Lucas, and Ivan Titov. 2022. Can Transformer be too Compositional? Analysing Idiom Processing in Neural Machine Translation. In Proceedings of ACL. find our card
  2. Ben Ambridge. 2020. Against Stored Abstractions: A Radical Exemplar Model of Language Acquisition. First Language, 40(5-6):509–559. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. find
  4. Inbal Arnon and Neal Snider. 2010. More than Words: Frequency Effects for Multi-word Phrases. Journal of Memory and Language, 62(1):67–82. find
  5. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  6. Colin Bannard and Danielle Matthews. 2008. Stored Word Sequences in Language Learning: The Effect of Familiarity on Children’s Repetition of Four-word Combinations. Psychological Science, 19(3). find
  7. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as Psycholinguistic Subjects: Syntactic State and Grammatical Dependency. arXiv preprint arXiv:1809.01329. find
  8. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1):1– 48. find our card
  9. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021a. Assessing the Representations of Idiomaticity in Vector Models with a Noun Compound Dataset Labeled at Type and Token Levels. In Proceedings of ACL. find
  10. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv preprint arXiv:2303.08112. find our card
  11. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for Idiomaticity in Vector Space Models. In Proceedings of EACL. find our card
  12. Douglas Biber, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 2000. Longman Grammar of Spoken and Written English. Longman London. find
  13. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An Online Platform for Targeted Evaluation of Language Models. In Proceedings of ACL: Demo. find
  14. Nyssa Z. Bulkes and Darren Tanner. 2017. “Going to town”: Large-scale Norming and Statistical Analysis of 870 American English Idioms. Behavior Research Methods, 49(2):772–783. find
  15. Adele E Goldberg. 2003. Constructions: A New Theoretical Approach to Language. Trends in Cognitive Sciences, 7(5):219–224. find
  16. Joan Bybee. 2006. From Usage to Grammar: The Mind’s Response to Repetition. Language, pages 711–733. find
  17. Adele E. Goldberg. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford University Press on Demand. find
  18. Joan L. Bybee. 2010. Language, Usage and Cognition. Cambridge University Press. find
  19. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless Green Recurrent Networks Dream Hierarchically. In Proceedings of NAACL. find our card
  20. Cristina Cacciari and Patrizia Tabossi. 1988. The Comprehension of Idioms. Journal of Memory and Language, 27(6):668–683. find
  21. John Hale. 2001. A Probabilistic Earley Parser as a Psycholinguistic Model. In Proceedings of NAACL. find our card
  22. Won Ik Cho, Emmanuele Chersoni, Yu-Yin Hsu, and Chu-Ren Huang. 2021. Modeling the Influence of Verb Aspect on the Activation of Typical Event Locations with BERT. In Findings of ACL-IJCNLP. find
  23. Ray Jackendoff. 2002. Foundations of Language. Oxford University Press. find
  24. Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlý, and Vít Suchomel. 2013. The tenten corpus family. In 7th International Corpus Linguistics Conference CL, pages 125–127. Lancaster University. find
  25. Noam Chomsky. 1993. A Minimalist Program for Linguistic Theory. The View from Building 20: Essays in Linguistics in Honor of Sylvain Bromberger. find
  26. Kathy Conklin and Norbert Schmitt. 2008. Formulaic Sequences: Are They Processed More Quickly than Nonformulaic Language by Native and Nonnative Speakers? Applied Linguistics, 29(1):72–89. find
  27. Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2023. Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts. In Transfer Learning for Natural Language Processing Workshop, pages 52–62. PMLR. find
  28. Pablo Contreras Kallens and Morten H Christiansen. 2022. Models of Language and Multiword Expressions. Frontiers in Artificial Intelligence, 5:24. find
  29. Jill Jegerski. 2013. Self-paced Reading. In Research Methods in Second Language Psycholinguistics, pages 36–65. Routledge. find
  30. Hajnal Jolsvai, Stewart M McCauley, and Morten H Christiansen. 2020. Meaningfulness Beats Frequency in Multiword Chunk Processing. Cognitive Science, 44(10):e12885. find
  31. Byung-Doh Oh and William Schuler. 2022. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? In Proceedings of EMNLP. find
  32. Adam Kilgarriff, Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý, and Vít Suchomel. 2014. The Sketch Engine: Ten Years On. Lexicography, 1(1):7–36. find
  33. Ludovica Pannitto and Aurélie Herbelot. 2023. CALaMo: A Constructionist Assessment of Language Models. arXiv preprint arXiv:2302.03589. find
  34. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models Are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9. find
  35. Alexandra Kuznetsova, Per B. Brockhoff, and Rune HB Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software, 82(1):1–26. find
  36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. Journal of Machine Learning Research, 21(1):5485–5551. find
  37. Marianna Kyriacou, Kathy Conklin, and Dominic Thompson. 2020. Passivizability of Idioms: Has the Wrong Tree Been Barked Up? Language and speech, 63(2):404–435. find
  38. Roger Levy. 2008. Expectation-based Syntactic Comprehension. Cognition, 106(3):1126–1177. find
  39. Giulia Rambelli, Emmanuele Chersoni, Philippe Blache, Chu-Ren Huang, and Alessandro Lenci. 2019. Distributional Semantics Meets Construction Grammar. Towards a Unified Usage-based Model of Grammar and Meaning. In Proceedings of the ACL Workshop on Designing Meaning Representations. find
  40. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural Reality of Argument Structure Constructions. In Proceedngs of ACL. find
  41. Maya R. Libben and Debra A. Titone. 2008. The Multi-determined Nature of Idiom Processing. Memory & Cognition, 36(6):1103–1121. find
  42. Giulia Rambelli, Emmanuele Chersoni, Philippe Blache, and Alessandro Lenci. 2022. Compositionality as an Analogical Process: Introducing ANNE. In Proceedings of the AACL-IJCNLP Workshop on Cognitive Aspects of the Lexicon. find
  43. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxGBERT: BERT Meets Construction Grammar. In Proceedings of COLING. find
  44. Giulia Rambelli, Emmanuele Chersoni, Alessandro Lenci, Philippe Blache, and Chu-Ren Huang. 2020. Comparing Probabilistic, Distributional and Transformer-based Models on Logical Metonymy Interpretation. In Proceedings of AACL-IJCNLP. find
  45. Alec Marantz. 1995. The Minimalist Program. In The Principles and Parameters Approach to Linguistic Theory, pages 351–382. Blackwell. find
  46. Danny Merkx and Stefan L Frank. 2021. Human Sentence Processing: Recurrence or Attention? In Proceedings of the NAACL Workshop on Cognitive Modeling and Computational Linguistics. find
  47. Keith Rayner and Susan A Duffy. 1986. Lexical Complexity and Fixation Times in Reading: Effects of Word Frequency, Verb Complexity, and Lexical Ambiguity. Memory & Cognition, 14(3):191–201. find
  48. James A Michaelov and Benjamin K Bergen. 2020. How Well Does Surprisal Explain N400 Amplitude under Different Experimental Conditions? In Proceedings of CONLL. find
  49. Erik D. Reichle, Keith Rayner, and Alexander Pollatsek. 2003. The EZ Reader Model of Eye-Movement Control in Reading: Comparisons to Other Models. Behavioral and Brain Sciences, 26(4):445–476. find
  50. James A Michaelov and Benjamin K Bergen. 2022a. Collateral Facilitation in Humans and Language Models. In Proceedings of CONLL. find
  51. Pavel Rychlý. 2008. A Lexicographer-Friendly Association Score. In RASLAN, pages 6–9. find our card
  52. James A Michaelov and Benjamin K Bergen. 2022b. ’Rarely’ a Problem? Language Models Exhibit Inverse Scaling in their Predictions Following ’Few’-type Quantifiers. arXiv preprint arXiv:2212.08700. find
  53. Soo Hyun Ryu and Richard L Lewis. 2021. Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention. In Proceedings of the NAACL Workshop on Cognitive Modeling and Computational Linguistics. find
  54. James A Michaelov, Seana Coulson, and Benjamin K Bergen. 2023. Can Peanuts Fall in Love with Distributional Semantics? arXiv preprint arXiv:2301.08731. — [arXiv:2301.08731](https://arxiv.org/abs/2301.08731) find
  55. Marco S. G. Senaldi, Junyan Wei, Jason W Gullifer, and Debra Titone. 2022. Scratching your Tête over Language-switched Idioms: Evidence from Eye-movement Measures of Reading. Memory & Cognition, 50(6):1230–1256. find
  56. Kanishka Misra. 2022. minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models. arXiv preprint arXiv:2203.13112. find
  57. Marco S.G. Senaldi and Debra Titone. 2022. Less Direct, More Analytical: Eye-Movement Measures of L2 Idiom Reading. Languages, 7(2):91. find
  58. Francesco Vespignani, Paolo Canal, Nicola Molinaro, Sergio Fonda, and Cristina Cacciari. 2010. Predictive Mechanisms in Idiom Comprehension. Journal of Cognitive Neuroscience, 22(8):1682–1700. find
  59. Vered Shwartz and Ido Dagan. 2019. Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition. Transactions of the Association for Computational Linguistics, 7:403–419. find our card
  60. Jason Wei, Yi Tay, and Quoc V Le. 2022. Inverse Scaling Can Become U-shaped. arXiv preprint arXiv:2211.02011. find
  61. Leonie Weissweiler, Taiqi He, Naoki Otani, David R Mortensen, Lori Levin, and Hinrich Schütze. 2023. Construction Grammar Provides Unique Insight into Neural Language Models. arXiv preprint arXiv:2302.02178. find
  62. Anna Siyanova-Chanturia, Kathy Conklin, and Norbert Schmitt. 2011. Adding More Fuel to the Fire: An Eye-tracking Study of Idiom Processing by Native and Non-native Speakers. Second Language Research, 27(2):251–272. find
  63. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative. In Proceedings of EMNLP. find
  64. Nathaniel J Smith and Roger Levy. 2013. The Effect of Word Predictability on Reading Time Is Logarithmic. Cognition, 128(3):302–319. find
  65. Merity Stephen, Xiong Caiming, Bradbury James, and Richard Socher. 2017. Pointer Sentinel Mixture Models. Proceedings of ICLR. find our card
  66. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What Do RNN Language Models Learn about Filler-gap Dependencies? arXiv preprint arXiv:1809.00042. find our card
  67. David A Swinney and Anne Cutler. 1979. The Access and Processing of Idiomatic Expressions. Journal of Verbal Learning and Verbal Behavior, 18(5):523– 534. find
  68. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior. arXiv preprint arXiv:2006.01912. find
  69. Zoltán Gendler Szabó. 2004. Compositionality. Stanford Encyclopedia of Philosophy. find
  70. Stefanie Wulff. 2008. Rethinking Idiomaticity: A Usage-based Approach. A&C Black. find
  71. Debra Titone, Kyle Lovseth, Kristina Kasparian, and Mehrgol Tiv. 2019. Are Figurative Interpretations of Idioms Directly Retrieved, Compositionally Built, or Both? Evidence from Eye Movement Measures of Reading. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimental… find
  72. Kayo Yin and Graham Neubig. 2022. Interpreting Language Models with Contrastive Explanations. In Proceedings of EMNLP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rambelli-2023-are-frequent-phrases-directly-retrieved/