Elamur Research⌕
☰
home / papers / 2025 / Surprisal Dynamics for the Detection of Multi-Word Expressions in English

Surprisal Dynamics for the Detection of Multi-Word Expressions in English

Paper · 2025 · Diego Alves, Sergei Bagdasarov and Elke Teich

Summary

The authors ask whether a language model's predictability signal can characterize multi-word expressions (MWEs) in English in an interpretable, cognitively grounded way. They do not aim to build a full extractor. Using GPT-2 small (124M parameters) through the surprisal Python library, they compute word-level surprisal and fit a straight line over the surprisal values of each candidate expression's tokens. The slope of that line is the feature. A negative slope means each later component becomes easier to predict.

On DiMSUM, they compare annotated MWEs against random non-MWE n-grams of matched length using a logistic GAM. On SemEval-2022 idiomaticity data, they compare idiomatic and literal uses of the same bigrams. Slope separates MWEs from random sequences moderately well, with high recall and lower precision. It works best on conventionalized bigrams, where both idiomatic and literal readings decline in surprisal, and idiomatic readings decline more steeply. The work matters because it gives an explainable, theory-motivated cue that can sit alongside association measures, rather than one more opaque classifier.

Key points

  • Data: DiMSUM, which combines STREUSLE 2.1 web reviews with the Ritter and Lowlands Twitter sets in BIO tagging. It has 5,044 MWE occurrences, of which 4,541 are continuous and 503 discontinuous. The authors drew 5,044 length-matched random non-MWE n-grams as controls.
  • SemEval-2022 English training set: 3,327 sentences covering 163 noun-containing bigrams, each labelled idiomatic or literal. After subword-to-word alignment problems, 2,755 sentences were kept.
  • Surprisal is computed on GPT-2 subword tokens and summed per word. The slope comes from a first-degree polynomial fit in numpy. Significance was tested with a pygam logistic GAM using slope as the only smooth predictor.
  • Results on DiMSUM: precision 60.6% and recall 79.3%. False positives cluster in formulaic-looking POS patterns such as adjective–noun and verb–pronoun.
  • Results on SemEval-2022: negative slopes identify bigrams as MWEs with 85.4% accuracy. Some items, such as big wig, white spirit and fashion plate, show mostly rising surprisal instead. Adjective–noun bigrams follow the expected pattern less reliably.

Interesting ideas

  • I1. Rather than scoring a single word, the method summarizes how predictability changes across a span with one slope, which gives a compact, explainable span-level feature usable for any multi-token unit.
  • I2. The control set is built from random n-grams of the same corpus with the same length distribution, so any difference cannot be explained by span length alone.
  • I3. The paper contrasts idiomatic and literal uses of the identical bigram, which isolates the effect of meaning from the effect of word form.
  • I4. The error analysis by POS pattern shows where a feature fails (adjective–noun, verb–pronoun), turning a moderate metric into actionable knowledge about the feature's blind spots.
  • I5. The authors argue slope should complement association measures such as PMI rather than replace them, which suggests stacking independent witnesses for one decision.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A surprisal-slope check could flag candidate fixed expressions (e.g. at all, due to) so the tokenizer or lemmatizer treats them as single lexical units, and the slope value itself serves as the stated reason.
  • I2 (learn): When evaluating any span detector (MWEs, named entities, frame triggers), length-matched random spans are a cheap negative control that exposes length-driven artefacts.
  • I3 (frames): The steeper slope in idiomatic contexts could act as a weak cue that a phrase like break the ice should map to a non-literal frame instead of composing its parts' roles. It should be combined with other evidence, since literal uses also slope downward.
  • I4 (world): When compiling a lexicon of multi-word entries, POS-pattern breakdowns could decide where surprisal evidence is trusted and where association scores or WordNet membership must confirm the entry.
  • I5 (learn): Surprisal slope could become one more witness in silver-data agreement for MWE annotation. Disagreements with PMI or the treebank's fixed/compound labels would then be routed to the LLM teacher tier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksIdioms & multiword expressions · 217 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Diego Alves, Sergei Bagdasarov and Elke Teich. 2025. Surprisal Dynamics for the Detection of Multi-Word Expressions in English. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-ijcnlp.72

@inproceedings{alves2025surprisal,
  title = {{Surprisal Dynamics for the Detection of Multi-Word Expressions in English}},
  author = {Diego Alves, Sergei Bagdasarov and Elke Teich},
  booktitle = {Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-ijcnlp.72},
  url = {https://doi.org/10.18653/v1/2025.findings-ijcnlp.72}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diego Alves, Stefan Fischer, Stefania Degaetano-Ortlieb, and Elke Teich. 2024. Multi-word expressions in english scientific writing. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 67–76. ↗
  2. Kathy Conklin and Norbert Schmitt. 2008. Formulaic sequences: Are they processed more quickly than nonformulaic language by native and nonnative speakers? Applied Linguistics, 29(1):72–89. ↗
  3. Kathy Conklin and Norbert Schmitt. 2012. The processing of formulaic language. Annual Review of Applied Linguistics, 32:45–61. ↗
  4. Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255– 308.
  5. Vera Demberg and Frank Keller. 2009. A computational model of prediction in human parsing: Unifying locality and surprisal effects. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31. find
  6. Stefan Th. Gries. 2022. Multi-word units (and tokenization more generally): A multi-dimensional and largely information-theoretic approach. Lexis, (19). Online since 26 March 2022. ↗
  7. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  8. Natalia Klyueva, Antoine Doucet, and Milan Straka. 2017. Neural networks for multi-word expression detection. In Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017), pages 60–65. Association for Computational Linguistics. find
  9. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. arXiv preprint arXiv:2406.10851. ↗
  10. Luca Onnis and Falk Huettig. 2021. Can prediction and retrodiction explain whether frequent multi-word phrases are accessed ‘precompiled’ from memory or compositionally constructed on the fly? Brain Research, 1772:147674. find
  11. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. arXiv preprint arXiv:2406.14561. ↗
  12. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  13. Carlos Ramisch, Abigail Walsh, Thomas Blanchard, and Shiva Taslimipoor. 2023. A survey of MWE identification experiments: The devil is in the details. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 106–120, Dubrovnik, Croatia. Association for Computational Linguistics. find
  14. Alan Ritter, Sam Clark, Oren Etzioni, and 1 others. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1524–1534.
  15. Agata Savary, Carlos Ramisch, Silvio Ricardo Cordeiro, Federico Sangati, Veronika Vincze, Behrang Qasemi Zadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and 1 others. 2017. The parseme shared task on automatic identification of verbal multiword expressions. In The 13th Workshop on Multiword Expression at EACL, pages 31–47.
  16. Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. 2016. Semeval-2016 task˜ 10: Detecting minimal semantic units and their meanings (dimsum). In 10th International Workshop on Semantic Evaluation, pages 546–559. Association for Computational Linguistics.
  17. Anna Siyanova-Chanturia, Kathy Conklin, Sendy Caffarra, Edith Kaan, and Walter JB van Heuven. 2017. Representation and processing of multi-word expressions in the brain. Brain and language, 175:111–122. ↗
  18. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107– 121, Seattle, United States. Association for Computational Linguistics. ↗
  19. Antoine Tremblay and R. Harald Baayen. 2010. Holistic processing of regular four-word sequences: A behavioral and erp study of the effects of structure, frequency, and probability on immediate free recall. In David Wood, editor, Perspectives on Formulaic Language: Acquisition and Communication, pages 151–173. Continuum International, London, UK. find
  20. Antoine Tremblay, Bruce Derwing, Gary Libben, and Chris Westbury. 2011. Processing advantages of lexical bundles: Evidence from self-paced reading and sentence recall tasks. Language Learning, 61:569– 613. ↗
  21. Jake Ryland Williams. 2016. Boundary-based mwe segmentation with text partitioning. arXiv preprint arXiv:1608.02025. ↗
  22. Chiraz Ben Youssef. 2024. mMERGE: A Corpus Driven Multiword Expressions Discovery Algorithm. Ph.D. thesis, University of California, Santa Barbara. Pro-Quest ID: BenYoussef_ucsb_0035D_16720; Merritt ID: ark:/13030/m5457324. find
  23. Wen Zhang, Taketoshi Yoshida, Xijin Tang, and Tu-Bao Ho. 2009. Improving effectiveness of mutual information for substantival multiword expression extraction. Expert Systems with Applications, 36(8):10919– 10930. ↗
  24. Nathan Schneider and Noah A Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In A Corpus and Model Integrating Multiword Expressions and Supersenses, pages 1537– 1547.
  25. Claude E Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. ↗
  26. Rita Simpson-Vlach and Nick C Ellis. 2010. An academic formulas list: New methods in phraseology research. Applied Linguistics, 31(4):487–512. ↗
  27. A. Siyanova-Chanturia, K. Conklin, and W. J. B. van Heuven. 2011. Seeing a phrase "time and again" matters: The role of phrasal frequency in the processing of multiword sequences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(3):776–784. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alves-2025-surprisal-dynamics-detection-multi-word/