Elamur Research⌕
☰
home / papers / 2026 / The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

Summary

Researchers often treat language-model surprisal as a measure of how predictable a word is in context, and earlier work found that it tracks human judgments of metaphor novelty. This paper asks whether that link is driven by plain lexical frequency instead. The authors use the novelty ratings that Do Dinh et al. (2018) collected for metaphoric content words in VUAMC. They compare these ratings with two predictors:

  • word-level surprisal from all eight Pythia sizes and from the 154 training checkpoints of Pythia-70M;
  • two negative log frequency measures: NLF-Human, taken from the wordfreq library, and NLF-LM, counted in the Pythia pretraining tokens seen up to each checkpoint.

Frequency predicts novelty better than surprisal in every setting. Surprisal comes closest to novelty in small models at an early training stage, and these are also the conditions where surprisal most closely mirrors frequency. The authors conclude that the "optimal" small or early models reported in the literature may mostly be capturing frequency effects. They argue this should not be read as evidence for predictability-based accounts of metaphor novelty or of processing difficulty.

Key points

  • Data: 15,155 metaphoric content words from VUAMC, with novelty scores between -1 and +1. Of these, 353 count as novel when the cut-off is set at 0.5.
  • Frequency vs. novelty: Frequency outperforms surprisal throughout.
    • NLF-Human: ρ = .66, AUC = .90
    • NLF-LM: ρ = .63, AUC = .90
  • Training dynamics in Pythia-70M: The surprisal–novelty association peaks after 128 steps (about 268M tokens), at ρ = .62. It then settles near ρ = .45 and AUC = .83.
  • Surprisal vs. frequency: At that same peak, surprisal correlates almost perfectly with frequency (ρ = .95 with NLF-LM, .89 with NLF-Human). This later drops to around .60. Across model sizes the correlation ranges from .40 to .61 and decreases as models grow, matching the inverse scaling pattern seen for novelty.
  • Limitation: The checkpoint analysis covers only Pythia-70M. Code and data are available at github.com/OmarMomen14/surprisal_frequency_novelty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.31

@inproceedings{momen2026frequency,
  title = {{The Frequency Confound in Language-Model Surprisal and Metaphor Novelty}},
  author = {Momen, Omar and Zarrieß, Sina},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.31},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.31}
}

References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across trainin… find
  2. Eileen R. Cardillo, Christine E. Watson, Gwenda L. Schmidt, Alexander Kranjec, and Anjan Chatterjee. 2012. From novel to familiar: Tuning the brain for metaphors. NeuroImage, 59(4):3212–3221. find
  3. Andrea de Varda and Marco Marelli. 2023. Scaling in cognitive modelling: a multilingual approach to human reading times. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 139– 149, Toronto, Canada. Association for Computational… find
  4. Erik-Lân Do Dinh, Hannah Wieland, and Iryna Gurevych. 2018. Weeding out conventionalized metaphors: A corpus of novel metaphor annotations. find
  5. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Linguistics. find
  6. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. find
  7. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies, NAACL ’01, page 1–8, USA. Association for Computational Linguistics. find our card
  8. Vicky Tzuyin Lai, Tim Curran, and Lise Menn. 2009. Comprehending conventional and novel metaphors: An erp study. Brain Research, 1284:145–155. find
  9. George Lakoff and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press, Chicago, IL. find
  10. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  11. Omar Momen, Emilie Sitter, Berenike Herrmann, and Sina Zarrieß. 2026. Surprisal and metaphor novelty judgments: Moderate correlations and divergent scaling effects revealed by corpus-based and synthetic datasets. In Proceedings of the 19th Conference of the European Chapter of the Association for Co… find
  12. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… find
  13. Processing, pages 3042–3058, Miami, Florida, USA. Association for Computational Linguistics. find
  14. Gill Philip. 2016. Conventional and novel metaphors in language. In Elena Semino and Zsófia Demjén, editors, The Routledge Handbook of Metaphor and Language. Routledge, London / New York. “Chapter 15: Conventional and Novel Metaphors in Language”. find
  15. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. find
  16. Sebastian Reimann and Tatjana Scheffler. 2024. When is a metaphor actually novel? annotating metaphor novelty in the context of automatic metaphor detection. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 87–97, St. Julians, Malta. Association for Computational Linguist… find
  17. G.J. Steen, A.G. Dorst, J.B. Herrmann, A.A. Kaal, T. Krennmayr, and T. Pasma. 2010. A method for linguistic metaphor identification. From MIP to MIPVU. Number 14 in Converging Evidence in Language and Communication Research. John Benjamins. find

↑ back to top

This page in Markdown →