Elamur Research⌕
☰
home / papers / 2026 / Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets

Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets

Summary

The paper asks whether surprisal from language models can serve as a measure of how novel a metaphor is. Separating creative metaphors from conventionalized ones is difficult even for human annotators. The authors computed surprisal on metaphor-related words with 16 causal LM variants of different sizes, including instruction-tuned ones. They used four datasets: two corpus-based (VUA-ratings and VUA-dictionary, both built on VUAMC) and two synthetic (the Lai2009 psycholinguistic items and a toy set generated with GPT-4o). They also introduce cloze-surprisal, which uses a fill-in-the-blank prompt so that the word's probability reflects the whole sentence, not only its left context.

Surprisal correlated with novelty significantly but only moderately across all annotation setups. Model size had opposite effects depending on the data. Larger models fit corpus-based data worse, an inverse scaling pattern the authors link to word frequency. They fit the frequency-controlled synthetic data better, as the quality–power hypothesis predicts. The work positions surprisal as a partial but limited measure of linguistic creativity.

Key points

  • Best associations between surprisal and novelty: r = .49, ρ = .50, r_b = .69 and AUC = .84. These are comparable to the earlier feature-based result of r = .44 (Parde and Nielsen, 2018).
  • VUAMC contains 186,673 words, of which 15,155 are content MRWs.
    • VUA-ratings (Do Dinh et al., 2018) gives Best-Worst Scaling scores, with 353 metaphors labelled novel at the 0.5 threshold.
    • VUA-dictionary (Reimann and Scheffler, 2024) labels 409 metaphors as novel.
  • Lai2009 is built around 104 target words, each placed in literal, conventional, novel and anomalous sentences.
  • Cloze-surprisal raised correlations by a few points for many models. Instruction tuning tended to weaken alignment with human novelty annotations.
  • Genre, the density of novel metaphors and LM perplexity all affected how well surprisal predicted novelty. Code and data are available at github.com/OmarMomen14/surprisal-metaphor-novelty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Psycholinguistics & language acquisition Idioms & multiword expressions

Cite

Cite as Omar Momen, Emilie Sitter, Berenike Herrmann and Sina Zarrieß. 2026. Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.378

@inproceedings{momen2026surprisal,
  title = {{Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets}},
  author = {Omar Momen, Emilie Sitter, Berenike Herrmann and Sina Zarrieß},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.378},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.378}
}

References

53 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037– 2050, D… find
  2. Kathleen Ahrens, Christian Burgers, and Yin Zhong. 2024. Making the unseen seen: The role of signaling and novelty in rating metaphors. Journal of Psycholinguistic Research, 53(3):36. find
  3. Yossi Arzouan, Abraham Goldstein, and Miriam Faust. 2007. Brainwaves are stethoscopes: Erp correlates of novel metaphor comprehension. Brain Research, 1160:69–81. find
  4. Beata Beigman Klebanov, Chee Wee (Ben) Leong, and Michael Flor. 2018. A corpus of non-native written English annotated for metaphor. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pape… find
  5. Brian F. Bowdle and Dedre Gentner. 2005. The career of metaphor. Psychological Review, 112(1):193–216. find
  6. E. R. Cardillo, G. L. Schmidt, A. Kranjec, and A. Chatterjee. 2010. The neural bases of metaphoric expression comprehension. Behavior Research Methods, 42(3):651–660. find
  7. Nicholas Ichien, Dušan Stamenković, and Keith J. Holyoak. 2024. Large language model displays emergent ability to interpret novel literary metaphors. Preprint, arXiv:2308.01497. — [arXiv:2308.01497](https://arxiv.org/abs/2308.01497) find
  8. E. R. Cardillo, C. E. Watson, G. L. Schmidt, A. Kranjec, and A. Chatterjee. 2012. Selective metaphor impairments after left, not right, hemisphere injury. Frontiers in Psychology, 3:87. find
  9. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. As… our card
  10. Goran Djokic, Ekaterina Shutova, and Verna Dankers. 2021. Episodic memory demands modulate novel metaphor use during event narration. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society. find
  11. Erik-Lân Do Dinh, Hannah Wieland, and Iryna Gurevych. 2018. Weeding out conventionalized metaphors: A corpus of novel metaphor annotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Lin… find
  12. Markus Egg and Valia Kordoni. 2022. Metaphor annotation for German. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2556–2562, Marseille, France. European Language Resources Association. find
  13. Tom Fawcett. 2006. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861–874. find
  14. Raymond W. Gibbs, Jr. 2005. Embodiment and Cognitive Science. Cambridge University Press. find
  15. Gene V. Glass. 1966. Note on rank-biserial correlation. Educational and Psychological Measurement, 26(3):623–631. find
  16. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Comput… our card
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and 1 others. 2024. The llama 3 herd of models. arXiv preprint. find
  18. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. our card
  19. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  20. Tatsuki Kuribayashi, Yohei Oseki, and Timothy Baldwin. 2024. Psychometric predictive power of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1983–2005, Mexico City, Mexico. Association for Computational Linguistics. find
  21. Vicky Tzuyin Lai, Tim Curran, and Lise Menn. 2009. Comprehending conventional and novel metaphors: An erp study. Brain Research, 1284:145–155. find
  22. George Lakoff and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press, Chicago, IL. find
  23. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. find
  24. Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2025. Causal estimation of tokenisation bias. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28325– 28340, Vienna, Austria. Association for Com… find
  25. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  26. Rowan Hall Maudslay and Simone Teufel. 2022. Metaphorical polysemy detection: Conventional metaphor meets word sense disambiguation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 65–77, Gyeongju, Republic of Korea. International Committee on Computational Li… find
  27. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963– 980, Online and Punta Cana, Dominican Republic… find
  28. Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33, Berlin, Germany. Association for Computational Linguistics. our card
  29. Michael Mohler, Mary Brunson, Bryan Rink, and Marc Tomlinson. 2016. Introducing the LCC metaphor datasets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC‘16), pages 4221–4227, Portorož, Slovenia. European Language Resources Association (ELRA). find
  30. Arthur Neidlein, Philip Wiesenbach, and Katja Markert. 2020. An analysis of language models for metaphor recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3722–3736, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  31. Byung-Doh Oh and William Schuler. 2023a. Transformer-based language model surprisal predicts human reading times best with about two billion training tokens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1915–1921, Singapore. Association for Computational Linguistic… find
  32. Byung-Doh Oh and William Schuler. 2023b. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. our card
  33. Byung-Doh Oh and William Schuler. 2024. Leading whitespaces of language models’ subword vocabulary pose a confound for calculating word probabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3464–3472, Miami, Florida, USA. Association for Comp… find
  34. Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal‘s fit to reading times. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (… find
  35. Natalie Parde and Rodney Nielsen. 2018. A Corpus of Metaphor Novelty Scores for Syntactically-Related Word Pairs. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  36. Karl Pearson. 1895. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242. find
  37. Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192–204, Punta Cana, Dominican Re… find
  38. Gill Philip. 2016. Conventional and novel metaphors in language. In Elena Semino and Zsófia Demjén, editors, The Routledge Handbook of Metaphor and Language, page 14. Routledge, London / New York. “Conventional and Novel Metaphors in Language”. find
  39. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. find
  40. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. find
  41. W. Gudrun Reijnierse, Christian Burgers, Tina Krennmayr, and Gerard J. Steen. 2019. Metaphor in communication: the distribution of potentially deliberate metaphor across register and word class. Corpora, 14(3):301–326. find
  42. Sebastian Reimann and Tatjana Scheffler. 2024. When is a metaphor actually novel? annotating metaphor novelty in the context of automatic metaphor detection. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 87–97, St. Julians, Malta. Association for Computational Linguist… find
  43. C. Roncero and R. G. de Almeida. 2014. The role of expectedness and predication in metaphor comprehension. Language and Cognition, 6(2):141–168. find
  44. Elisa Sanchez-Bayona and Rodrigo Agerri. 2022. Leveraging a new Spanish corpus for multilingual and cross-lingual metaphor detection. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 228–240, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com… find
  45. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. Large-scale evidence for logarithmic effects of word predictability on reading time. Proceedings of the National Academy of Sciences, 121(10):e2307876121. find
  46. Claude E. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27(3 and 4):379–423, 623–656. Reprinted with corrections. find
  47. Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15:72–101. find
  48. G.J. Steen, A.G. Dorst, J.B. Herrmann, A.A. Kaal, T. Krennmayr, and T. Pasma. 2010. A method for linguistic metaphor identification. From MIP to MIPVU. Number 14 in Converging Evidence in Language and Communication Research. John Benjamins. find
  49. The Pragglejaz Group. 2007. Mip: A method for identifying metaphorically used words in discourse. Metaphor and Symbol, 22(1):1–39. find
  50. Lindia Tjuatja, Graham Neubig, Tal Linzen, and Sophie Hao. 2025. What goes into a LM acceptability judgment? rethinking the impact of frequency and length. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Te… find
  51. Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor understanding challenge dataset for LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3517–3536, Bangkok, Thailand. Association for Computa… our card
  52. Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. 2023. Language model quality correlates with psychometric predictive power in multiple languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7503–7511, Singapore. Association for Co… find
  53. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. CoRR, abs/2006.01912. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/momen-2026-surprisal-metaphor-novelty-judgments-moderate/