Elamur Research⌕
☰
home / papers / 2025 / A Survey of Idiom Datasets for Psycholinguistic and Computational Research

A Survey of Idiom Datasets for Psycholinguistic and Computational Research

Summary

Idioms are hard for NLP systems and tricky for human experiments. Their figurative sense often cannot be derived from the component words, and one phrase can be literal in one context and figurative in another. Resources for studying idioms come from two fields and vary in annotation, languages, idiom types and size, with no common benchmark. The authors review 53 idiom datasets in two groups. Psycholinguistic norming collections rate idioms as types on Likert scales, covering familiarity, transparency, compositionality, literality, age of acquisition, predictability, syntactic flexibility, and newer affective and imageability dimensions. Computational corpora are drawn from real text and treat idioms as tokens in context, supporting idiomatic/literal classification, paraphrasing, sentiment and multilingual modeling. Language and task coverage is growing. However, annotation levels are inconsistent, datasets favor either depth or breadth, few idioms are aligned across languages, and genre coverage is narrow. The two research traditions remain essentially disconnected. The authors propose shared metadata and interoperable formats, and see idiom valence and computational modeling of human ratings as possible bridges.

Key points

  • The survey covers 53 datasets, 16 of them psycholinguistic norming resources; an updated list is maintained at github.com/maafiah/IdiomsResearch.
  • Classification benchmarks:
    • VNC-Tokens: about 3,000 English verb-noun usages.
    • IDIX: over 5,800 sentences, not public.
    • SemEval-2013 Task 5b: 85 idioms with over 4,000 instances.
    • RU Idioms: 2,420 idiomatic and 3,027 literal Russian instances.
    • MAGPIE: 56,622 instances covering 1,756 English idioms.
  • Sentiment and emotion resources:
    • IDIOMENT: 580 idioms.
    • SLIDE: more than 5,000 idioms.
    • IDEM: about 9,685 emotion-labelled sentences.
  • Multilingual resources:
    • LIdioms: 815 idioms in five languages, in RDF.
    • IMIL: more than 2,200 English idioms plus seven Indian languages, 250K sentences.
    • PETCI: 4,310 Chinese idioms with 29,936 English translations.
    • SemEval-2022 Task 2: English, Portuguese, and zero-shot Galician.
    • IdiomKB: English, Chinese and Japanese, built for LLM translation.
  • Main gaps are heterogeneous annotation schemes, few cross-lingually aligned idioms, limited attention to genre and register, and no link yet between psycholinguistic and computational idiom research.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{flor2025survey,
  title = {{A Survey of Idiom Datasets for Psycholinguistic and Computational Research}},
  author = {Flor, Michael and Liu, Xinyi and Feldman, Anna},
  booktitle = {Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Long and Short Papers},
  year = {2025},
  url = {https://aclanthology.org/2025.konvens-1.9/}
}

References

69 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tosin Adewumi, Roshanak Vadoodi, Aparajita Tripathy, Konstantina Nikolaido, Foteini Liwicki, and Marcus Liwicki. 2022. Potential idiomatic expression (PIE)-English: Corpus for classes of idioms. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 689–696, Marseille,… find
  2. Francesca M. M. Citron, Cristina Cacciari, Michael Kucharski, Luna Beck, Markus Conrad, and Arthur M. Jacobs. 2016. When emotions are expressed figuratively: Psycholinguistic and Affective Norms of 619 Idioms for German (PANIG). Behavior Research Methods, 48:91–111. find
  3. Ruchit Agrawal, Vighnesh Chenthil Kumar, Vignesh-waran Muralidharan, and Dipti Sharma. 2018. No more beating about the bush : A step towards idiom handling for Indian language NLP. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Jap… find
  4. Louise Connell and Dermot Lynott. 2012. Strength of perceptual experience predicts word processing performance better than concreteness or imageability. Cognition, 125(3):452–465. find
  5. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The vnc-tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008), Marrakech, Morocco. find
  6. Katsiaryna Aharodnik, Anna Feldman, and Jing Peng. 2018. Designing a Russian idiom-annotated corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  7. Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1–57. find
  8. Ricarda Dormeyer and Ingrid Fischer. 1998. Building Lexicons out of a Database for Idioms. In Proceedings of the First International Conference on Language Resources and Evaluation, pages 833–838. find
  9. Chikara Hashimoto and Daisuke Kawahara. 2008. Construction of an idiom corpus and its application to idiom identification based on WSD incorporating idiom-specific features. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 992–1001, Honolulu, Hawaii. A… find
  10. M. Teresa Espinal and Jaume Mateu. 2019. Idioms and Phraseology. In Oxford Research Encyclopedia of Linguistics. Oxford University Press. find
  11. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  12. Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 248–264, Dubrovnik, Croatia. Assoc… find our card
  13. Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61–103. find our card
  14. Ferdy Hubers, Catia Cucchiarini, Helmer Strik, and Ton Dijkstra. 2019. Normative Data of Dutch Idiomatic Expressions: Subjective Judgments You Can Bank on. Frontiers in Psychology, 10:1075. find
  15. Christiane Fellbaum and Alexander Geyken. 2005. Transforming a Corpus Into a Lexical Resource - the Berlin Idiom Project. Revue française de linguistique appliquée, X(2):49–62. find
  16. Zhiying Jiang, Boliang Zhang, Lifu Huang, and Heng Ji. 2018. Chengyu cloze test. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 154–158, New Orleans, Louisiana. Association for Computational Linguistics. find
  17. Fraser. 1970. Idioms within a transformational grammar. Foundation of Language, 6:22–42. find
  18. Charles Jochim, Francesca Bonin, Roy Bar-Haim, and Noam Slonim. 2018. SLIDE - a sentiment lexicon of common idioms. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  19. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. As… find our card
  20. Robert A. Johnston and Christopher Barry. 2006. Age of acquisition and lexical processing. Visual Cognition, 13(7-8):789–845. find
  21. José M. Gavilán, Juan Haro, José Antonio Hinojosa, Isabel Fraga, and Pilar Ferré. 2021. Psycholinguistic and affective norms for 1,252 Spanish idiomatic expressions. PLoS ONE, 16(7). find
  22. Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, and Yadollah Yaghoobzadeh. 2024. Comparative study of multilingual idioms and similes in large language models. Preprint, arXiv:2410.16461. find our card
  23. Raymond W. Gibbs and Gayle P. Gonzales. 1985. Syntactic frozenness in processing and remembering idioms. Cognition, 20(3):243–259. find
  24. Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEv… find
  25. Raymond W. Gibbs and Jennifer E. O’Brien. 1990. Idioms and mental imagery: The metaphorical motivation for idiomatic meaning. Cognition, 36(1):35–68. find
  26. Begoña Góngora, Andre Gómez-Lombardi, and Alonso Ortega González. 2022. Descriptive Norms for 1,082 Chilean-Spanish Idiomatic Expressions. Revista signos, 55(110). find
  27. Murathan Kurfalı, Robert Östling, Johan Sjons, and Mats Wirén. 2020. A multi-word expression dataset for Swedish. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4402–4409, Marseille, France. European Language Resources Association. find
  28. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  29. Anastasia Lada, Philippe Paquier, Ifigenia Dosi, Christina Manouilidou, Simone Sprenger, and Stefanie Keulen. 2024. Four hundred Greek idiomatic expressions: Ratings for subjective frequency, ambiguity, and decomposability. Behavior Research Methods, 56:8181–8195. find
  30. Jennifer L. Hamblin and Raymond W. Gibbs. 1999. Why You Can’t Kick the Bucket as You Slowly Die: Verbs in Idiom Comprehension. Journal of Psycholinguistic Research, 28:25–39. find
  31. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. CultureLLM: Incorporating Cultural Differences into Large Language Models. Preprint, arXiv:2402.10946. find our card
  32. Diego Moussallem, Mohamed Ahmed Sherif, Diego Esteves, Marcos Zampieri, and Axel-Cyrille Ngonga Ngomo. 2018. LIdioms: A multilingual linked idioms data set. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re… find
  33. Degao Li, Yu Zhang, and Xiaolu Wang. 2016. Descriptive norms for 350 Chinese idioms with seven syntactic structures. Behavior Research Methods, 48:1678–1693. find
  34. Emily Nordmann and Antonia A. Jambazova. 2017. Normative data for idiomatic expressions. Behavior Research Methods, 49:198–215. find
  35. Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, and Yanghua Xiao. 2024b. Translate meanings, not just words: IdiomKB’s role in optimizing idiomatic translation with language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth C… find
  36. Geoffrey Nunberg, Ivan A. Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491–538. find
  37. Charles E. Osgood, George J. Suci, and Percy H. Tannenbaum. 1957. The measurement of meaning. University of Illinois Press. find
  38. Irene Pagliai. 2023. Bridging the Gap: Creation of a Lexicon of 150 Pairs of English and Italian Idioms Including Normed Variables for the Exploration of Idiomatic Ambiguity. Journal of Open Humanities data, 9(1). find
  39. Maya R. Libben and Debra A. Titone. 2008. The multi-determined nature of idiom processing. Memory & Cognition, 36(6):1103–1123. find
  40. Changsheng Liu and Rebecca Hwa. 2016. Phrasal substitution of idiomatic expressions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 363–373, San Diego, California. Association for Computational… find
  41. Allan Paivio. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45(3):255–287. find
  42. Allan Paivio, John C. Yuille, and Stephen A. Madigan. 1968. Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of Experimental Psychology, 76(1):1–25. find
  43. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  44. Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2014. Classifying idiomatic and literal expressions using topic models and intensity of emotions. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2019–2027, Doha, Qatar. Association for Computat… find
  45. Maria Pershina, Yifan He, and Ralph Grishman. 2015. Idiom paraphrases: Seventh heaven vs cloud nine. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 76–82, Lisbon, Portugal. Association for Computational Linguistics. find
  46. Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184, Melbourne, Australia. Association for Computational Li… find our card
  47. Mahsa Morid and Laura Sabourin. 2024. Affective and sensory–motor norms for idioms by L1 and L2 English speakers. Applied Psycholinguistics, 45(1):138–155. find
  48. Alexander Prochnow, Johannes E. Bendler, Caroline Lange, Foivos Ioannis Tzavellos, Bas Marco Göritzer, Marijn ten Thij, and Riza Batista-Navarro. 2024. IDEM: The IDioms with EMotions dataset for emotion recognition. In Proceedings of the 2024 Joint International Conference on Computational Linguisti… find
  49. Catriona M. Morrison, Tameron D. Chappell, and Andrew W. Ellis. 1997. Age of Acquisition Norms for a Large Set of Object Names and Their Relation to Adult Estimates and Other Variables. The Quarterly Journal of Experimental Psychology Section A, 50(3):528–559. find
  50. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 210–218, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find our card
  51. Rosamund Moon. 1998. Fixed Expressions and Idioms in English: A Corpus-Based Approach. Clarendon Press, Oxford. find
  52. James A. Russell. 2003. Core affect and the psychological construction of emotio. Psychological Review, 110(1):145–172. find
  53. Debra Titone and Maya Libben. 2014. Time-dependent effects of decomposability, familiarity and literal plausibility on idiom priming: A cross-modal priming investigation. The Mental Lexicon, 9(3):473–496. find
  54. Prateek Saxena and Soma Paul. 2021. Labelled epie: A dataset for idiom sense disambiguation. In Text, Speech, and Dialogue, pages 210–221, Cham. Springer International Publishing. find
  55. Lei Wang and Shiwen Yu. 2010. Construction of Chinese idiom knowledge-base and its applications. In Proceedings of the 2010 Workshop on Multiword Expressions: from Theory to Applications, pages 11–18, Beijing, China. find
  56. Coling 2010 Organizing Committee. find
  57. Wendy A. Schweigert. 1986. The comprehension of familiar and less familiar idioms. Journal of Psycholinguistic Research, 15(1):33–45. find
  58. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior Research Methods, 45(4):1191–1207. find
  59. Wendy A. Schweigert and Danny R. Moates. 1988. Familiar idiom comprehension. Journal of Psycholinguistic Research, 17(4):281–296. find
  60. Lowri Williams, Christian Bannister, Michael Arribas-Ayllon, Alun Preece, and Irena Spasić. 2015. The role of idioms in sentiment analysis. Expert Systems with Applications, 42(21):7375–7385. find
  61. Marco Silvio Giuseppe Senaldi. 2019. Working both sides of the street: computational and psycholinguistic investigations on idiomatic variability. Ph.D. thesis, Scuola Normale Superiore, Pisa. find
  62. Caroline Sporleder, Linlin Li, Philip Gorinski, and Xaver Koch. 2010. Idioms in context: The IDIX corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC‘10), Valletta, Malta. European Language Resources Association (ELRA). find
  63. Bodo Winter. 2022. Managing Semantic Norms for Cognitive Linguistics, Corpus Linguistics, and Lexicon Studies. In Andrea L. Berez-Kroeker, Bradley McDonnell, Eve Koller, and Lauren B. Collister, editors, The Open Handbook of Linguistic Data Management, pages 489–497. MIT Press. find
  64. Patrizia Tabossi, Lisa Arduino, and Rachele Fanari. 2011. Descriptive norms for 245 Italian idiomatic expressions. Behavior Research Methods, 43:110–123. find
  65. Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. ChID: A large-scale Chinese IDiom dataset for cloze test. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 778–787, Florence, Italy. Association for Computational Linguistics. find
  66. Patrizia Tabossi, Rachele Fanari, and Kinou Wolf. 2008. Processing idiomatic expressions: Effects of semantic compositionality. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(2):313–327. find
  67. Jianing Zhou, Hongyu Gong, and Suma Bhat. 2021. PIE: A parallel idiomatic expression corpus for idiomatic sentence generation and paraphrasing. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 33–48, Online. Association for Computational Linguistics. find
  68. Kenan Tang. 2022. Petci: A parallel english translation dataset of chinese idioms. Preprint, arXiv:2202.09509. — [arXiv:2202.09509](https://arxiv.org/abs/2202.09509) find
  69. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/flor-2025-survey-idiom-datasets-psycholinguistic-computational/