Elamur Research⌕
☰
home / papers / 2026 / Decomposing Creativity: Two Small Datasets Combining Originality Ratings and Metaphor Annotations

Decomposing Creativity: Two Small Datasets Combining Originality Ratings and Metaphor Annotations

Paper · 2026 · Emilie Sitter, Sina Zarrieß, Omar Momen et al.

Summary

The paper addresses a gap in NLP resources: few datasets pair holistic judgments of creativity with fine-grained annotation of specific devices such as metaphor. The authors release MetaphOrig, two small German datasets of sentence-level spatial descriptions. One draws on literary prose from the 18th–20th centuries (KOLIMO), the other on 21st-century travel reports (Wikivoyage). Spatial descriptions were chosen because they are similar in content across genres and make sense out of context, which should make originality easier to judge.

Each sentence has two kinds of annotation:

  • originality scores on a 1–6 scale, collected from crowdworkers and from four LLMs;
  • word-level metaphor labels following MIPVU, separating direct from indirect metaphors.

In an illustrative study, originality serves as a proxy for creativity and is correlated with per-sentence metaphor density. A positive link appears only in the literary KOLIMO data, not in Wikivoyage. Human and LLM ratings show the same pattern. The resource therefore supports research on creativity and metaphor within one domain or across both, and on whether LLM ratings rely on cues similar to human judgments.

Key points

  • MetaphOrig consists of two genre-specific German datasets: KOLIMO (literary prose, 18th–20th c.) and Wikivoyage (factual travel writing, 21st c.).
  • Originality ratings come from crowdsourcing via Prolific and from four LLMs: GPT-5, Qwen2.5-32B-Instruct, Mistral-Small-3.2-24B-Instruct and Llama-3.2-3B. The LLMs received a German prompt that mimicked the human rater instructions and asked for a JSON-formatted rating plus justification.
  • Metaphor annotation follows MIPVU at the word level and distinguishes direct metaphors (e.g., similes) from indirect ones. Metaphor density is computed for each sentence.
  • Example contrast: a Keyserling sentence has density ≈0.24 and a human originality score of 4.8. A Wikivoyage sentence has density ≈0.05 and a human score of 2.1.
  • Main finding: metaphor density correlates with originality only in KOLIMO, for both human and LLM ratings. In Wikivoyage the trend is not significant.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bielefeld UniversityBielefeld · Germany35 works · 1,196 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Idioms & multiword expressions · 217 works

Cite

Cite as Emilie Sitter, Sina Zarrieß, Omar Momen and Berenike Herrmann. 2026. Decomposing Creativity: Two Small Datasets Combining Originality Ratings and Metaphor Annotations. Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/486rgpcgguib

@inproceedings{sitter2026decomposing,
  title = {{Decomposing Creativity: Two Small Datasets Combining Originality Ratings and Metaphor Annotations}},
  author = {Emilie Sitter, Sina Zarrieß, Omar Momen and Berenike Herrmann},
  booktitle = {Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026},
  year = {2026},
  doi = {10.63317/486rgpcgguib},
  url = {https://doi.org/10.63317/486rgpcgguib}
}

References

51 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Selcuk Acar. 2025. Creativity Assessment, Research, and Practice in the Age of Artificial Intelligence. Creativity Research Journal, 37(2):181–187. Publisher: Routledge _eprint: https://doi.org/10.1080/10400419.2023.2271749. — [doi:10.1080/10400419.2023.2271749](https://doi.org/10.1080/10400419.2023.2271749) ↗
  2. Teresa M. Amabile. 1982. Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5):997–1013. Place: US Publisher: American Psychological Association. find
  3. Christian Burgers, Elly A. Konijn, Gerard J. Steen, and Marlies A.R. Iepsma. 2015. Making ads less complex, yet more creative and persuasive: the effects of conventional metaphors and irony in print advertising. International Journal of Advertising, 34(3):515–532. find
  4. Qian Cao, Xiting Wang, Yuzhuo Yuan, Yahui Liu, Fang Luo, and Ruihua Song. 2026. Evaluating text creativity across diverse domains: a dataset and large language model evaluator. In The Fourteenth International Conference on Learning Representations. find
  5. Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. find
  6. Rune H. B. Christensen. 2023. ordinal—Regression Models for Ordinal Data. find
  7. EAV Crompvoets. 2025. Behind the scenes of pairwise comparison for educational measurement. Ridderprint. find
  8. Jennifer Diedrich, Mathias Benedek, Emanuel Jauk, and Aljoscha C. Neubauer. 2015. Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1):35– 40. Place: US Publisher: Educational Publishing Foundation. find
  9. Stefanie Dipper, Adam Roussel, Alexandra Wiemann, Won Kim, and Tra-My Nguyen. 2024. Guidelines for the Annotation of Intentional Linguistic Metaphor. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 53–58, Mexico City, Mexico (Hybrid). Association for Comput… find
  10. J. Berenike Herrmann and Gerhard Lauer. 2018. Korpusliteraturwissenschaft. Zur Konzeption und Praxis am Beispiel eines Korpus zur literarischen Moderne. Osnabrücker Beiträge zur Sprachtheorie, 92:127–156. find
  11. J. Berenike Herrmann, Karola Woll, and Aletta G. Dorst. 2019. Linguistic metaphor identification in German. Metaphor Identification in Multiple Languages. MIPVU around the world. ISBN: 9789027204721. find
  12. Rebecca M. M. Hicke and Ross Deans Kristensen-McLachlan. 2024. SCIENCE IS EXPLO-RATION: Computational Frontiers for Conceptual Metaphor Theory. In CHR 2024: Computational Humanities Research Conference,, pages 1105– 1116, Aarhus, Denmark. find
  13. Jan Horstmann. 2019. KOLIMO: Korpus der literarischen Moderne. forTEXT. Literatur digital erforschen. find
  14. Paul V. DiStefano, John D. Patterson, and Roger E. Beaty. 2024. Automatic Scoring of Metaphor Creativity with Large Language Models. Creativity Research Journal. Publisher: Routledge _eprint: https://doi.org/10.1080/10400419.2024.2326343. — [doi:10.1080/10400419.2024.2326343](https://doi.org/10.1080/10400419.2024.2326343) ↗
  15. Jan Horstmann and E Akazawa. 2024. Ressourcenbeitrag: Korpus der literarischen Moderne (KOLIMO). forTEXT, Korpusbildung, 1(2). Medium: PDF,XML Publisher: Universitäts-und Landesbibliothek Darmstadt. find
  16. Erik-Lân Do Dinh, Hannah Wieland, and Iryna Gurevych. 2018. Weeding out Conventionalized Metaphors: A Corpus of Novel Metaphor Annotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Lin… find
  17. James C. Kaufman, John Baer, and Jason C. Cole. 2009. Expertise, Domains, and the Consensual Assessment Technique. The Journal of Creative Behavior, 43(4):223–233. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2162-6057.2009.tb01316.x. — [doi:10.1002/j.2162-6057.2009.tb01316.x](https://doi.org/10.1002/j.2162-6057.2009.tb01316.x) ↗
  18. Lettie Dorst. 2015. More or different metaphors in fiction? A quantitative cross-register comparison. Language and Literature, 24:3–22. find
  19. Sungeun Kim and Dongsuk Oh. 2025. Evaluating Creativity: Can LLMs Be Good Evaluators in Creative Writing Tasks? Applied Sciences, 15(6):2971. Number: 6 Publisher: Multidisciplinary Digital Publishing Institute. find
  20. Markus Egg and Valia Kordoni. 2022. Metaphor annotation for German. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2556–2562, Marseille, France. European Language Resources Association. find
  21. Christine A. Knoop and Stefan Blohm. 2025. Liter-arinesses—A Bag of Three-Sided Coins. Literature, 5(3):21. find
  22. B.A. Hennessey, Teresa M. Amabile, and J.S. Mueller. 2011. Consensual Assessment. In Encyclopedia of Creativity, pages 253–260. Elsevier. find
  23. Christine A. Knoop, Thomas Nehrlich, Sabrina Aristei, Oliver Lubrich, Kirsten Stark, Alexander Enge, Werner Sommer, and Rasha Abdel Rahman. 2024. The usual miracles: How narrative style affects the processing of counterintuitive concepts. Psychology of Aesthetics, Creativity, and the Arts. find
  24. J. Berenike Herrmann. 2015. High on metaphor, low on simile? An examination of metaphor type in sub-registers of academic prose. In J. Berenike Herrmann and Tony Berber Sardinha, editors, Metaphor in Language, Cognition, and Communication, volume 4, pages 163–190. John Benjamins Publishing Company,… find
  25. Katrin Kohl, Marianna Bolognesi, and Ana Werkmann Horvat. 2020. The Creative Power of Metaphor. In Katrin Kohl, Rajinder Dudrah, Andrew Gosler, Suzanne Graham, Martin Maiden, and Wen-chin Ouyang, editors, Creative Multilingualism, 1 edition, pages 25–46. Open Book Publishers, Cambridge, UK. find
  26. Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. ArXiv:2407.21783 [cs].
  27. Zoltán Kövecses. 2010. Metaphor: a practical introduction, 2. edition edition. Oxford University Press, Oxford. find
  28. Zoltán Kövecses. 2020. Conceptual metaphor theory. In Elena Semino and Zsófia Demjén, editors, The Routledge handbook of metaphor and language, first issued in paperback edition, Routledge handbooks in linguistics, pages 13–27. Routledge, Taylor & Francis Group, London New York. find
  29. George Lakoff and Mark Johnson. 2003. Metaphors We Live By. University of Chicago Press, Chicago, IL. find
  30. Antonio Jr. Laverghetta, Tuhin Chakrabarty, Tom Hope, Jimmy Pronchick, Krupa Bhawsar, and Roger E. Beaty. 2025. How do Humans and Language Models Reason About Creativity? A Comparative Analysis. ArXiv:2502.03253 [cs]. — [arXiv:2502.03253](https://arxiv.org/abs/2502.03253) find
  31. Simone A. Luchini, Ibraheem Muhammad Moosa, John D. Patterson, Dan Johnson, Matthijs Baas, Baptiste Barbot, Iana Bashmakova, Mathias Benedek, Qunlin Chen, Giovanni E. Corazza, Boris Forthmann, Benjamin Goecke, Sameh Said-Metwaly, Maciej Karwowski, Yoed N. Kenett, Izabela Lebuda, Todd Lubart, Kirill… find
  32. Rowan Hall Maudslay and Simone Teufel. 2022. Metaphorical Polysemy Detection: Conventional Metaphor meets Word Sense Disambiguation. ArXiv:2212.08395 [cs]. find
  33. Mistral AI. 2025. Mistral-Small-3.2-24B-Instruct-2506. find
  34. Peter Organisciak, Selcuk Acar, Denis Dumas, and Kelly Berthiaume. 2023. Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity, 49:101356. find
  35. Pragglejaz Group. 2007. MIP: A Method for Identifying Metaphorically Used Words in Discourse. Metaphor and Symbol, 22(1):1–39. find
  36. Meihua Qian and Jonathan A Plucker. 2017. Creativity assessment. In Creativity and innovation, pages 223–234. Routledge. find
  37. Abdullah Al Rabeyah, Fabrício Góes, Marco Volpe, and Talles Medeiros. 2024. Do LLMs Agree on the Creativity Evaluation of Alternative Uses? ArXiv:2411.15560 [cs]. — [arXiv:2411.15560](https://arxiv.org/abs/2411.15560) find
  38. W. Gudrun Reijnierse, Christian Burgers, Tina Krennmayr, and Gerard J. Steen. 2018. DMIP: A Method for Identifying Potentially Deliberate Metaphor in Language Use. Corpus Pragmatics, 2(2):129–147. find
  39. Sebastian Reimann and Tatjana Scheffler. 2024. When is a Metaphor Actually Novel? Annotating Metaphor Novelty in the Context of Automatic Metaphor Detection. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 87–97, St. Julians, Malta. Association for Computational Linguist… find
  40. Sebastian Reimann and Tatjana Scheffler. 2025. Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection. In Proceedings of the 2nd Workshop on Analogical Abstraction in Cognition, Perception, and Language (Analogy-Angle II), pages 10–21, Vienna, Austria. Association for Com… find
  41. Mark A. Runco and Garrett J. Jaeger. 2012. The Standard Definition of Creativity. Creativity Research Journal, 24(1):92–96. Publisher: Routledge _eprint: https://doi.org/10.1080/10400419.2012.650092. — [doi:10.1080/10400419.2012.650092](https://doi.org/10.1080/10400419.2012.650092) ↗
  42. Seyedeh Hamideh Mozaffari. 2013. An Analytical Rubric for Assessing Creativity in Creative Writing. Theory and Practice in Language Studies, 3. find
  43. Alec Sánchez-Montero, Gemma Bel-Enguix, Sergio-Luis Ojeda-Trueba, and Gerardo Sierra. 2025. Prompting metaphoricity: Soft labeling with large language models in popular communication of science tweets in Spanish. In Proceedings of the 2nd Workshop on Analogical Abstraction in Cognition, Perception,… find
  44. Andreas Nolda. 2024. Wikivoyage-Korpus: Ko-rpusquellen der deutschen Sprachversion von Wikivoyage im TEI-Format. find
  45. Elena Semino and Gerard J. Steen. 2008. Metaphor in Literature. In Jr. Gibbs, Raymond W., editor, The Cambridge Handbook of Metaphor Omar Momen, Emilie Sitter, Berenike Herrmann, and Sina Zarrieß. 2026. Surprisal and metaphor novelty: Moderate correlations and divergent scaling effects. arXiv prepri… — [arXiv:2601.02015](https://arxiv.org/abs/2601.02015) find
  46. Emilie Sitter, Omar Momen, Florian Steig, J. Berenike Herrmann, and Sina Zarrieß. 2025. Annotating Spatial Descriptions in Literary and Non-Literary Text. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 308–325, Vienna, Austria. Association for Computational Linguisti… find
  47. Gerard J. Steen. 2017. Deliberate Metaphor Theory: Basic assumptions, main tenets, urgent issues. Intercultural Pragmatics, 14(1):1–24. Publisher: De Gruyter Mouton. find
  48. Gerard J. Steen, Aletta G. Dorst, J. Berenike Herrmann, Anna A. Kaal, and Tina Krennmayr. 2010a. Metaphor in usage. 21(4):765–796. Publisher: De Gruyter Mouton Section: Cognitive Linguistics. find
  49. Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 Technical Report. ArXiv:2407.10671 [cs]. find
  50. Fengying Ye, Shanshan Wang, Lidia S. Chao, and Derek F. Wong. 2025. Unveiling LLMs’ Metaphorical Understanding: Exploring Conceptual Irrelevance, Context Leveraging and Syntactic Influence. ArXiv:2510.04120 [cs]. — [arXiv:2510.04120](https://arxiv.org/abs/2510.04120) find
  51. Claire M. Zedelius, Caitlin Mills, and Jonathan W. Schooler. 2019. Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features. Behavior Research Methods, 51(2):879–894. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sitter-2026-decomposing-creativity-two-small-datasets/