Elamur Research⌕
☰
home / papers / 2026 / Artful Writing, Authentic Emotions: Distinguishing Human-Written from LLM-Generated Metaphors by Annotation and Classification

Artful Writing, Authentic Emotions: Distinguishing Human-Written from LLM-Generated Metaphors by Annotation and Classification

Summary

The paper asks whether novel metaphors written by people differ in detectable ways from those produced by LLMs, and whether people or machines can tell them apart. The authors built two small sets of human-written novel metaphors. One comes from poetry (Persian, German and a few English sources) and one from science communication. Every item was normalized into the "X is like a Y" analogy form so that differences in syntax would not confound the comparison. For each target concept, four LLMs wrote their own analogy, which gave 200 items in total. Crowdworkers rated each item on several dimensions, including the emotions conveyed in the poetry set. Classifiers and zero-shot chatbots were then tested on detecting authorship. Human raters could not identify the origin of a metaphor. Human poetic metaphors did stand out in one respect: they conveyed emotion more effectively. Automated detectors, and especially some LLMs, identified the human texts with high accuracy. This suggests the two sources differ structurally in ways people do not notice.

Key points

  • Dataset and models: The data has 20 human poetic and 20 human scientific analogies. Each is matched with generations from Claude Sonnet 4.5, ChatGPT 5, Gemini 2.5 and Mistral, giving 200 items. The annotated dataset is released on OSF.
  • Annotation design: Each analogy received 10 crowd ratings on 1–5 scales. These covered writing quality, creativity, comprehensibility and perceived machine authorship. Poetry items also received emotion labels from six categories (love, pain, longing, fear, pleasure, regret). Science items were also rated for helpfulness and expertise.
  • Human detection: Annotators were near chance at telling human from machine authorship. Accuracy was 0.42 on poetry and 0.41 on science.
  • Classifier results on poetry: A text-based classifier reached 0.83 accuracy, compared with a length baseline of 0.73 and an annotation-based classifier at 0.70. On science metaphors, text length was the most dependable cue.
  • Zero-shot LLM detection: Claude and Gemini scored 1.00 on poetry and ChatGPT scored about 0.93, while Mistral fell below chance at 0.35. On science, Gemini reached 0.875 and Claude 0.725.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Language models & pretraining Sentiment & emotion Idioms & multiword expressions

Institutions

🎓Universität HamburgHamburg · Germany33 works · 4,538 citations

Cite

Cite as Michaela Regneri, Nooshin Aghajari and Thomas Kroedel. 2026. Artful Writing, Authentic Emotions: Distinguishing Human-Written from LLM-Generated Metaphors by Annotation and Classification. Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2q6dt9in7fm5

@inproceedings{regneri2026artful,
  title = {{Artful Writing, Authentic Emotions: Distinguishing Human-Written from LLM-Generated Metaphors by Annotation and Classification}},
  author = {Michaela Regneri, Nooshin Aghajari and Thomas Kroedel},
  booktitle = {Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026},
  year = {2026},
  doi = {10.63317/2q6dt9in7fm5},
  url = {https://doi.org/10.63317/2q6dt9in7fm5}
}

References

35 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. versus human-written medical student essays: Semirandomized controlled study. JMIR Med Educ, 11:e62779. find
  2. Antoine Bellemare-Pepin, François Lespinasse, Philipp Thölke, Yann Harel, Kory Mathewson, Jay A. Olson, Yoshua Bengio, and Karim Jerbi. 2026. Divergent creativity in humans and large language models. Scientific Reports, 16(1):1279. find
  3. Matteo Fuoli, Weihang Huang, Jeannette Littlemore, Sarah Turner, and Ellen Wilding. 2025. Metaphor identification using large language models: A comparison of rag, prompt engineering, and fine-tuning. arXiv. Preprint. find
  4. Yuri Bizzoni and Shalom Lappin. 2019. The effect of context on metaphor paraphrase aptness judgments. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 165– 175, Gothenburg, Sweden. Association for Computational Linguistics. find
  5. Jonathan Gordon, Jerry Hobbs, Jonathan May, Michael Mohler, Fabrizio Morbini, Bryan Rink, Marc Tomlinson, and Suzanne Wertheim. 2015. A corpus of rich metaphor annotation. In Proceedings of the Third Workshop on Metaphor in NLP, pages 56– 66, Denver, Colorado. Association for Computational Linguisti… find
  6. Brian F. Bowdle and Dedre Gentner. 2005. The career of metaphor. Psychological review, 112 1:193–216. find
  7. Hessel Haagsma and Johannes Bjerva. 2016. Detecting novel metaphor using selectional preference information. In Proceedings of the Fourth Workshop on Metaphor in NLP, pages 10–17. find
  8. Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021. Melbert: Metaphor detection via contextualized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association f… find
  9. Dirk Hovy, Shashank Srivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proceedings of the First Workshop on Metaphor in NLP, pages 52–57, Atlanta, Georgia. Association for Computati… find
  10. Erik-Lân Do Dinh, Hannah Wieland, and Iryna Gurevych. 2018. Weeding out conventionalized metaphors: A corpus of novel metaphor annotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Lin… find
  11. Nicholas Ichien, Dušan Stamenković, and Keith J. Holyoak. 2024. Large language model displays emergent ability to interpret novel literary metaphors. Metaphor and Symbol, 24(4):296–309. find
  12. Ellen Dodge, Jisup Hong, and Elise Stickles. 2015. MetaNet: Deep semantic automatic metaphor analysis. In Proceedings of the Third Workshop on Metaphor in NLP, pages 40–49, Denver, Colorado. Association for Computational Linguistics. find
  13. Hyeju Jang, Seungwhan Moon, Yohan Jo, and Carolyn Rosé. 2015. Metaphor detection in discourse. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 384–392, Prague, Czech Republic. Association for Computational Linguistics. find
  14. Berin Doru, Christoph Maier, Johanna Sophie Busse, Thomas Lücke, Judith Schön-hoff, Elena Enax-Krumova, Steffen Hessler, Maria Berger, and Marianne Tokic. 2025. Detecting artificial intelligence–generated Kaidi Jia and Rongsheng Li. 2024. MD-PK: Metaphor detection via prompt learning and knowledge d… find
  15. Kaidi Jia, Yanxia Wu, Ming Liu, and Rongsheng Li. 2025. Curriculum-style data augmentation for llm-based metaphor detection. Preprint. find
  16. Conference on Computational Linguistics, pages 3722–3736. find
  17. William Orwig, Emma R. Edenbaum, Joshua D. Greene, and Daniel L. Schacter. 2024. The language of creativity: Evidence from humans and large language models. The Journal of Creative Behavior, 58(1):128–136. find
  18. Jeongyeon Kim, Sangho Suh, Lydia B Chilton, and Haijun Xia. 2023. Metaphorian: Leveraging large language models to support extended metaphor creation for science writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference, DIS ’23, pages 115–135, New York, NY, USA. Association f… find
  19. Natalie Parde and Rodney D. Nielsen. 2018. A corpus of metaphor novelty scores for syntactically-related word pairs. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, pages 1535–1540. find
  20. Oliver Kramer. 2025. Conceptual metaphor theory as a prompting paradigm for large language models. Preprint. find
  21. Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192–204, Punta Cana, Dominican Re… find
  22. George Lakoff and Mark Johnson. 2003. Metaphors we live by. University of Chicago Press, Chicago. Originally published: Chicago : University of Chicago Press, 1980. find
  23. Yujie Lin, Jingyao Liu, Yan Gao, Ante Wang, and Jinsong Su. 2024. A dual-perspective metaphor detection framework using large language models. Preprint. find
  24. Brian Porter and Edouard Machery. 2024. Ai-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Scientific Reports, 14(1):26133. find
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  26. Elisa Sanchez-Bayona and Rodrigo Agerri. 2025. Metaphor and large language models: When surface features matter more than deep understanding. Preprint. find
  27. Sofia Lugli and Carlo Strapparava. 2024. Multimodal chain-of-thought prompting for metaphor generation. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 523–530, Pisa, Italy. CEUR Workshop Proceedings. find
  28. Sadat Shahriar, Navid Ayoobi, and Arjun Mukherjee. 2025. The erosion of LLM signatures: Can we still distinguish human and LLM-generated scientific ideas after iterative paraphrasing? In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Lang… find
  29. Michael Mohler, Mary Brunson, Bryan Rink, and Marc Tomlinson. 2016. Introducing the lcc metaphor datasets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pages 4221–4227. find
  30. Xincheng Shou, Xiaoxi Huang, and Wenlong Xi. 2024. Conceptual metaphor theory guides gans for generating metaphors and interpretations. IEEE Access, PP(99):1–1. find
  31. Arthur Neidlein, Philipp Wiesenbach, and Katja Markert. 2020. An analysis of language models for metaphor recognition. In Proceedings of the 28th International Ekaterina Shutova and Simone Teufel. 2010. Metaphor corpus annotated for source - target domain mappings. In Proceedings of the Seventh Inte… find
  32. Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. 2021. Metaphor generation with conceptual mappings. Preprint. find
  33. Shanshan Wang, Junchao Wu, Fengying Ye, Derek F. Wong, Jingming Yao, and Lidia S. Chao. 2025. Benchmarking the detection of LLMs-generated Modern Chinese poetry. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9533–9552, Suzhou, China. Association for Computational Li… find
  34. Olga Zamaraeva, Dan Flickinger, Francis Bond, and Carlos Gómez-Rodríguez. 2025. Comparing LLM-generated and human-authored news text using formal syntactic theory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9041–9060, Vie… find
  35. Omnia Zayed, John P. McCrae, and Paul Buitelaar. 2019. Crowd-sourcing a high-quality dataset for metaphor identification in tweets. In 2nd Conference on Language, Data and Knowledge (LDK 2019), OpenAccess Series in Informatics (OASIcs), pages 10:1– 10:17. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/regneri-2026-artful-writing-authentic-emotions-distinguishing/