Elamur Research⌕
☰
home / papers / 2025 / Evaluating LLM-Generated Explanations of Metaphors – A Culture-Sensitive Study of Danish

Evaluating LLM-Generated Explanations of Metaphors – A Culture-Sensitive Study of Danish

Paper · 2025 · Bolette S. Pedersen, Nathalie Sørensen, Sanni Nimb et al.

Summary

This study asks whether large language models can explain Danish metaphors that have no English counterpart as well as those shared with English. The authors built a pilot set of 150 Danish metaphors and idioms drawn from The Danish Dictionary, using its main-sense/subsense structure and its metaphor labels. Each item was marked as culture-specific or cross-cultural, and English-speaking informants checked that split. The two strongest Danish chatbots according to Scandeval, ChatGPT and Llama, were then prompted to explain every item. Prompts were given in Danish and in English, with and without a textual context. Native speakers rated all outputs on a four-level scale. Both models did clearly better on metaphors that also exist in English. This suggests they carry English meanings over to Danish expressions, which produces fluent but wrong or invented explanations. The most frequent failure was connotation: negative, sarcastic or sexual undertones were softened or missed entirely. The authors see this as a risk to linguistic and cultural diversity as such chatbots spread through society.

Key points

  • The dataset has 150 items: 75 single-word metaphors (e.g. sejle) and 75 metaphorical idioms (e.g. høste frugterne). Within each type, 50 are culture-specific and 25 are cross-cultural.
  • Items come from The Danish Dictionary (Den Danske Ordbog). Its 152 unpublished domain labels helped group items by source domain, such as farming and nautical terms. Idioms took the domain of their central lemmas.
  • Unclear cases were labelled culture-specific whenever English had no exact match. Examples are studehandel (English uses "horse trading") and myreflittig (English uses "busy as a bee").
  • All 1,200 model explanations were rated by humans: 150 items × 2 models × 2 prompt languages × with/without context.
  • Typical failures were turning derogatory terms positive, such as havelåge and skærveknuser for old bikes, tågehorn and sejle sin egen sø, and ignoring sexual meanings. The data is released in the danish-semantic-reasoning-benchmark GitHub repository.

Interesting ideas

  • I1. Pair the hard culture-specific items with a smaller set of items shared with English, so that any drop in quality can be traced to cultural distance rather than to metaphor difficulty in general.
  • I2. Harvest figurative senses directly from a dictionary's subsense hierarchy and domain tags, and let multiword idioms take the domain of their head lemmas.
  • I3. Judge connotation separately from literal gist, because models often get the meaning roughly right while reversing its sentiment.
  • I4. Cross prompt language with presence or absence of context in a small factorial design, so each factor's effect on explanation quality can be isolated.
  • I5. When the cultural status of an item is unclear, apply a strict rule (no exact equivalent means culture-specific) rather than leave it ambiguous.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluations could include a matched "easy twin" set as a negative control, showing whether a failure comes from the construction itself or from rare or idiomatic wording.
  • I2 (world): figurative senses could be compiled into the ontology with explicit links to their literal source sense and domain, so readers can tell when a word like "sail" is not about boats.
  • I3 (frames): the irony and implicature layer could mark evaluative polarity on figurative frames and test it on its own, since a positive reading of a derogatory idiom is a distinct error.
  • I4 (learn): LLM teacher tiers could be probed the same way before their silver labels are trusted, keeping only conditions where the teacher is stable.
  • I5 (learn): a fixed rule for borderline items keeps silver-data labels consistent and evaluation splits reproducible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations

Also on the byline: Det Danske Sprog- og Litteraturselskab.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksIdioms & multiword expressions · 217 works

Cite

Cite as Bolette S. Pedersen, Nathalie Sørensen, Sanni Nimb, Dorte Haltrup Hansen, Sussi Olsen and Ali Al-Laith. 2025. Evaluating LLM-Generated Explanations of Metaphors – A Culture-Sensitive Study of Danish. Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pedersen2025evaluating,
  title = {{Evaluating LLM-Generated Explanations of Metaphors – A Culture-Sensitive Study of Danish}},
  author = {Bolette S. Pedersen, Nathalie Sørensen, Sanni Nimb, Dorte Haltrup Hansen, Sussi Olsen and Ali Al-Laith},
  booktitle = {Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.nodalida-1.51/}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037–2050, Dublin, Ireland. Association for Computational Linguistics. find
  2. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Vı́ctor Gutiérrez-Basulto, Yazmı́n Ibáñez-Garcı́a, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Ousidhoum, Jose Camacho-Collados, and Alice Oh. 2024. BLEnD: A benchmark for LLMs on everyday knowledge in diverse cultures and languages.
  3. Naveen Badathala, Abisek Rajakumar Kalarani, Tejpalsingh Siledar, and Pushpak Bhattacharyya. 2023. A match made in heaven: A multi-task framework for hyperbole and metaphor detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 388–401, Toronto, Canada. Association for Computational Linguistics. find
  4. Susan Nacey, W Gudrun Reijnierse, Tina Krennmayr, and Aletta G Dorst. 2019. Metaphor Identification in Multiple Languages. John Benjamins Publishing Company. ↗
  5. Valentina Bambini, Paolo Canal, Donatella Resta, and Mirko Grimaldi. 2019. Time course and neurophysiological underpinnings of metaphor in literary context. Discourse Processes, 56(1):77–97. ↗
  6. Dan Saattrup Nielsen. 2023. Scandeval: A benchmark for Scandinavian natural language processing. Proceedings of Nodalida 2023, The Faroe Islands. ↗
  7. Christian Burgers, Kiki Y Renardel de Lavalette, and Gerard J Steen. 2018. Metaphor, hyperbole, and irony: Uses in isolation and in combination in written discourse. Journal of Pragmatics, 127:71–83. ↗
  8. Dan Saattrup Nielsen. 2024. Status på Scandinavian embedding benchmark (seb). Slides from Bench-markworkshop in The Danish Agency for Digital Government, September 20, 2024. find
  9. Ira A Noveck, Maryse Bianco, and Alain Castry. 2001. The costs and benefits of metaphor. Metaphor and Symbol, 16(1-2):109–121. ↗
  10. Bolette Sandford Pedersen, Nathalie C Hau Sørensen, Sussi Olsen, and Sanni Nimb. 2024. Evaluering af sprogforståelsen i danske sprogmodeller - med udgangspunkt i semantiske ordbøger. NyS, Ny-danske Sprogstudier, pages 8–40. find
  11. Paolo Pedinotti, Eliana Di Palma, Ludovica Cerini, and Alessandro Lenci. 2021. A howling success or a working sea? Testing what BERT knows about metaphors. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192–204. find
  12. Ciyuan Peng, Dang Thinh Vu, and Jason J Jung. 2021. Knowledge graph-based metaphor representation for literature understanding. Digital Scholarship in the Humanities, 36(3):698–711. ↗
  13. Vinodkumar Prabhakaran, Marek Rei, and Ekaterina Shutova. 2021. How metaphors impact political discourse: A large-scale topic-agnostic study using neural metaphor detection. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 503–512. find
  14. Elisa Sanchez-Bayona and Rodrigo Agerri. 2024. Meta4xnli: A crosslingual parallel corpus for metaphor detection and interpretation. arXiv preprint arXiv:2404.07053. ↗
  15. Andrew Szot, Bogdan Mazoure, Harsh Agrawal, R Devon Hjelm, Zsolt Kira, and Alexander T Toshev. 2024. Grounding multimodal large language models in actions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  16. Lennart Wachowiak and Dagmar Gromann. 2023. Does GPT-3 grasp metaphors? Identifying metaphor mappings with generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1018–1032. find
  17. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust Chat-GPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927, Singapore. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pedersen-2025-evaluating-llm-generated-explanations-metaphors/