Elamur Research⌕
☰
home / papers / 2026 / DAMETA: An LLM Benchmark for Danish Metaphor Interpretation with Systematically Varied Distractors

DAMETA: An LLM Benchmark for Danish Metaphor Interpretation with Systematically Varied Distractors

Summary

LLMs are trained mostly on English, so it is unclear how well they understand figurative meaning in smaller languages, where many metaphors carry local cultural associations. Existing metaphor evaluations also rarely go beyond detection. DAMETA addresses this for Danish with a manually built multiple-choice benchmark of 900 items. Each item pairs a sentence containing one metaphorically used word with four human-written paraphrases: the correct reading and three deliberately designed wrong ones. These are a literal (usually concrete) misreading, an incorrect figurative (usually abstract) reading, and an option that contradicts the intended sense. Items come from the DDO dictionary, the DaFig corpus and newspaper culture reviews. Each is tagged for conventionality and source domain. Seven language models were evaluated, and humans answered a small subset for comparison. Error patterns depend on the prompt. When told the word is a metaphor, models most often pick the wrong figurative option. Without that hint, the literal distractor misleads them most. Because each distractor type targets a distinct failure mode, the benchmark shows why models misinterpret metaphors, not just how often.

Key points

  • 900 instances, each with one correct paraphrase and three distractor types: false literal, false figurative and false contradictory.
  • Sources: about 62% from the DDO (Den Danske Ordbog), about 30% from the DaFig corpus, and the remainder from ad-hoc metaphors found manually in Danish newspaper culture reviews.
  • Metadata includes a three-level conventionality scale (lexicalised, implicit, ad-hoc) and dictionary-derived source domains such as military, gastronomy, health and meteorology.
  • Seven proprietary and open models were tested, and 5% of the data was also answered by non-expert humans. The dominant distractor shifts from figurative to literal depending on whether the prompt reveals that the word is metaphorical.
  • Released on GitHub (password-protected) within the kuhumcst danish-semantic-reasoning-benchmark repository. It is described as the first manually annotated single-word metaphor interpretation benchmark for Danish.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5myuqkhnpm7e

@inproceedings{schneidermann2026dameta,
  title = {{DAMETA: An LLM Benchmark for Danish Metaphor Interpretation with Systematically Varied Distractors}},
  author = {Schneidermann, Nina Skovgaard and Nimb, Sanni and Norman, Nathalie Carmen Hau and Olsen, Sussi and Pedersen, Bolette},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5myuqkhnpm7e},
  url = {https://doi.org/10.63317/5myuqkhnpm7e}
}

References

23 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between Chat-GPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, United States. Association for Computational Linguistics (ACL). Publisher Copyright: © 2023 Association for Computational Linguistics.; 1st Workshop on Cross-Cultural Considerations in NLP, C3NLP 2023; Conference date: 05-05-2023. find
  2. Francesca MM Citron and Adele E Goldberg. 2014. Metaphorical sentences are more emotionally engaging than their literal counterparts. Journal of Cognitive Neuroscience, 26(11):2585–2595. ↗
  3. Iulia Coms , a, Julian Eisenschlos, and Srini Narayanan. 2022. MiQA: A Benchmark for Inference on Metaphorical Questions. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 373–381, Online only. Association for Computational Linguistics. find
  4. Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. 2024. The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding. find
  5. Pragglejaz Group. 2007. MIP: A Method for Identifying Metaphorically Used Words in Discourse. Metaphor and Symbol, 22(1):1–39. ↗
  6. Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Indra Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and Multi-cultural Figurative Language Understanding. Annual Meeting of the Association for Computational Linguistics. ↗
  7. George Lakoff and Mark Johnson. 1980a. Conceptual Metaphor in Everyday Language. The journal of Philosophy, 77(8):453–486. ↗
  8. George Lakoff and Mark Johnson. 1980b. Metaphors We Live By. University of Chicago Press. find
  9. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the Ability of Language Models to Interpret Figurative Language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452, Seattle, United States. Association for Computational Linguistics. ↗
  10. Rowan Hall Maudslay, Simone Teufel, Francis Bond, and James Pustejovsky. 2024. ChainNet: Structured metaphor and metonymy in Word-Net. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2984–2996, Torino, Italia. ELRA and ICCL. find
  11. Jamshid Mozafari, Bhawna Piryani, and Adam Jatowt. 2025. Wrong Answers Can Also Be Useful: Plausibleqa -A Large-Scale QA Dataset with Answer Plausibility Scores. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  12. Susan Nacey, W Gudrun Reijnierse, Tina Krennmayr, and Aletta G Dorst. 2019. Metaphor Identification in Multiple Languages. John Benjamins Publishing Company. ↗
  13. Dan Saattrup Nielsen. 2023. Scandeval: A benchmark for Scandinavian Natural Language Processing. Proceedings of Nodalida 2023, The Faroe Islands. ↗
  14. Ira A Noveck, Maryse Bianco, and Alain Cas-try. 2001. The costs and benefits of metaphor. Metaphor and Symbol, 16(1-2):109–121. ↗
  15. Bolette S. Pedersen, Nathalie Sørensen, Sanni Nimb, Dorte Haltrup Hansen, Sussi Olsen, and Ali Al-Laith. 2025. Evaluating LLM-generated explanations of metaphors – a culture-sensitive study of Danish. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 470–479, Tallinn, Estonia. University of Tartu Library. find
  16. Bolette Sandford Pedersen, Nathalie C Hau Sørensen, Sussi Olsen, and Sanni Nimb. 2024. Evaluering af sprogforståelsen i danske sprogmodeller - med udgangspunkt i semantiske ordbøger. NyS, Nydanske Sprogstudier, pages 8–40. find
  17. Elisa Sanchez-Bayona and Rodrigo Agerri. 2024. Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation. (arXiv:2404.07053). ↗
  18. Elisa Sanchez-Bayona and Rodrigo Agerri. 2025. Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17462– 17477. Association for Computational Linguistics. find
  19. Nathalie Hau Sørensen, Sanni Nimb, Agnes Aggergaard Mikkelsen, and Jonas Jensen. 2025. The Danish idiom dataset: A collection of 1000 Danish idioms and fixed expressions. In Proceedings of the 1st Workshop on Nordic-Baltic Responsible Evaluation and Alignment of Language Models (NB-REAL 2025), pages 55–63, Tallinn, Estonia. The University of Tartu Library. find
  20. Gerard Steen, Lettie Dorst, J Berenike Herrmann, Anna Kaal, Tina Krennmayr, and Tryntje Pasma. 2019. MIPVU: A manual for identifying metaphor-related words. Metaphor identification in multiple languages: MIPVU around the world, 22:23. ↗
  21. Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What’s the Meaning of Superhuman Performance in Today’s NLU? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12471–12491. Association for Computational Linguistics. find
  22. Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor Understanding Challenge Dataset for LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3517–3536. Association for Computational Linguistics. find
  23. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust Chat-GPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927, Singapore. Association for Computational Linguistics. 9. Language Resource References Det Danske Sprog- og Litteraturselskab. 2024. Den Danske Ordbog. (September 2024). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/schneidermann-2026-dameta-llm-benchmark-danish-metaphor/