Elamur Research⌕
☰
home / papers / 2025 / Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions

Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions

Paper · 2025 · Meghdut Sengupta, Maximilian Muschalik, Fabian Fumagalli et al.

Summary

The paper asks whether LLMs can do natural language inference when the hypothesis contains an implicit metaphor, and whether naming the underlying conceptual metaphor helps. Following Lakoff and Johnson, a conceptual metaphor is a mapping from a source domain to a target domain. The authors took the 1500 metaphorical pairs of the FLUTE corpus and added source and target domain labels, creating FLUTE.st. The labels were proposed by GPT-4 through in-context prompting and then rated by human experts.

They then ran five open LLMs in zero-shot and 10-shot settings, giving each model every combination of premise, hypothesis, source domain, target domain and explanation. To measure how much each input part matters, they framed the setup as a cooperative game and computed Shapley values and pairwise Shapley interactions. Explanations reliably raise F1. In about 70% of experiments, however, the best score also requires adding one or both domains. This suggests that explicit conceptual-metaphor knowledge helps models reason over figurative text.

Key points

  • FLUTE.st: covers 750 entailment and 750 contradiction metaphor pairs from FLUTE. Only the hypotheses are metaphorical. Domain vocabulary is drawn from the inventory of Gordon et al. (2015).
  • Annotation process: GPT-4 and Mistral-7B-Instruct-v0.2 were prompted 10-shot under three prompt designs (no metaphor span, with span, span plus rationale), giving six variants. Two annotators rated 200 samples from each variant on a four-level scale. The chosen variant was GPT-4 without span or rationale. Its agreement was low, with Cohen's κ of 0.325 for source and 0.323 for target domains, which reflects how subjective the task is.
  • Models and baseline scores: Mistral-7B, Llama-3.2-3B, Gemma-2-27B-IT, DeepSeek-R1-Distill-Qwen-32B and Llama-3.3-70B-Instruct were tested. With premise and hypothesis alone, F1 ranges from 0.632 to 0.927, and the full input reaches up to 0.969. Significance was tested with Wilcoxon rank-sum and Bonferroni correction.
  • Two explanation games: a local game explains a single prediction, with phrase chunks, the domains and the explanation as players. A global game uses the domains and explanation as players and corpus-level F1 as the payout. Premises and hypotheses were split into at most 9 NP/VP/PP chunks with the nltk chunker, giving at most 12 players and 690 samples per experiment.
  • Error analysis and cost:
    • Zero-shot Llama-3.2-3B is hurt by the domains and especially by the explanation.
    • The Shapley analysis is expensive: roughly 40 hours on an A100 for each quarter of the data (about 172 instances).

Interesting ideas

  • I1. Instead of measuring one ablation at a time, score every subset of auxiliary inputs and attribute the gains with Shapley values and interactions; positive interactions reveal complementary inputs, negative ones reveal redundancy.
  • I2. Chunking the input into phrases rather than tokens keeps the 2^n coalition space tractable while keeping attribution units linguistically meaningful.
  • I3. Annotate a metaphor with an explicit source→target domain pair as a structured label, so a reasoner can reinterpret figurative wording literally before inference.
  • I4. Generate candidate labels with several LLM prompt variants, have experts grade a fixed random sample on a graded yes/weak-yes/weak-no/no scale, and keep the variant with the best inter-annotator agreement.
  • I5. Separate a local game, which explains a single prediction, from a global game, which explains a metric over the dataset; the two views can disagree and expose failure modes like unhelpful explanations in small zero-shot models.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treat optional knowledge sources (WordNet senses, VerbNet roles, coreference links) as players and compute which ones, and which pairs, actually lift answer accuracy on the evaluation sets.
  • I2 (tree): Use UD subtrees or phrase spans from the parse as masking units when explaining why a downstream decision changed, instead of masking single tokens.
  • I3 (frames): Add a metaphor-mapping slot to frames (e.g. "torpedoed" read as ATTACK in a debate context) so a figurative verb maps to the role structure of its target-domain sense.
  • I4 (learn): Apply this as a selection step among LLM teacher tiers, choosing which prompt or teacher produces silver frame or sense labels by graded human spot checks.
  • I5 (answer): Report both a per-question attribution of which world facts drove an answer and a corpus-level attribution of which components raise test scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksIdioms & multiword expressions · 217 works

Cite

Cite as Meghdut Sengupta, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, Eyke Hüllermeier, Debanjan Ghosh and Henning Wachsmuth. 2025. Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.942

@inproceedings{sengupta2025investigating,
  title = {{Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions}},
  author = {Meghdut Sengupta, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, Eyke Hüllermeier, Debanjan Ghosh and Henning Wachsmuth},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.942},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.942}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  2. Sebastian Bordt and Ulrike von Luxburg. 2023. From Shapley Values to Generalized Additive Models and back. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 709–745. ↗
  3. Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. 2021. Figurative language in recognizing textual entailment. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3354–3361, Online. Association for Computational Linguistics.
  4. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  5. Iulia Coms , a, Julian Eisenschlos, and Srini Narayanan. 2022. MiQA: A benchmark for inference on metaphorical questions. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 373–381, Online only. Association for Computational Linguistics. find
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. ↗
  7. Alexander Vladislavovitch Dmitrijev, Elena Sergeevna Krupnova, and Anastasia Aleksandrovna Protopopova. 2024. Metaphors and analogies in the context of large language models. In International Conference on Professional Culture of the Specialist of the Future, pages 326–341. Springer. find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  9. Fabian Fumagalli, Maximilian Muschalik, Eyke Hüllermeier, Barbara Hammer, and Julia Herbinger. 2025. Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory. In 28th International Conference on Artificial Intelligence and Statistics (AISTATS). ↗
  10. Torda Gábor. 2014. Metaphors in everyday english. find
  11. Giuseppe Gallipoli and Luca Cagliero. 2025. It is not a piece of cake for GPT: Explaining textual entailment recognition in the presence of figurative language. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9656–9674, Abu Dhabi, UAE. Association for Computational Linguistics.
  12. Jonathan Gordon, Jerry Hobbs, Jonathan May, Michael Mohler, Fabrizio Morbini, Bryan Rink, Marc Tomlinson, and Suzanne Wertheim. 2015. A corpus of rich metaphor annotation. In Proceedings of the Third Workshop on Metaphor in NLP, pages 56–66, Denver, Colorado. Association for Computational Linguistics. find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  15. Zoltán Kövecses. 2010. Metaphor, language, and culture. DELTA: Documentação de Estudos em Lingüística Teórica e Aplicada, 26:739–757. ↗
  16. George Lakoff and Mark Johnson. 2003. Metaphors We Live By. University of Chicago Press. ↗
  17. Chee Wee (Ben) Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 VUA metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56–66, New Orleans, Louisiana. Association for Computational Linguistics.
  18. Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, and Eyke Hüllermeier. 2024. shapiq: Shapley interactions for machine learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). ↗
  19. Elisa Sanchez-Bayona and Rodrigo Agerri. 2024. Meta4xnli: A crosslingual parallel corpus for metaphor detection and interpretation. arXiv preprint arXiv:2404.07053. ↗
  20. Meghdut Sengupta, Milad Alshomary, Ingrid Scharlau, and Henning Wachsmuth. 2023. Modeling highlighting of metaphors in multitask contrastive learning paradigms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4636– 4659, Singapore. Association for Computational Linguistics. find
  21. Meghdut Sengupta, Milad Alshomary, and Henning Wachsmuth. 2022. Back to the roots: Predicting the source domain of metaphors using contrastive learning. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 137–142, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  22. Meghdut Sengupta, Roxanne El Baff, Milad Alshomary, and Henning Wachsmuth. 2024. Analyzing the use of metaphors in news editorials for political framing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3621–3631, Mexico City, Mexico. Association for Computational Linguistics. ↗
  23. L. S. Shapley. 1953. A Value for n-Person Games. In Contributions to the Theory of Games (AM-28), Volume II, pages 307–318. Princeton University Press. ↗
  24. Ekaterina Shutova, Simone Teufel, and Anna Korhonen. 2013. Statistical metaphor processing. Computational Linguistics, 39(2):301–353. ↗
  25. Inna V Skrynnikova. 2024. Interpreting metaphorical language: A challenge to artificial intelligence. Bulletin of Volgograd State University. Series 2: Linguistics, 23(5):99–107. ↗
  26. Shane Storks, Qiaozi Gao, and Joyce Y. Chai. 2020. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. Preprint, arXiv:1904.01172. ↗
  27. Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. 2014. Upset: Visualization of intersecting sets. IEEE Transactions on Visualization and Computer Graphics (InfoVis), 20(12):1983–1992. ↗
  28. Kevin Stowe, Nils Beck, and Iryna Gurevych. 2021. Exploring metaphoric paraphrase generation. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 323–336, Online. Association for Computational Linguistics. find
  29. Scott M. Lundberg, Gabriel G. Erion, Hugh Chen, Alex J. DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1):56–67. ↗
  30. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. Impli: Investigating nli models’ performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375– 5388.
  31. Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446. ↗
  32. Xiaoyu Tong, Ekaterina Shutova, and Martha Lewis. 2021. Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4673–4686, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sengupta-2025-investigating-impact-conceptual-metaphors-llm/