Elamur Research⌕
☰
home / papers / 2026 / Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing

Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing

Paper · 2026 · Fengying Ye, Shanshan Wang, Lidia S. Chao et al.

Summary

LLMs score well on metaphor detection and interpretation benchmarks, but it is unclear whether those scores reflect real metaphor understanding. Ye et al. (University of Macau) do not compare models on accuracy. Instead, they run a diagnostic study along three axes. Spatial Analysis is a geometric probe. It places a model's interpretation in an embedding space and measures how far it falls from a local reference plane. Two human interpretations sharing the intended semantic attribute, plus a literal paraphrase, define that plane. Metaphorical Imagination asks models to propose metaphorical and literal substitutes for a word, with and without context, to see whether the associations stay fixed. Syntactic Shuffle distorts sentences from MUNCH and checks how detection responds.

The authors find three patterns:

  • Interpretations drift away from the intended attribute.
  • Lexical associations barely change with context. This may help with conventional metaphors but mislead on novel ones.
  • Syntactic irregularity often raises detection accuracy, which suggests models treat it as a shortcut cue.

The takeaway is that benchmark success may come from a mix of heuristic signals rather than integrated semantic understanding.

Key points

  • Data: the study relies on MUNCH (Tong et al., 2024) and Fig-QA (Liu et al., 2022), both publicly available in English. The authors note that generality to other languages is untested.
  • Alignment is quantified by two geometric measures: the perpendicular distance d_p from the plane spanned by {R_i, R′_i, S_i}, and an angle measure cos θ. Both depend on the chosen embedding model.
  • The perturbations are word-order shuffling, POS shuffle, random shuffle, and moving the metaphorical word to the beginning, middle or end of the sentence. POS shuffle produced higher accuracy than the unaltered sentences.
  • The absolute position of the metaphorical word mattered less than the type of structural disruption. Random shuffle is treated only as a stress test; under it, V3-671B behaved unstably.
  • The Metaphorical Imagination probe covers only single-word metaphors. Multi-word and discourse-level metaphors remain future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China43 works · 2,368 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksIdioms & multiword expressions · 222 worksInterpretability · 149 works

Cite

Cite as Fengying Ye, Shanshan Wang, Lidia S. Chao and Derek F. Wong. 2026. Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1286

@inproceedings{ye2026probing,
  title = {{Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing}},
  author = {Fengying Ye, Shanshan Wang, Lidia S. Chao and Derek F. Wong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1286},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1286}
}

References

49 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037– 2050, D… find
  3. Marc Brysbaert and Boris New. 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41(4):977–990. find
  4. Man Yee Cheung, Chuan Luo, Choon Ling Sia, and Huaping Chen. 2009. Credibility of electronic word-of-mouth: Informational and normative determinants of on-line consumer recommendations. International journal of electronic commerce, 13(4):9–38. find
  5. Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021. MelBERT: Metaphor detection via contextualized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association f… find
  6. Kenneth Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29.
  7. William Croft. 1993. The role of domains in the interpretation of metaphors and metonymies. Walter de Gruyter, Berlin/New York Berlin, New York. find
  8. Paul V DiStefano, John D Patterson, and Roger E Beaty. 2024. Automatic scoring of metaphor creativity with large language models. Creativity Research Journal, pages 1–15. find
  9. Erik-Lân Do Dinh, Hannah Wieland, and Iryna Gurevych. 2018. Weeding out conventionalized metaphors: A corpus of novel metaphor annotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Lin… find
  10. Ellen Dodge, Jisup Hong, and Elise Stickles. 2015. MetaNet: Deep semantic automatic metaphor analysis. In Proceedings of the Third Workshop on Metaphor in NLP, pages 40–49, Denver, Colorado. Association for Computational Linguistics. find
  11. Mengshi Ge, Rui Mao, and Erik Cambria. 2022. Explainable metaphor identification inspired by conceptual metaphor theory. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 10681–10689. find
  12. Mengshi Ge, Rui Mao, and Erik Cambria. 2023. A survey on computational metaphor processing techniques: From identification, interpretation, generation to application. Artificial Intelligence Review, 56(Suppl 2):1829–1895. find
  13. Raymond W Gibbs Jr. 2008. The Cambridge handbook of metaphor and thought. Cambridge University Press. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  15. Pragglejaz Group. 2007. MIP: A method for identifying metaphorically used words in discourse. Metaphor and symbol, 22(1):1–39. find
  16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.
  17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  18. Nicholas Ichien, Dušan Stamenković, and Keith J Holyoak. 2024. Large language model displays emergent ability to interpret novel literary metaphors. Metaphor and Symbol, 39(4):296–309. find
  19. Kristin Janschewitz. 2008. Taboo, emotionally valenced, and emotionally neutral word norms. Behavior research methods, 40(4):1065–1074. find
  20. George Lakoff and Mark Johnson. 1980. Metaphors we live by. Chicago University Press. find
  21. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202…
  22. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  23. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452,…
  24. Rui Mao, Kai He, Claudia Ong, Qian Liu, and Erik Cambria. 2024. MetaPro 2.0: Computational metaphor processing on the effectiveness of anomalous language modeling. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9891– 9908, Bangkok, Thailand. Association for Computation… find
  25. John Philip McCrae, Alexandre Rademaker, Ewa Rudnicka, and Francis Bond. 2020. find
  26. English WordNet 2020: Improving and extending a WordNet for English using an open-source methodology. In Proceedings of the LREC 2020 Workshop on Multimodal Wordnets (MMW2020), pages 14–19, Marseille, France. The European Language Resources Association (ELRA). find
  27. Maria Montefinese, Ettore Ambrosini, Beth Fairfield, and Nicola Mammarella. 2014. The adaptation of the affective norms for english words (ANEW) for italian. Behavior research methods, 46:887–903. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  29. Sunny Rai and Shampa Chakraverty. 2020. A survey on computational metaphor processing. ACM Computing Surveys (CSUR), 53(2):1–37. find
  30. Meghdut Sengupta, Roxanne El Baff, Milad Alshomary, and Henning Wachsmuth. 2024. Analyzing the use of metaphors in news editorials for political framing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  31. Yujie Shao, Xinrong Yao, Xingwei Qu, Chenghua Lin, Shi Wang, Wenhao Huang, Ge Zhang, and Jie Fu. 2024. CMDAG: A Chinese metaphor dataset with annotated grounds as CoT for boosting metaphor generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re… find
  32. Gerard J Steen, Aletta G Dorst, J Berenike Herrmann, Anna A Kaal, Tina Krennmayr, and Tryntje Pasma. 2010. A method for linguistic metaphor identification: From MIP to MIPVU. John Benjamins Publishing Company. find
  33. Chang Su, Xingyue Wang, Yongzhu Chang, Kechun Wu, and Yijiang Chen. 2025. Metaphor generation based on noval evaluation method. Neurocomputing, 611:128651. find
  34. Karen Sullivan. 2013. Frames and Constructions in Metaphoric Language, volume 14. John Benjamins Publishing. find
  35. Yuan Tian, Nan Xu, and Wenji Mao. 2024. A theory guided scaffolding instruction framework for LLM-enabled metaphor reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pag… find
  36. Michael Toker, Oren Mishali, Ophir Münz-Manor, Benny Kimelfeld, and Yonatan Belinkov. 2024. A dataset for metaphor detection in early medieval Hebrew poetry. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages…
  37. Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor understanding challenge dataset for LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3517–3536, Bangkok, Thailand. Association for Computa…
  38. Lennart Wachowiak and Dagmar Gromann. 2023. Does GPT-3 grasp metaphors? identifying metaphor mappings with generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1018–1032, Toronto, Canada. Association for… find
  39. Mingyu Wan, Kathleen Ahrens, Emmanuele Chersoni, Menghan Jiang, Qi Su, Rong Xiang, and Chu-Ren Huang. 2020. Using conceptual norms for metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 104–109, Online. Association for Computational Linguistics. find
  40. Jie Wang, Jin Wang, and Xuejie Zhang. 2024a. Chinese metaphor recognition using a multi-stage prompting large language model. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 234–246. Springer. find
  41. Shanshan Wang, Derek Wong, Jingming Yao, and Lidia Chao. 2024b. What is the best way for ChatGPT to translate poetry? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14025– 14043. find
  42. Shanshan Wang, Junchao Wu, Fengying Ye, Derek F. Wong, Jingming Yao, and Lidia S. Chao. 2025. Benchmarking the detection of LLMs-generated Modern Chinese poetry. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 9533–9552. Association for Computational Linguistics. find
  43. Anna Wegmann and Dong Nguyen. 2021. Does it capture STEL? a modular, similarity-based linguistic style evaluation framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7109–7130, Online and Punta Cana, Dominican Republic. Association for Computat… find
  44. Yorick Wilks. 1975. A preferential, pattern-seeking, semantics for natural language inference. Artificial intelligence, 6(1):53–74. find
  45. Yanzhi Xu, Yueying Hua, Shichen Li, and Zhongqing Wang. 2024. Exploring chain-of-thought for multimodal metaphor detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 91–101, Bangkok, Thailand. Association for Computation… find
  46. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  47. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024b. Harnessing the power of LLMs in practice: A survey on Chat-GPT and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32. find
  48. Fengying Ye, Yanming Sun, Runzhe Zhan, Lidia S. Chao, Zheqi Zhang, and Derek F. Wong. 2026. G-idiomalign: A gloss-pivoted benchmark for cross-lingual idiom alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, Califo…
  49. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-probing-semantic-alignment-lexical-invariance/