📄 Memorization or Reasoning? Exploring the Idiom Understanding of LLMs

Summary

The paper asks whether large language models understand idioms by recalling them or by working out their figurative meaning. It frames this as a multilingual question, since evaluation resources outside English and Chinese are scarce. The authors build MIDAS, a dataset of 64,660 idioms in English, German, Chinese, Korean, Arabic and Turkish. The idioms come from authoritative dictionaries and collections. GPT-4o cleaned the meanings (GPT-4o mini for English), and native speakers then checked them. Example sentences and surface variants are kept where available.

Understanding is tested with five-option multiple-choice questions about an idiom's meaning. Further probes:

• separate memorized from unmemorized idioms with a final-word completion test;
• group idioms by the model's own compositionality estimate;
• compare results with and without example sentences;
• examine chain-of-thought and reasoning-focused models.

Models perform clearly better in high-resource languages like English than in Korean. Their strategy appears hybrid: memory matters, but compositional idioms and context also help. Gains from reasoning differ across languages. In practice, supplying explicit idiom meanings improves generation, translation and LLM-based evaluation.

Key points

• MIDAS size and sources: EN 9,766, DE 10,097, ZH 11,851, KO 11,316, AR 8,051, TR 13,579 idioms. Sources include Wiktionary, Duden–Redewendungen, the Xinhua dictionary and the Korean Standard Dictionary. The dataset is released at github.com/HYU-NLP/MIDAS.
• Question design: each MCQ has one correct meaning and four distractors. Two distractors resemble the idiom's surface form and two resemble its figurative meaning.
• Context and memorization: with Qwen2.5-32B-Instruct, example sentences help most on idioms the model has not memorized. Higher compositionality seems to go with easier memorization.
• GPT-4o as a judge: agreement with human scores rises sharply when the idiom's meaning is provided.
  • Sentence generation: Kendall's τ 0.739 with meaning vs. 0.118 without.
  • Translation: 0.615 vs. 0.403.
• Downstream tasks: for Korean and Turkish, MIDAS meanings improved idiom sentence generation and translation into English. This held for both Qwen2.5-32B-Instruct and Aya-Expanse-32B.