Elamur Research⌕
☰
home / papers / 2026 / Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection

Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection

Summary

The paper asks whether chain-of-thought (CoT) "reasoning" helps LLMs decide if a potentially idiomatic expression (PIE) is used figuratively or literally, and how this depends on model size. The authors compare three groups of open models: base Qwen2.5 (1.5B–32B) and Llama3.3-70B, the Qwen2.5-Math checkpoints that serve as an intermediate step for the 1.5B and 7B distillations, and the DeepSeek-R1 distilled reasoning models. They evaluate with zero-shot prompts on four datasets: SemEval 2022 Task 2a, the idiom subset of FLUTE, MAGPIE and the newer DICE. Scores are Macro F1.

Reasoning helped less, and less consistently, than expected. Small reasoning models recover part of the drop caused by math-tuning but stay below their base versions. The 14B–70B models gain only modestly. Manual analysis of the CoT shows that larger models usually define idioms correctly but still struggle to use context. Small models often state wrong meanings. Adding definitions written by larger models to the prompts of small models helps on FLUTE. This points to a cheap form of knowledge transfer and shows that reasoning alone does not solve figurative-language understanding.

Key points

  • Datasets:
    • SemEval 2022 Task 2a: English, Portuguese and Galician; 150 PIEs; 2342 examples.
    • FLUTE idiom subset: 250 NLI examples covering 69 idioms.
    • MAGPIE: 4840 examples covering 1134 expressions.
    • DICE: 2066 sense-balanced sentences, released after the models were trained, so contamination is ruled out.
  • Setup: models ran via vLLM with quantized weights on a single A100 80GB. Results are mean Macro F1 over 5 seeds. GPT-4o was used to extract labels from outputs that were hard to parse.
  • DeepSeek-R1 Qwen-32B performed best overall (e.g., 0.948 on FLUTE), slightly ahead of DeepSeek-R1 Llama-70B. The base Qwen2.5-7B was the only model to exceed 0.7 on SemEval (0.737).
  • Math-tuning sharply reduced performance at 1.5B and 7B. Subsequent reasoning-tuning partly recovered it, with gains of up to about 0.2 per task for the 7B model.
  • Definitions generated by larger models raised the smallest models' FLUTE score by 0.069 Macro F1 on average, with no significant effect on DICE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/24iojjct3mvu

@inproceedings{phelps2026stands,
  title = {{Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection}},
  author = {Phelps, Dylan and Wilkens, Rodrigo and Gow-Smith, Edward and Pickard, Thomas M. R. and Mi, Maggie and Idiart, Marco and Villavicencio, Aline},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/24iojjct3mvu},
  url = {https://doi.org/10.63317/24iojjct3mvu}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… find our card
  3. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  5. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611… find
  6. Llama Team. 2024. The Llama 3 Herd of Models. ArXiv:2407.21783 [cs]. find
  7. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter… find our card
  8. OpenAI. 2024. Gpt-4o system card. find our card
  9. OpenAI. 2025. Learning to reason with LLMs. find
  10. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  11. Dylan Phelps, Thomas M. R. Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2… find
  12. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  13. Qwen. 2024. Qwen2.5: A Party of Foundation Models! Section: blog. find
  14. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… find
  15. Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel. 2024. Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups. In Proceedings of the 2024 Conference on Empirical Methods in Natural Languag… find
  16. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, pages 74952–74965. Curran Associates, Inc. find our card
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R… find our card
  18. Language Resource References Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative Language Understanding through Textual Explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159. find
  19. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287. European Language Resources Association. find our card
  20. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2025. Rolling the DICE on idiomaticity: How LLMs fail to grasp context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7314–7332, Vienna, Austria. Association for Comp… find
  21. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/phelps-2026-stands-reason-investigating-effect-reasoning/