A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models

Summary

The paper asks whether small conversational LLMs can tell literal uses of Potentially Idiomatic Expressions (PIEs) from figurative ones in English sentences. Existing benchmarks such as MAGPIE may have leaked into pre-training data, so the authors built a new test suite from scratch, IdioTS. They compiled 93 colloquial idioms covering a range of syntactic patterns. Native-speaker language professionals then wrote new idiomatic sentences for them. The researchers themselves added "distractor" sentences, in which idiom words appear with a literal sense.

Idiom detection is framed as binary sentence classification. Three instruction-tuned, open-source models of roughly 7B parameters were prompted to decide whether a sentence contains an idiom and to explain it. The outputs were evaluated both automatically and manually, followed by an error analysis.

All models struggled with ambiguous sentences. They showed a strong acquiescence bias: they "found" idioms where none existed and gave fluent but wrong justifications. This matters because it shows that apparent language understanding in chatbots can mask unreliable handling of figurative meaning.

Key points

IdioTS composition: 250 English sentences, of which 164 are idiomatic (positive class) and 86 are distractors (negative class). It is released under CC BY-SA-NC 4.0.
Sentence authoring: idiomatic sentences came from eight volunteer native English speakers at C1+ level, mostly British, including teachers, linguists, translators and NLP experts. The text was also gender-neutralised where possible.
Distractor strategies:
- placing the full idiom in a context that forces a literal reading;
- reusing only some of its words;
- shifting a word's syntactic or morphological role;
- using sound-alike phrases, e.g. "speed and span" vs. "spick and span".
Models: Llama2, Mistral and Vicuna, all ~7B-parameter dialogue models, run on a local machine without a GPU.
Main error type: distractors where the model wrongly gave the associated PIE an idiomatic reading. Counts were 55 for Llama2 (0.640), 53 for Mistral (0.616) and 47 for Vicuna (0.546). Vicuna made the fewest such errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.figlang-1.5

@inproceedings{fornaciari2024hard,
  title = {{A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models}},
  author = {Fornaciari, Francesca De Luca and Altuna, Begoña and Gonzalez-Dios, Itziar and Melero, Maite},
  booktitle = {Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.figlang-1.5},
  url = {https://doi.org/10.18653/v1/2024.figlang-1.5}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative Language Understanding through Textual Explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159. find
Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI Models’ Performance on Figurative Language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computatio… find our card
Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland… find our card
Minghuan Tan and Jing Jiang. 2021. Does BERT understand idioms? a probing-based empirical study of BERT encodings of idioms. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1397–1407, Held Online. INCOMA Ltd. find
Alexey Drutsa, Dmitry Ustalov, Valentina Fedorova, Olga Megorskaya, and Daria Baidakova. 2021. Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
Simone Tedeschi, Federico Martelli, and Roberto Navigli. 2022. ID10M: Idiom Identification in 10 Languages. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715–2726, Seattle, United States. Association for Computational Linguistics. find our card
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… find
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fornaciari-2024-hard-nut-crack-idiom-detection/

📄 A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models