Elamur Research⌕
☰
home / papers / 2023 / How Are Idioms Processed Inside Transformer Language Models?

How Are Idioms Processed Inside Transformer Language Models?

Summary

This study asks how Transformer encoders internally handle idiomatic expressions, which carry meanings that cannot be derived from their component words. The authors took 838 English idioms from the EPIE dataset and built a new resource around them. For each idiom it contains an idiomatic sentence, a literal paraphrase conveying the same meaning, and two unrelated literal sentences that reuse a key word from the idiom. They then probed BERT, Multilingual BERT and DistilBERT layer by layer with three analyses: sentence-level cosine similarity and similarity ranking between idiom sentences and their paraphrases, word-level comparison of embeddings in idiomatic versus literal contexts, and the average attention a target word receives from the rest of the sentence.

All three models encode idioms differently from literal language. Words inside idioms drift away from their literal sense in deeper layers and receive less attention from the surrounding tokens. This fits the linguistic view that idioms behave like single units. The findings are relevant to building and interpreting models that must handle figurative language.

Key points

  • The data comprise 838 frequent English idioms from EPIE, each with an idiom sentence, a spelt-out paraphrase and two literal sentences; the full dataset is released with the paper.
  • The models were the Hugging Face checkpoints BERT-base-uncased, BERT-base-multilingual-uncased (12 layers each) and distilBERT-base-uncased (6 layers).
  • Sentence level: idiom sentences were closer to their paraphrases than to a random baseline in every layer (p < 0.001), and t-values grew with depth. The ranking metric peaked at layer 10 for BERT, layer 9 for Multilingual BERT and layer 5 for DistilBERT.
  • Word level and attention: a word was less similar across idiom and literal contexts than across two literal contexts, with the effect strongest in BERT. Other tokens also attended less to idiom-internal words (p < 0.01).
  • DistilBERT and Multilingual BERT behaved more like each other than like BERT. The study is limited to English and to BERT-family models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.starsem-1.16

@inproceedings{tian2023how,
  title = {{How Are Idioms Processed Inside Transformer Language Models?}},
  author = {Tian, Ye and James, Isobel and Son, Hye},
  booktitle = {Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.starsem-1.16},
  url = {https://doi.org/10.18653/v1/2023.starsem-1.16}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. Handbook of natural language processing, 2:267–292. 346 347 348 349 350 351 352 353 354 find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert’s attention. In Proceedings of the Second BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 276–286. Association for Computational Linguistics. 357… find
  4. Verna Dankers, Christopher G Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. arXiv preprint arXiv:2205.15301. 363 364 365 366 find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find
  6. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021a. Assessing the representations of idiomaticity in vector models with a noun compound dataset labeled at type and token levels. In Proceedings of the 59th Annual Meeting of the Association for Computati… find
  7. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. A… find our card
  8. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 391 392 393 394 find our card
  9. Alessandro Lenci, Magnus Sahlgren, Patrick Jeuniaux, Amaru Cuba Gyllensten, and Martina Miliani. 2021. A comprehensive comparative evaluation and analysis of distributional semantic models. arXiv preprint arXiv:2105.09825v1. 395 396 397 398 399 — [arXiv:2105.09825](https://arxiv.org/abs/2105.09825) find
  10. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. Astitchinlanguagemodels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. CoRR, abs/2109.04413. 400 401 402 403 404 find our card
  11. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, bert? assessing bert as a distributional semantics model. Proceedings of the Society for Computation in Linguistics, 3(34). 405 406 407 408 409 178 5 410 411 412 413 414 find
  12. Vasudevan Nedumpozhimana and John Kelleher. 2021. Finding BERT’s idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57–62, Online. Association for Computational Linguistics. 415 416 find
  13. Geoffrey Nunberg, Ivan A Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491–538. 417 418 419 find
  14. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502. 420 421 422 423 find our card
  15. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2021. On the effect of dropping layers of pre-trained transformer models. Journal of Computer Speech and Language. 424 425 426 427 find
  16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. 428 429 find
  17. Prateek Saxena and Soma Paul. 2020. Epie dataset: A corpus for possible idiomatic expressions. 430 431 find
  18. John Sinclair and Les Sinclair. 1991. Corpus, concordance, collocation. Oxford University Press, USA. 432 433 434 435 find
  19. Michaela Socolof, Jackie Chi Kit Cheung, Michael Wagner, and Timothy J O’Donnell. 2021. Characterizing idioms: Conventionality and contingency. arXiv preprint arXiv:2104.08664. 436 437 438 find
  20. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950. 439 440 441 find our card
  21. Matej Ulčar and Marko Robnik-Sikonja. 2021. Training dataset and dictionary sizes matter in bert models: the case of baltic languages. 442 443 444 445 find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. 446 447 448 449 450 451 452 453 find
  23. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langu… find our card
  24. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual bert? pages 120–130. 179 6 find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/tian-2023-how-are-idioms-processed-inside/