Elamur Research⌕
☰
home / papers / 2022 / Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Summary

Neural machine translation systems often render idioms word by word, missing their figurative meaning. This paper asks whether that failure is visible inside Transformer's internal mechanics. The authors used pre-trained Marian-MT models (trained on OPUS) that translate from English into seven European languages. They translated sentences from the MAGPIE corpus of potentially idiomatic expressions (PIEs) and automatically labelled each output as a paraphrase or a literal rendering. Human evaluation confirmed that this heuristic labelling works as intended. They then compared attention patterns and hidden states for figurative and literal PIEs. When the model correctly paraphrases an idiom, the encoder treats the idiom's words more like a single unit: attention inside the PIE rises and exchange with the surrounding context falls. In cross-attention, the decoder relies less on the source tokens. Amnesic probing with INLP removed figurativeness information from the hidden states. This pushed many translations back to literal ones and shifted attention toward literal-like patterns. The authors conclude that Transformer is overly compositional with idioms. They advise checking how compositionality-promoting techniques affect non-compositional phenomena.

Key points

  • Models: Transformer models (6 encoder and 6 decoder layers, 8 heads each) pre-trained by Tiedemann and Thottingal (2020) with Marian-MT on OPUS. Target languages are Dutch, German, Swedish, Danish, French, Italian and Spanish, with Dutch analysed in most detail.
  • Data: MAGPIE (Haagsma et al., 2020) contains 1756 idioms with 57k occurrences. The study used 37k fully figurative or fully literal samples covering 1482 idioms. Labels were derived from whether the idioms' keywords (mostly nouns) were translated literally, using beam search with beam size 5.
  • Literal bias: Most figurative PIEs were still translated word for word. Only about 18–24% of them were paraphrased, depending on the language.
  • Attention patterns: Paraphrased figurative PIEs show stronger attention within the PIE and weaker interaction with their context. They also show less cross-attention to the source and more cross-attention to </s>. These patterns hold across all seven languages.
  • INLP intervention: Intervening on encoder layers 0–4 turned paraphrases into literal translations for an average of 27% (Swedish) to 40% (Spanish) of translations per idiom. This shows that idiomatic translations are brittle and only partly linearly encoded. Code and data are available in the mt_idioms GitHub repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.252

@inproceedings{dankers2022can,
  title = {{Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation}},
  author = {Dankers, Verna and Lucas, Christopher and Titov, Ivan},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.252},
  url = {https://doi.org/10.18653/v1/2022.acl-long.252}
}

References

53 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, and Hans Uszkoreit. 2019. Linguistic evaluation of German-English machine translation using a test suite. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 445–454. find our card
  2. Mona Baker et al. 1992. In Other Words. Routledge. find
  3. Anabela Barreiro, Johanna Monti, Brigitte Orliac, Fernando Batista, et al. 2013. When multiwords go bad in machine translation. In Proceedings of the Workshop on Multi-word Units in Machine Translation and Translation Technology, pages 26–33. find
  4. Samuel A Bobrow and Susan M Bell. 1973. On catching on to idiomatic expressions. Memory & cognition, 1(3):343–346. find
  5. Stéphanie Caillies and Kirsten Butcher. 2007. Processing of idiomatic expressions: Evidence for a new hybrid view. Metaphor and Symbol, 22(1):79–108. find
  6. Kate Cain, Andrea S Towse, and Rachael S Knight. 2009. The development of idiom comprehension: An investigation of semantic and contextual processing skills. Journal of experimental child psychology, 102(3):280–298. find
  7. Rahma Chaabouni, Roberto Dessì, and Eugene Kharitonov. 2021. Can transformers jump around right in natural language? Assessing performance transfer from scan. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 136–148. find
  8. Rosner, and Amalia Todirascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837–892. find
  9. Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022. The paradox of the compositionality of natural language: a neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175. find our card
  12. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  13. Marcos García, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021a. Assessing the representations of idiomaticity in vector models with a noun compound dataset labeled at type and token levels. In Proceedings of the 59th Annual Meeting of the Association for Computati… find
  14. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. find
  15. Marcos García, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564. find our card
  16. Jean-Pierre Colson. 2019. Multi-word units in machine translation: why the tip of the iceberg remains problematic–and a tentative corpus-driven solution. Computational and Corpus-based Phraseology, page 145. find
  17. Raymond W Gibbs Jr, Raymond W Gibbs, and Jr Gibbs. 1994. The poetics of mind: Figurative thought, language, and understanding. Cambridge University Press. find
  18. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find our card
  19. Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Lonneke Van Der Plas, Carlos Ramisch, Michael H. Paul Grice. 1975. Logic and conversation. Syntax and Semantics, 3:41–58. find
  20. H. Paul Grice. 1989. Studies in the Way of Words. Harvard University Press. find
  21. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 279–287. find our card
  22. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika, 28(3/4):321–377. find
  23. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496. find our card
  24. Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556. find our card
  25. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, et al. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121. find our card
  26. Koenraad Kuiper, Marie-Elaine Van Egmond, Gerard Kempen, and Simone Sprenger. 2007. Slipping on superlemmas: Multi-word lexical items in speech production. The Mental Lexicon, 2(3):313–357. find
  27. Murathan Kurfalı and Robert Östling. 2020. Disambiguation of potentially idiomatic expressions with contextual embeddings. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 85–94. find
  28. Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021. On compositional generalization of neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  29. Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational L… find our card
  30. Ari S Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5732–5741. find
  31. Vasudevan Nedumpozhimana and John Kelleher. 2021. Finding BERT’s idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57–62. find
  32. Geoffrey Nunberg, Ivan A Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491–538. find
  33. Robert Östling, Jörg Tiedemann, et al. 2016. Efficient word alignment with markov chain monte carlo. The Prague Bulletin of Mathematical Linguistics. find
  34. Paolo Pedinotti and Alessandro Lenci. 2020. Don’t invite BERT to drink a bottle: Modeling the interpretation of metonymies using BERT and distributional representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6831–6837. find
  35. Elisabeth Piirainen. 2013. Widespread idioms in europe and beyond. toward a lexicon of common figurative units. Neuphilologische Mitteilungen, 13(4):489–. find
  36. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. find
  37. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6078–6087. find
  38. Vikas Raunak, Vaibhav Kumar, Florian Metze, and Jaimie Callan. 2019. On compositionality in neural machine translation. In NeurIPS 2019 Context and Compositionality in Biological and Artificial Neural Systems Workshop. find
  39. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256. find our card
  40. Yutong Shao, Rico Sennrich, Bonnie Webber, and Federico Fancellu. 2018. Evaluating machine translation performance on chinese idioms with a blacklist method. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  41. Vered Shwartz and Ido Dagan. 2019. Still a pain in the neck: Evaluating text representations on lexical composition. Transactions of the Association for Computational Linguistics, 7:403–419. find our card
  42. David A Swinney and Anne Cutler. 1979. The access and processing of idiomatic expressions. Journal of verbal learning and verbal behavior, 18(5):523– 534. find
  43. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 26– 35. find
  44. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019a. Encoders help you disambiguate word senses in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJ… find
  45. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019b. Understanding neural machine translation by simplification: The case of encoder-free models. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1186–1193. sub-layer functionalities… find
  46. Andrea Zaninello and Alexandra Birch. 2020. Multiword expression aware neural machine translation. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3816–3825. find
  47. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT–building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480. find
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000–6010. find
  49. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join… find
  50. Mila Vulchanova, Evelyn Milburn, Valentin Vulchanov, and Giosuè Baggio. 2019. Boon or burden? The role of compositional meaning in figurative language processing and acquisition. Journal of Logic, Language and Information, 28(2):359–387. find
  51. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20. find our card
  52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  53. Yilin Yang, Longyue Wang, Shuming Shi, Prasad Tadepalli, Stefan Lee, and Zhaopeng Tu. 2020. On the find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/dankers-2022-can-transformer-be-too-compositional/