Elamur Research⌕
☰
home / papers / 2024 / Semantics of Multiword Expressions in Transformer-Based Models: A Survey

Semantics of Multiword Expressions in Transformer-Based Models: A Survey

Summary

Multiword expressions (MWEs) such as noun compounds, particle verbs and idioms spread their meaning across several words. They range from transparent to opaque, and their reading can shift with context. This TACL 2024 survey asks how well transformer language models handle such meanings. The authors review a fast-growing but fragmented literature, mostly intrinsic evaluations of pretrained and optimized models. These are largely encoder-only (BERT, mBERT, RoBERTa, XLM-R, SBERT), with some BART, T5, XLNet and GPT. Tasks include phrase similarity, compound compositionality and paraphrase, and idiomaticity classification of potentially idiomatic expressions (PIEs). Findings are organized around three questions: overall ability and generalization, where MWE information is located in the model, and which linguistic properties matter. The verdict is that models capture MWE semantics only partially and unevenly. They rely on surface overlap and memorization, generalize poorly to unseen expressions, and encode MWE meaning mostly in lower layers. Representations are better for expressions that are less idiosyncratic, less ambiguous and more concrete. The survey matters because it challenges assumptions about transformer semantics and calls for comparable evaluation setups.

Key points

  • On phrase similarity, model–human correlation falls from about 0.6 on the full dataset to about 0.2 on pairs with swapped constituents (e.g., law school vs. school law, 12% of pairs). This points to reliance on word overlap.
  • For noun compound compositionality, Nandakumar et al. (2019) obtain Pearson r between 0.15 and 0.60, with BERT roughly 0.2 below static embeddings. Garcia et al. (2021a) report ρ = 0.37 for English and 0.26 for Portuguese, versus 0.73 and 0.60 for the static-embedding SOTA (Cordeiro et al., 2019).
  • Fine-tuning and knowledge enhancement can improve MWE representations, but the gains vary widely across models, tasks and expression types.
  • There is broad agreement that lower layers work best for compound compositionality, PIE idiomaticity and paraphrase similarity. Modeling the target expression within a wider context also gives the most adequate representations.
  • Cross-lingual evidence is thin, since most comparisons rest on single studies (e.g., English vs. Russian PIE transfer). Future work should cover more MWE types and non-English languages and use harder tasks that test generalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00657

@article{miletic2024semantics,
  title = {{Semantics of Multiword Expressions in Transformer-Based Models: A Survey}},
  author = {Miletić, Filip and Walde, Sabine Schulte im},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00657},
  url = {https://doi.org/10.1162/tacl_a_00657}
}

References

79 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Katsiaryna Aharodnik, Anna Feldman, and Jing Peng. 2018. Designing a Russian idiom-annotated corpus. In Proceedings of the Eleventh International Conference on Mojca Brglez. 2023. Dispersing the clouds of doubt: Can cosine similarity of word embeddings help identify relation-level metaphors in the g… find
  2. Slovene? In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), pages 61–69, Dubrovnik, Croatia. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.bsnlp-1.8 ↗
  3. Amir Cohen, Hila Gonen, Ori Shapira, Ran Levy, and Yoav Goldberg. 2022. McPhraSy: Multi-context phrase similarity and clustering. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3538–3550, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. htt… find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… find
  5. Albert Coil and Vered Shwartz. 2023. From chocolate bunny to chocolate crocodile: Do language models understand noun compounds? In Findings of the Association for Computational Linguistics: ACL 2023, pages 2698–2710, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10 .186… find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  7. Lars Buijtelaar and Sandro Pezzelle. 2023. A psycholinguistic analysis of BERT’s representations of compounds. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2230–2241, Dubrovnik, Croatia. Association for Computational Linguistic… find
  8. Laura Burdick, Jonathan K. Kummerfeld, and Rada Mihalcea. 2022. Using paraphrases to study properties of contextual embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4558–4568, Seattle,… find
  9. Mathieu Constant, Gülşen Eryiǧit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837–892. https://doi.org/10.1162/COLI a 00302 — [doi:10.1162/coli](https://doi.org/10.1162/coli) ↗
  10. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008), pages 19–22, Marrakech, Morocco. find
  11. Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022. It’s not rocket science: Interpreting figurative language in narratives. Transactions of the Association for Computational Linguistics, 10:589–606. https://doi.org/10.1162/tacl a 00478 ↗ our card
  12. Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1–57. https://doi.org/10.1162/coli a 00341 ↗
  13. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? Analysing idiom processing in neural machine translation. In Proceedings of… find our card
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  15. Neele Falk, Yana Strakatova, Eva Huber, and Erhard Hinrichs. 2021. Automatic classification of attributes in German adjective-noun phrases. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 239–249, Groningen, The Netherlands (online). Association for Compu… find
  16. Pablo Gamallo, Manuel Corral, and Marcos Garcia. 2021. Comparing dependency-based compositional models with contextualized word embeddings. In Proceedings of the 13th International Conference on Agents and Artificial Intelligence, pages 1258–1265. SCITEPRESS - Science and Technology Publications. ht… find
  17. Luis Espinosa Anke, Joan Codina-Filba, and Leo Wanner. 2021. Evaluating language models for the retrieval and categorization of lexical collocations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1406–1417, Online.… find
  18. Pablo Gamallo, Marcos Garcia, and Iria de-Dios-Flores. 2022. Evaluating contextualized vectors from both large language models and compositional strategies. Procesamiento del Lenguaje Natural, 153–164. https://doi.org/10.26342/2022-69-13 — [doi:10.26342/2022-69-13](https://doi.org/10.26342/2022-69-13) ↗
  19. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021a. Assessing the representations of idiomaticity in vector models with a noun compound dataset labeled at type and token levels. In Proceedings of the 59th Annual Meeting of the Association for Computati… find
  20. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  21. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. https://doi.org/10.1162/tacl a 00298 ↗ our card
  22. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. A… find our card
  23. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  24. Christopher Jenkins, Filip Miletić, and Sabine Schulte im Walde. 2023. To split or not to split: Composing compounds in contextual vector spaces. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16131–16136, Singapore. Association for Computational Li… find
  25. Matthias Hartung. 2015. Distributional Semantic Models of Attribute Meaning in Adjectives and Nouns. Ph.D. thesis, Heidelberg University. find
  26. Chikara Hashimoto and Daisuke Kawahara. 2008. Construction of an idiom corpus and its application to idiom identification based on WSD incorporating idiom-specific features. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 992–1001, Honolulu, Hawaii. A… find
  27. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10 .1162/tacl a 00300 find our card
  28. Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 248–264, Dubrovnik, Croatia. Assoc… find our card
  29. Barbara J. Juhasz, Yun-Hsuan Lai, and Michelle L. Woodcock. 2015. A database of 629 English compound words: Ratings of familiarity, lexeme meaning dominance, semantic transparency, age of acquisition, imageability, and sensory experience. Behavior Research Methods, 47(4):1004–1019. https://doi.org/… — [doi:10.3758/s13428-014-0523-6](https://doi.org/10.3758/s13428-014-0523-6) find
  30. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the Ninth International Conference on Learning Representations (ICLR 2021). find our card
  31. Filip Klubička, Vasudevan Nedumpozhimana, and John Kelleher. 2023. Idioms, probing and dangerous things: Towards structural probing for idiomaticity in vector space. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 45–57, Dubrovnik, Croatia. Association for Computation… find
  32. Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013. SemEval-2013 task 4: Free paraphrases of noun compounds. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Worksh… find
  33. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020). find
  34. Beth Levin, Lelia Glass, and Dan Jurafsky. 2019. Systematicity in the semantics of noun Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about compounds: The role of artifacts vs. natural kinds. Linguistics, 57(3):429–471. https://doi.org/10.1515/ling-2019-0013 do you me… — [doi:10.1515/ling-2019-0013](https://doi.org/10.1515/ling-2019-0013) ↗
  35. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  36. Filip Miletić and Sabine Schulte im Walde. 2023. A systematic search for compound semantics in pretrained BERT architectures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1499–1512, Dubrovnik, Croatia. Association for Computat… find
  37. Siyan Li, Riley Carlson, and Christopher Potts. 2022. Systematicity in GPT-3’s interpretation of novel English noun compounds. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 717–728, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https://… find
  38. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL-08: HLT, pages 236–244, Columbus, Ohio. Association for Computational Linguistics. find
  39. Diego Moussallem, Mohamed Ahmed Sherif, Diego Esteves, Marcos Zampieri, and Axel-Cyrille Ngonga Ngomo. 2018. LIdioms: A multilingual linked idioms data set. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re… find
  40. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452,… find
  41. Navnita Nandakumar, Timothy Baldwin, and Bahar Salehi. 2019. How well do embedding models capture non-compositionality? A view from multiword expressions. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 27–34, Minneapolis, USA. Association for Computation… find
  42. Emmy Liu and Graham Neubig. 2022. Are representations built from the ground up? An empirical examination of local composition in language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9053–9073, Abu Dhabi, United Arab Emirates. Association f… find
  43. Vasudevan Nedumpozhimana and John Kelleher. 2021. Finding BERT’s idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57–62, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.mwe-1.7 ↗
  44. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv: 1907.11692. https://doi.org/10.48550/arXiv.1907.11692 ↗
  45. Vasudevan Nedumpozhimana, Filip Klubička, and John D. Kelleher. 2022. Shapley idioms: Analysing BERT sentence embeddings for general idiom token identification. Frontiers in Artificial Intelligence, 5:813967. https://doi.org/10.3389/frai.2022.813967, PubMed: 35360661 — [doi:10.3389/frai.2022.813967](https://doi.org/10.3389/frai.2022.813967) ↗
  46. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What Diarmuid Ó. Séaghdha and Ann Copestake. 2007. Co-occurrence contexts for noun compound interpretation. In Proceedings of the Workshop on A Broader Perspective on Multiword Expressions, pages 57–64, Prague, Czech Rep… find
  47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485–5551. find
  48. Giulia Rambelli, Emmanuele Chersoni, Marco S. G. Senaldi, Philippe Blache, and Alessandro Lenci. 2023. Are frequent phrases directly retrieved like idioms? An investigation with self-paced reading and language models. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 87–… find our card
  49. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational… find our card
  50. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 210–218, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find our card
  51. Thang Pham, Seunghyun Yoon, Trung Bui, and Anh Nguyen. 2023. PiC: A phrase-in-context dataset for phrase understanding and semantic search. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1–26, Dubrovnik, Croatia. Association for… find
  52. Yuval Pinter, Cassandra L. Jacobs, and Max Bittker. 2020a. NYTWIT: A dataset of novel words in the New York Times. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6509–6515, Barcelona, Spain (Online). International Committee on Computational Linguistics. https… find
  53. Yuval Pinter, Cassandra L. Jacobs, and Jacob Eisenstein. 2020b. Will it unblend? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1525–1535, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .findings-emnlp.138 ↗
  54. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  55. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  56. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl_a_00349 ↗ our card
  57. Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In Computational Linguistics and Intelligent Text Processing, pages 1–15, Berlin. Springer. https://doi.org/10.1007/3 -540-45715-1_1 find our card
  58. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing – NeurIPS 2019. find
  59. Dominican Republic. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.emnlp-main.230 find our card
  60. Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017. The PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the 13… find our card
  61. Tadej Škvorc, Polona Gantar, and Marko Robnik-Šikonja. 2022. MICE: Mining idioms with contextual embeddings. Knowledge-Based Systems, 235:107606. https://doi.org/10 .1016/j.knosys.2021.107606 find
  62. Yana Strakatova, Neele Falk, Isabel Fuhrmann, Erhard Hinrichs, and Daniela Rossmann. 2020. All that glitters is not gold: A gold standard of adjective-noun collocations for German. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4368–4378, Marseille, France. Europea… find
  63. Prateek Saxena and Soma Paul. 2020. EPIE dataset: A corpus for possible idiomatic expressions. In Petr Sojka, Ivan Kopeček, Karel Pala, and Aleš Horák, editors, Text, Speech, and Dialogue, volume 12284, pages 87–94, Cham. Springer. https://doi.org/10 .1007/978-3-030-58323-1_9 find
  64. Ryosuke Takahashi, Ryohei Sasano, and Koichi Takeda. 2022. Leveraging three types of embeddings from masked language models in idiom token classification. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 234–239, Seattle, Washington. Association for Computati… find
  65. Sabine Schulte im Walde. forthc. Collecting and investigating features of compositionality ratings. In Voula Giouli and Verginica Barbu Mititelu, editors, Multiword Expressions in Lexical Resources. Linguistic, Lexicographic and Computational Perspectives, Phraseology and Multiword Expressions, Lang… find
  66. Minghuan Tan and Jing Jiang. 2020. A BERT-based dual embedding model for Chinese idiom prediction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1312–1322, Barcelona, Spain (Online). International Committee on Computational Linguistics. https://doi.org/10 .1… find
  67. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the Fifth International Conference on Learning Representations (ICLR 2017). find our card
  68. Minghuan Tan and Jing Jiang. 2021. Does BERT understand idioms? A probing-based empirical study of BERT encodings of idioms. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1397–1407, Held Online. INCOMA Ltd. https://doi.org /10.26… ↗
  69. Vered Shwartz and Ido Dagan. 2019. Still a pain in the neck: Evaluating text representations on lexical composition. Transactions of the Association for Computational Linguistics, 7:403–419. https://doi.org/10.1162/tacl_a_00277 ↗ our card
  70. Minghuan Tan, Jing Jiang, and Bing Tian Dai. 2021. A BERT-based two-stage model for Chinese chengyu recommendation. ACM Transactions on Asian and Low-Resource Language Information Processing, 20(6):1–18. https://doi.org/10.1145/3453185 — [doi:10.1145/3453185](https://doi.org/10.1145/3453185) ↗
  71. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 288… find our card
  72. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates. find our card
  73. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computation… find our card
  74. Shufan Wang, Laure Thompson, and Mohit Iyyer. 2021. Phrase-BERT: Improved phrase embeddings from BERT with an application to corpus exploration. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10837–10851, Online and Punta Cana, Dominican Republic. As… find
  75. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disambiguation with contextualized embeddings. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 161–170, Erlangen, Germany.… find
  76. Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896–4907, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /20… find
  77. Lang Yu and Allyson Ettinger. 2021. On the interplay between fine-tuning and composition in Transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2279–2293, Online. Association for Computational Linguistics. https://doi.org /10.18653/v1/2021.findings-acl.… ↗
  78. Ziheng Zeng and Suma Bhat. 2021. Idiomatic expression identification using semantic compatibility. Transactions of the Association for Computational Linguistics, 9:1546–1562. https://doi.org/10.1162/tacl a 00442 ↗ our card
  79. Ziheng Zeng and Suma Bhat. 2022. Getting BART to ride the idiomatic train: Learning to represent idiomatic expressions. Transactions of the Association for Computational Linguistics, 10:1120–1137. https://doi.org/10.1162/tacl a 00510 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/miletic-2024-semantics-multiword-expressions-transformer-based/