Elamur Research⌕
☰
home / papers / 2019 / Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Summary

Phrase meaning often cannot be built by simply combining word meanings. Composition can change what a word means (meaning shift, as in guilt trip) or add unstated content that needs world knowledge (implicit meaning, as in olive oil vs. baby oil). Shwartz and Dagan (TACL, 2019) test how well current text representations handle both effects.

They built an evaluation suite of six classification tasks covering MWEs and free word combinations. The tasks reuse existing annotated datasets, recast in a common format. Where the original data had no context, they added Wikipedia sentences, and they used lexically disjoint splits to limit memorization. Generic classifiers were trained on top of six pre-trained static and contextualized representations. Human performance was estimated on annotated test samples.

Contextualized embeddings outperform static ones overall. On meaning-shift tasks they roughly match human performance. On recovering implicit meaning they remain clearly below humans. The authors argue that a language-modeling objective alone may not be enough to capture implicit meaning. The released suite gives future representation research a reusable benchmark for lexical composition.

Key points

  • Six tasks, with train/val/test sizes:
    • VPC classification (Tu and Roth 2012), 919/209/220.
    • LVC classification (Tu and Roth 2011), 1521/258/383.
    • NC Literality (Reddy et al. 2011, augmented with Tratz 2011), 2529/323/138.
    • NC Relations (SemEval 2013 Task 4), 1274/162/130.
    • AN Attributes (HeiPLAS), 837/108/106.
    • Phrase Type tagging (STREUSLE), 3017/372/376.
  • Source data: the VPC data has 1,348 BNC sentences covering 23 phrasal verbs built from six frequent verbs. The SemEval data contains 356 noun compounds with 12,446 human-written paraphrases.
  • Added contexts: for out-of-context datasets, sentences of 15–20 words were taken from a January 2018 English Wikipedia dump. Data was split about 80/10/10 under lexical constraints, for example by verb, head noun, or adjective.
  • Representations tested: static word2vec, GloVe and fastText, and contextualized ELMo, OpenAI GPT and BERT.
  • Availability: data and code are released at github.com/vered1986/lexcomp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

56 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00277

@article{shwartz2019still,
  title = {{Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition}},
  author = {Shwartz, Vered and Dagan, Ido},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00277},
  url = {https://doi.org/10.1162/tacl_a_00277}
}

References

60 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations (ICLR). find our card
  2. Wei-Te Chen, Claire Bonial, and Martha Palmer. 2015. English light verb construction identification using lexical knowledge. In Twenty-Ninth AAAI Conference on Artificial Intelligence. find
  3. Asaf Amrami and Yoav Goldberg. 2018. Word sense induction with neural biLM and symmetric patterns. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4860–4867, Brussels. find
  4. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find our card
  5. Fatemeh Mohamadi Asl. 2013. The impact of context on learning idioms in EFL classes. TESOL Journal, 37(1):2. find
  6. Thomas C Cooper. 1999. Processing of idioms by L2 learners of English. TESOL Quarterly, 33(2):233–262. find
  7. Geoff Bacon and Terry Regier. 2018. Probing sentence embeddings for structure-dependent tense. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 334–336. find
  8. Silvio Cordeiro, Carlos Ramisch, Marco Idiart, and Aline Villavicencio. 2016. Predicting the compositionality of nominal compounds: Giving word embeddings a hard time. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1986–1997,… find
  9. Timothy Baldwin. 2005. Deep lexical acquisition of verb–particle constructions. Computer Speech & Language, 19(4):398–414. find
  10. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1183–1193, Cambridge, MA. find
  11. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Zanzotto. 2013. Recognizing Textual Entailment, Morgan & Claypool Publishers. find
  12. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and improving morphological learning in the neural machine translation Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models lea… find
  13. Matthias Hartung, Fabian Kaupmann, Soufian Jebbara, and Philipp Cimiano. 2017. Learning compositionality functions on word embeddings for modelling attribute meaning in Adjective-noun phrases. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguist… find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  15. Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013. Semeval-2013 task 4: Free paraphrases of noun compounds. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Worksh… find
  16. Corina Dima. 2016. On the compositionality and semantic interpretation of English noun compounds. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 27–39. find
  17. Otto Jespersen. 1965. A Modern English Grammar: On Historical Principles, George Allen & Unwin Limited. find
  18. Georgiana Dinu, Nghia The Pham, and Marco Baroni. 2013. General estimation and evaluation of compositional distributional semantic models. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 50–58, Sofia. find
  19. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find our card
  20. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne. find our card
  21. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov): 2579–2605. find
  22. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neura… find
  23. Jean Maillard and Stephen Clark. 2015. Learning adjective meanings with a tensor-based skip-gram model. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 327–331, Beijing. find
  24. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations (ICLR). find
  25. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6):602–610. find
  26. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  27. Z Harris. 1954. Distributional Hypothesis. Word, 10(23):146–162. find
  28. Preslav Nakov. 2013. On the interpretation of noun compounds: Syntax, semantics, and entailment. Natural Language Engineering, 19(3): 291–330. find
  29. Matthias Hartung. 2015. Distributional Semantic Models of Attribute Meaning in Adjectives and Nouns, Ph.D. thesis, Heidelberg University. find
  30. Vivi Nastase, Preslav Nakov, Diarmuid Ó Séaghdha, and Stan Szpakowicz. 2013. Semantic relations between nominals. Synthesis Lectures on Human Language Technologies, 6(1): 1–119. embeddings. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisti… find
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws.com/openai-assets/ research-covers/language-unsupervised/language understanding paper. pdf. ↗
  32. Geoffrey Nunberg, Ivan A. Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491–538. find
  33. Diarmuid Ó Séaghdha and Ann Copestake. 2009. Using lexical and relational similarity to classify semantic relations. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 621–629, Athens. find
  34. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 210–218, Chiang Mai. find our card
  35. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Autodiff Workshop, NIPS 2017. find
  36. Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 1–15. find our card
  37. Ellie Pavlick and Chris Callison-Burch. 2016. Most ‘‘babies’’ are ‘‘little’’ and most ‘‘problems’’ are ‘‘huge’’: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173, Berlin. find
  38. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 977–983,… find
  39. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  40. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  41. Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1537–1547, Denver, CO. find our card
  42. Marco Silvio Giuseppe Senaldi, Yuri Bizzoni, and Alessandro Lenci. 2019. What do neural networks actually learn, when they learn to identify idioms? Proceedings of the Society for Computation in Linguistics, 2(1):310–313. find
  43. Karl Pichotta and John DeNero. 2013. Identifying phrasal verbs using many bilingual corpora. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 636–646, Seattle, WA. find
  44. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Adam Poliak, Pushpendre Rastogi, M. Patrick Martin, and Benjamin Van Durme… find our card
  45. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534. find our card
  46. Stephen Tratz. 2011. Semantically-enriched Parsing for Natural Language Understanding, University of Southern California. find
  47. Yuancheng Tu and Dan Roth. 2011. Learning English light verb constructions: Contextual or Statistical. In Proceedings of the Workshop on Multiword Expressions: from Parsing and Generation to the Real World, pages 31–39, Portland, OR. find
  48. Vered Shwartz and Ido Dagan. 2018. Paraphrase to Explicate: Revealing Implicit Noun-Compound Relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1200–1211, Melbourne. find
  49. Yuancheng Tu and Dan Roth. 2012. Sorting out the most confusing English phrasal verbs. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on… find
  50. Vered Shwartz and Chris Waterson. 2018. Olive oil is made of olives, baby oil is made for babies: Interpreting noun compounds using paraphrases in a neural model. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  52. Richard Socher, Brody Huval, D. Christopher Manning, and Y. Andrew Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. find
  53. Veronika Vincze, István Nagy T., and Gábor Berend. 2011. Multiword expressions and named entities in the Wiki50 corpus. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 289–295. find
  54. Gabriel Stanovsky and Mark Hopkins. 2018. Spot the odd man out: Exploring the associative power of lexical resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1533–1542, Brussels. find
  55. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In International Conference on Learning Representations (ICLR). find our card
  56. Nitesh Surtani and Soma Paul. 2015. A VSM-based statistical model for the semantic relation interpretation of noun-modifier pairs. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 636–645. find
  57. John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274–285, Copenhagen. find
  58. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word Zhibiao Wu and Martha Palmer. 1994. Verbs, semantic… find
  59. Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar. 2010. Estimating linear models for compositional distributional semantics. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1263–1271. find
  60. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial data set for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shwartz-2019-still-pain-neck-evaluating-text/