Elamur Research⌕
☰
home / papers / 2012 / Semantic Compositionality through Recursive Matrix-Vector Spaces

Semantic Compositionality through Recursive Matrix-Vector Spaces

Summary

Word vector models capture lexical meaning well, but they cannot represent how words combine into the meaning of longer phrases. This is a problem when a word acts as an operator, as in "extremely strong" or "not good." The authors propose the matrix-vector recursive neural network (MV-RNN). Every word, and every phrase node in a binarized parse tree, gets two things: a vector that encodes its own meaning, and a matrix that encodes how it changes the meaning of whatever it combines with. Composition runs bottom-up along the syntactic tree. Each child's matrix transforms the sibling's vector, and a shared nonlinear layer merges the results. Phrase matrices are built in a parallel way.

The model is evaluated on three tasks:

  • fine-grained sentiment distributions for adverb-adjective pairs,
  • sentence-level sentiment of movie reviews,
  • noun-pair relation classification on SemEval-2010 Task 8.

The authors report state-of-the-art results on all three. MV-RNN is the only compared model that correctly handles negation with "not." The approach matters because it unifies several earlier composition schemes and learns operator-like semantics without hand-built lexical resources.

Key points

  • Word vectors are initialized from 50-dimensional Collobert and Weston (2008) embeddings trained on Wikipedia. Each word matrix starts as the identity plus small Gaussian noise.
  • The parent vector is computed as g(W[Ba; Ab]), and the parent matrix as W_M[A; B]. Both W and W_M are global n×2n matrices, so every phrase keeps the same dimensions as a single word.
  • Several earlier models are special cases of this composition function:
    • additive Ba + Ab (Mitchell and Lapata; Zanzotto et al.),
    • adjective-matrix Ab (Baroni and Zamparelli),
    • the plain RNN of Socher et al. (2011), where the word matrices are fixed to identity.
  • On SemEval-2010 Task 8, without extra features, MV-RNN reaches 79.1 F1, versus 74.8 for the RNN and 73.0 for linear MVR. This beats all prior systems except Rink and Harabagiu (82.2).
  • Adding POS tags (+0.9), WordNet hypernyms (+1.3) and NER tags (+0.6) lifts MV-RNN to 82.4 F1, a new state of the art. The previous best system used far more resources, including FrameNet, PropBank and Google n-grams.

Interesting ideas

  • I1. Two roles per word. Each word carries both a meaning vector and an operator matrix. A purely functional word such as an intensifier can then shrink its vector toward zero and do its work through the matrix. This could help other languages with rich modifier and particle systems.
  • I2. Identity-plus-noise initialization. Starting each matrix near the identity means a word begins as a neutral modifier and only learns operator behaviour where the data supports it. This is a cheap, stable default for any model that adds per-token transformation matrices.
  • I3. Generalize, then ablate. The authors design one composition function that contains earlier models as restricted settings, then compare those settings directly. This makes clear which component drives the gains, and the same evaluation style suits other architecture comparisons.
  • I4. Composing along the syntactic path. For relation classification, the representation is built from the words on the parse path between the two nouns. This gives a learned, variable-length path feature instead of hand-crafted patterns, and it could transfer to dependency-parsed data such as UD treebanks.
  • I5. Appending a few symbolic features. Adding a handful of features (POS, WordNet hypernyms, NER) to the learned vector before the final classifier gave measurable gains. It is a simple way to mix learned and lightweight linguistic signals when resources are limited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

343 papers of our library cite it (of the 140884 whose bibliographies we read); 36 of them have a page here, 307 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{socher2012semantic,
  title = {{Semantic Compositionality through Recursive Matrix-Vector Spaces}},
  author = {Socher, Richard and Huval, Brody and Manning, Christopher D. and Ng, Andrew Y.},
  booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  year = {2012},
  url = {https://aclanthology.org/D12-1110/}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Baroni and A. Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  2. M. Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In EMNLP. find
  3. L. Bottou. 2011. From machine learning to machine reasoning. CoRR, abs/1102.1808. — [arXiv:1102.1808](https://arxiv.org/abs/1102.1808) find
  4. M. Ciaramita and Y. Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In EMNLP. find our card
  5. S. Clark and S. Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the AAAI Spring Symposium on Quantum Interaction, pages 52–55. find
  6. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  7. J. Curran. 2004. From Distributional to Semantic Similarity. Ph.D. thesis, University of Edinburgh. find
  8. J. L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2-3). find
  9. G. Frege. 1892. Über Sinn und Bedeutung. In Zeitschrift für Philosophie und philosophische Kritik, 100. find
  10. D. Garrette, K. Erk, and R. Mooney. 2011. Integrating Logical Representations with Probabilistic Information using Markov Logic. In Proceedings of the International Conference on Computational Semantics. find
  11. C. Goller and A. Küchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of the International Conference on Neural Networks (ICNN-96). find
  12. E. Grefenstette and M. Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In EMNLP. find
  13. T. L. Griffiths, J. B. Tenenbaum, and M. Steyvers. 2007. Topics in semantic representation. Psychological Review, 114. find
  14. I. Hendrickx, S.N. Kim, Z. Kozareva, P. Nakov, D. Ó Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, and S. Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation. find our card
  15. G. E. Hinton. 1990. Mapping part-whole hierarchies into connectionist networks. Artificial Intelligence, 46(1-2). find
  16. R. Jones, B. Rey, O. Madani, and W. Greiner. 2006. Generating query substitutions. In Proceedings of the 15th international conference on World Wide Web. find
  17. D. Klein and C. D. Manning. 2003. Accurate unlexicalized parsing. In ACL. find our card
  18. D. Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of COLING-ACL, pages 768–774. find our card
  19. E. J. Metcalfe. 1990. A compositive holographic associative recall model. Psychological Review, 88:627– 661. find
  20. J. Mitchell and M. Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  21. R. Montague. 1974. English as a formal language. Linguaggi nella Societa e nella Tecnica, pages 189–224. find
  22. T. Nakagawa, K. Inui, and S. Kurohashi. 2010. Dependency tree-based sentiment classification using CRFs with hidden variables. In NAACL, HLT. find
  23. M. Paşca, D. Lin, J. Bigham, A. Lifchits, and A. Jain. 2006. Names and similarities on the web: fact extraction in the fast lane. In ACL. find
  24. S. Pado and M. Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  25. B. Pang and L. Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115–124. find our card
  26. T. A. Plate. 1995. Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3):623– 641. find
  27. J. B. Pollack. 1990. Recursive distributed representations. Artificial Intelligence, 46, November. find
  28. C. Potts. 2010. On the negativity of negation. In David Lutz and Nan Li, editors, Proceedings of Semantics and Linguistic Theory 20. CLC Publications, Ithaca, NY. find
  29. L. Ratinov, D. Roth, D. Downey, and M. Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In ACL. find
  30. B. Rink and S. Harabagiu. 2010. UTD: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation. find
  31. S. Rudolph and E. Giesbrecht. 2010. Compositional matrix-space models of language. In ACL. find
  32. H. Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24:97–124. find
  33. R. Socher, C. D. Manning, and A. Y. Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop. find
  34. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. 2011a. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS. MIT Press. find
  35. R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. 2011b. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. find
  36. R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. 2011c. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In EMNLP. find our card
  37. P. D. Turney and P. Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  38. D. Widdows. 2008. Semantic vector products: Some initial investigations. In Proceedings of the Second AAAI Symposium on Quantum Interaction. find
  39. A. Yessenalina and C. Cardie. 2011. Compositional matrix-space models for sentiment analysis. In EMNLP. find
  40. F.M. Zanzotto, I. Korkontzelos, F. Fallucchi, and S. Manandhar. 2010. Estimating linear models for compositional distributional semantics. COLING. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/