Elamur Research
☰
home / papers / 2010 / Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space

Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D10-1115.pdf

Summary

The paper examines how, in distributional semantics, to build the vector meaning of an adjective–noun (AN) combination from the meanings of its constituents. The authors draw on an idea from formal semantics. They represent the noun as a vector and the adjective as a linear function encoded as a matrix. This function maps the noun vector to the vector of the whole combination. A separate matrix is trained for each adjective using partial least squares regression on pairs of the form "noun vector – AN vector". These pairs are collected automatically from a large corpus, so no manual annotation is required. The model was tested on reconstructing AN vectors that were not among the training examples. It clearly outperformed the additive and multiplicative models of Mitchell and Lapata, as well as Guevara's approach. A small error analysis shows that the discrepancies often stem not from the model but from unreliable corpus vectors of rare combinations. In addition, the approach yields two new ways of describing the meaning of an adjective: through its matrix or through the averaged vector of the combinations it occurs in. Both clustered adjectives better than ordinary co-occurrence vectors. This is an early and influential step toward compositional distributional semantics that combines linguistic theory with corpus data.

Key points

  • The training corpus combines ukWaC, a mid-2009 dump of the English Wikipedia, and the British National Corpus.
  • The space after SVD has 300 dimensions, so each adjective is a 300×300 matrix (unrolled into a vector of about 90 thousand values). In Mitchell and Lapata's terms, this is an additive model in which the contribution of the adjective vector is set to zero. The approach differs from Guevara's in that it learns a separate function for each adjective rather than a single shared one.
  • The rarer a combination is in the corpus, the worse the model approximates its observed vector (correlation ρ = −0.48). The authors argue that for such combinations the predicted vector may be even better than the corpus vector.
  • Clustering of 19 adjectives into 4 classes (color, positive evaluation, time, size) was performed in CLUTO, with quality evaluated by purity. The matrices and the AN centroids each achieved 73.7%, direct co-occurrence vectors 68.4%, and the random baseline 45.9%.
  • Among future directions, the authors name recursive combinations (several adjectives before a noun), affixation (e.g., the prefix re-), determiner constructions (determiner N), and gradual scaling up to whole sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

228 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 220 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baroni2010nouns,
  title = {{Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space}},
  author = {Baroni, Marco and Zamparelli, Roberto},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1115/}
}

References

34 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Clark and S. Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the First Symposium on Quantum Interaction, pages 52–55. find
  2. B. Efron and R. Tibshirani. 1994. An Introduction to the Bootstrap. Chapman and Hall, Boca Raton, FL. find
  3. K. Erk and S. Padó. 2008. A structured vector space model for word meaning in context. In Proceedings of EMNLP, pages 897–906. find
  4. K. Erk and S. Padó. 2009. Paraphrase assessment in structured vector space: Exploring parameters and datasets. In Proceedings of the EACL GEMS Workshop, pages 57–65. find
  5. S. Evert. 2005. The Statistics of Word Cooccurrences. Dissertation, Stuttgart University. find
  6. P. Foltz, W. Kintsch, and Th. Landauer. 1998. The measurement of textual coherence with Latent Semantic Analysis. Discourse Processes, 25:285–307. find
  7. G. Frege. 1892. Über sinn und bedeutung. Zeitschrift fuer Philosophie un philosophische Kritik, 100. find
  8. E. Guevara. 2010. A regression model of adjective-noun compositionality in distributional semantics. In Proceedings of the ACL GEMS Workshop, pages 33–37. find
  9. T. Hastie, R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning, 2nd ed. Springer, New York. find
  10. M. Jones and D. Mewhort. 2007. Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114:1–37. find
  11. H. Kamp. 1975. Two theories about adjectives. In E. Keenan, editor, Formal Semantics of Natural Language, pages 123–155. Cambridge University Press. find
  12. G. Karypis. 2003. CLUTO: A clustering toolkit. Technical Report 02-017, University of Minnesota Department of Computer Science. find
  13. W. Kintsch. 2001. Predication. Cognitive Science, 25(2):173–202. find
  14. Th. Landauer and S. Dumais. 1997. A solution to Plato’s problem: The Latent Semantic Analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  15. B. Mevik and R. Wehrens. 2007. The pls package: Principal component and partial least squares regression in R. Journal of Statistical Software, 18(2). find
  16. J. Mitchell and M. Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL, pages 236–244. find
  17. J. Mitchell and M. Lapata. 2009. Language models based on semantic composition. In Proceedings of EMNLP, pages 430–439. find
  18. R. Montague. 1970a. English as a formal language. In B. Visentini, editor, Linguaggi nella Società e nella Tecnica, pages 189–224. Edizioni di Comunità, Milan. Reprinted in Thomason (1974). find
  19. R. Montague. 1970b. Universal grammar. Theoria, 36:373–398. Reprinted in Thomason (1974). find
  20. R. Montague. 1973. The proper treatment of quantification in English. In K.J.J. Hintikka, editor, Approaches to Natural Language, pages 221–242. Reidel, Dordrecht. Reprinted in Thomason (1974). find
  21. B. Partee. 2004. Compositionality. In Compositionality in Formal Semantics: Selected Papers by Barbara H. Partee. Blackwell, Oxford. find
  22. R. Rapp. 2003. Word sense discovery based on sense descriptor dissimilarity. In Proceedings of the MT Summit, pages 315–322. find
  23. H. Rubenstein and J. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  24. S. Rudolph and E. Giesbrecht. 2010. Compositional matrix-space models of language. In Proceedings of ACL. find
  25. M. Sahlgren, A. Holst, and P. Kanerva. 2008. Permutations as a means to encode order in word space. In Proceedings of CogSci, pages 1300–1305. find
  26. M. Sahlgren. 2006. The Word-Space Model. Dissertation, Stockholm University. find
  27. H. Schmid. 1995. Improvements in part-of-speech tagging with an application to German. In Proceedings of the EACL-SIGDAT Workshop. find
  28. H. Schütze. 1997. Ambiguity Resolution in Natural Language Learning. CSLI, Stanford, CA. find
  29. M. Siegel. 1976. Capturing the Adjective. Ph.D. thesis, University of Massachusetts at Amherst. find
  30. P. Smolensky. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist networks. Artificial Intelligence, 46:159–216. find
  31. R. H. Thomason, editor. 1974. Formal Philosophy: Selected Papers of Richard Montague. Yale University Press, New York. find
  32. P. Turney and P. Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  33. D. Widdows. 2008. Semantic vector products: Some initial investigations. In Proceedings of the Second Symposium on Quantum Interaction, Oxford. find
  34. Y. Zhao and G. Karypis. 2003. Criterion functions for document clustering: Experiments and analysis. Technical Report 01-40, University of Minnesota Department of Computer Science. find