Elamur Research⌕
☰
home / papers / 2010 / Estimating Linear Models for Compositional Distributional Semantics

Estimating Linear Models for Compositional Distributional Semantics

Summary

The paper addresses how to set the parameters of additive compositional distributional semantics (CDS) models, which build a vector for a two-word phrase from the vectors of its words. Earlier work mostly used fixed or hand-tuned scalar weights. The authors generalise the additive model so that each word vector is multiplied by its own full square matrix (z = Ax + By). Learning these matrices is a regression problem with many output variables. They stack training triples into a linear system and solve it by least squares, using the Moore-Penrose pseudo-inverse computed through a truncated SVD. Training triples come from dictionaries: a WordNet headword whose gloss is a two-word phrase supplies the expected phrase vector, and Wiktionary supplies negative examples. Word vectors are built from the British National Corpus. Models are compared by how clearly they separate positive from negative pairs, measured with a t-test p-value. The estimated additive model (EAM) is the only one that reaches significance on both noun-phrase and verb-phrase data. This shows that learning richer composition parameters from data beats tuning simple weights.

Key points

  • Models compared: EAM against four baselines:
    • the basic additive model (BAM), with α = β = 1;
    • the basic multiplicative model (BMM);
    • the selectional-preference variants of Erk and Padó, BAM-SP and BMM-SP.
  • Data: two sets, with WordNet positives and Wiktionary negatives, each split 50/50 into training and testing.
    • NN set: 1065 WordNet pairs and 377 Wiktionary pairs.
    • VN set: 161 WordNet pairs and 111 Wiktionary pairs.
  • Vectors: derived from British National Corpus text, where each sentence serves as a context window and words act as features, giving 689191 dimensions.
  • Main result (k = 20):
    • EAM p-values are about 0.0043 on NN and 0.0045 on VN, both significant.
    • Each SP baseline reaches significance on only one set: BMM-SP on NN, BAM-SP on VN.
  • Tuning BAM is not enough: no setting of α (with β = 1 − α) beats EAM once k is large. EAM becomes significant from k ≥ 10 on NN and at k = 20 on VN.

Interesting ideas

  • I1. Dictionary glosses as free supervision: a word defined by a short phrase gives a natural target vector for composing that phrase. Glossaries and wordnets in other languages could supply training data for composition models in the same way.
  • I2. Composition learned as multi-output linear regression: fitting [A B] by least squares reduces learning the composition function to one closed-form pseudo-inverse. There is no iterative training, so the approach is easy to reuse for other vector-mapping tasks.
  • I3. SVD rank k as a capacity knob: truncating the pseudo-inverse to k singular values gives a family of models ranging from very constrained to expressive. This allows a controlled comparison with simpler baselines, since k = 1 has as many free parameters as BAM.
  • I4. Evaluation by separability: a model is ranked by the t-test p-value between similarity scores of correct and incorrect pairs. This requires no human similarity ratings, only positive and negative sets, so it is cheap to apply in low-resource settings.
  • I5. Testing per syntactic relation: running separate adjective/noun-noun and verb-noun sets checks whether a method's gains hold across construction types. This is a simple robustness check worth copying.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓University of YorkYork · United Kingdom10 works · 6 464 citations🎓University of Rome Tor VergataRome · Italy5 works · 1 207 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zanzotto2010estimating,
  title = {{Estimating Linear Models for Compositional Distributional Semantics}},
  author = {Zanzotto, Fabio Massimo and Korkontzelos, Ioannis and Fallucchi, Francesca and Manandhar, Suresh},
  booktitle = {Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)},
  year = {2010},
  url = {https://aclanthology.org/C10-1142/}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Charniak, Eugene. 2000. A maximum-entropy-inspired parser. In proceedings of the 1st NAACL, pages 132–139, Seattle, Washington. find our card
  2. Cook, Paul, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-Tokens Dataset. In proceedings of the LREC Workshop: Towards a Shared Task for Multiword Expressions (MWE 2008), Marrakech, Morocco. find
  3. Deerwester, Scott C., Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391–407. find
  4. Erk, Katrin and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 897– 906. Association for Computational Linguistics. find our card
  5. Erk, Katrin. 2007. A simple, similarity-based model for selectional preferences. In proceedings of ACL. Association for Computer Linguistics. find
  6. Fallucchi, Francesca and Fabio Massimo Zanzotto. 2009. SVD feature selection for probabilistic taxonomy learning. In proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 66–73. Association for Computational Linguistics, Athens, Greece. find
  7. Firth, John R. 1957. Papers in Linguistics. Oxford University Press, London. find
  8. Golub, Gene and William Kahan. 1965. Calculating the singular values and pseudo-inverse of a matrix. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis, 2(2):205–224. find
  9. Harris, Zellig. 1964. Distributional structure. In Katz, Jerrold J. and Jerry A. Fodor, editors, The Philosophy of Linguistics, New York. Oxford University Press. find
  10. Jones, Michael N. and Douglas J. K. Mewhort. 2007. Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114:1–37. find
  11. Kim, Su N. and Timothy Baldwin. 2008. Standardised evaluation of english noun compound interpretation. In proceedings of the LREC Workshop: Towards a Shared Task for Multiword Expressions (MWE 2008), pages 39–42, Marrakech, Morocco. find
  12. Korkontzelos, Ioannis and Suresh Manandhar. 2009. Detecting compositionality in multi-word expressions. In proceedings of ACL-IJCNLP 2009, Singapore. find our card
  13. Li, Ping, Curt Burgess, and Kevin Lund. 2000. The acquisition of word meaning through global lexical co-occurrences. In proceedings of the 31st Child Language Research Forum. find
  14. Lin, Dekang and Patrick Pantel. 2001. DIRT-discovery of inference rules from text. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD-01). San Francisco, CA. find
  15. Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK. find
  16. McCarthy, Diana and John Carroll. 2003. Disambiguating nouns, verbs, and adjectives using automatically acquired selectional preferences. Computational Linguistics, 29(4):639–654. find
  17. Miller, George A. and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, VI:1–28. find
  18. Miller, George A. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41. find our card
  19. Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In proceedings of ACL-08: HLT, pages 236–244, Columbus, Ohio. Association for Computational Linguistics. find
  20. Nicholson, Jeremy and Timothy Baldwin. 2008. Interpreting compound nominalisations. In proceedings of the LREC Workshop: Towards a Shared Task for Multiword Expressions (MWE 2008), pages 43–45, Marrakech, Morocco. find
  21. Pado, Sebastian and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161– 199. find our card
  22. Penrose, Roger. 1955. A generalized inverse for matrices. In Proceedings of Cambridge Philosophical Society. find
  23. Pollard, Carl J. and Ivan A. Sag. 1994. Head-driven Phrase Structured Grammar. Chicago CSLI, Stanford. find
  24. Schone, Patrick and Daniel Jurafsky. 2001. Is knowledge-free induction of multiword unit dictionary headwords a solved problem? In Lee, Lillian and Donna Harman, editors, proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 100–108. find
  25. Villavicencio, Aline. 2003. Verb-particle constructions and lexical resources. In proceedings of the ACL 2003 workshop on Multiword expressions, pages 57–64, Morristown, NJ, USA. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zanzotto-2010-estimating-linear-models-compositional-distributional/