Elamur Research⌕
☰
home / papers / 2012 / A Comparison of Vector-based Representations for Semantic Composition

A Comparison of Vector-based Representations for Semantic Composition

Summary

The paper asks how much the choice of word-level vectors matters when building meaning representations for phrases and sentences, as opposed to the choice of composition function. The authors cross three distributional representations with three ways of combining them, giving nine compositional models. The representations are a simple window-based co-occurrence space built from the BNC (SDS), Collobert and Weston-style neural embeddings (NLM), and Baroni and Lenci's Distributional Memory tensor (DM). The composition methods are vector addition, element-wise multiplication, and a recursive autoencoder (RAE). The models are tested on two tasks: human similarity judgments for short phrases (the Mitchell and Lapata 2010 data) and sentence-level paraphrase detection on the Microsoft Research paraphrase corpus. Across both tasks, cheap, syntax-blind combinations over the simple space hold their own against far more expensive alternatives. The authors conclude that the match between representation and composition method matters more than corpus or vector size. This matters for anyone deciding whether heavy resources such as parsers, huge corpora or long neural training are actually worth the cost.

Key points

  • SDS: This space uses a ±5-word window over the ~100M-token BNC. Its 2,000 dimensions are frequent context words, each weighted by a probability ratio. It needs only tokenization: no parsing and no learning.
  • NLM: The embeddings were trained on the BNC and tuned on WordSim353. The best setting was 50 dimensions, a context window of 4 and a learning rate of 10⁻⁹. Final training took about 2 weeks (≈1.5×10⁹ iterations).
  • DM: This representation is a third-order tensor of weighted word–link–word tuples. It is built from a parsed corpus of roughly 3 billion words.
  • Paraphrase detection results: The classifier used only three features: the cosine between composed sentence vectors, the two sentence lengths, and unigram overlap.
    • SDS + multiplication reached 73.0 accuracy and 82.3 F1.
    • DM + addition reached 73.5 accuracy and 82.2 F1.
    • The baseline scored 66.5 / 79.9, and Socher et al. (2011a) scored 76.8 / 83.6.
    • Accuracy gaps of 3.3 or more are significant at p<0.01.
  • Phrase similarity: Syntax-oblivious addition and multiplication beat the RAE. Neither NLM nor DM surpassed the simple SDS space. The RAE did better when its output fed a paraphrase classifier.

Interesting ideas

  • I1. Factorial design: Every representation is crossed with every composition function, which separates the effect of the word vectors from the effect of the combiner. This setup suits any pipeline where input features and an aggregation step are chosen independently.
  • I2. Two-level evaluation: Pairing a phrase-level task that stresses the representation with a sentence-level task that stresses composition covers compositionality from short to long units. A single benchmark could hide weaknesses at one of these levels.
  • I3. Minimal probing classifier: Restricting the downstream classifier to a cosine, two lengths and word overlap keeps the test about the quality of the composed vectors rather than feature engineering. This is a cheap, transparent way to compare embedding methods on new languages or domains.
  • I4. Simple baselines first: A tokenization-only co-occurrence space with addition or multiplication should be the first baseline. It can match parsed, billion-word resources, which is relevant for lower-resource settings such as Ukrainian.
  • I5. Joint design of representation and composition: The authors suggest that word vectors and the composition rule should be learned together rather than chosen separately. This principle could guide future model design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 81 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blacoe2012comparison,
  title = {{A Comparison of Vector-based Representations for Semantic Composition}},
  author = {Blacoe, William and Lapata, Mirella},
  booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  year = {2012},
  url = {https://aclanthology.org/D12-1050/}
}

References

37 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673– 721. find our card
  2. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1183–1193, Cambridge, MA. find
  3. Jerome R. Bellegarda. 2000. Exploiting latent semantic information in statistical language modeling. Proceedings of the Institute of of Electrical and Electronics Engineers, 88(8):1279–1296. find
  4. Yoshua Bengio. 2001. Neural net language models. Scholarpedia, 3(1):3881. find
  5. Stephen Clark, Bob Coecke, and Mehrnoosh Sadrzadeh. 2008. A compositional distributional model of meaning. In Proceedings of the 2nd Quantum Interaction Symposium, pages 133–140, Oxford, UK. find
  6. Daoud Clarke. 2012. A context-theoretic framework for compositionality in distributional semantics. Computational Linguistics, 38(1):41–71. find
  7. J Cohen and P Cohen. 1983. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum. find
  8. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167, New York, NY. ACM. find
  9. Dipanjan Das and Noah A. Smith. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 468–476, Suntec,… find our card
  10. G. Denhire and B. Lemaire. 2004. A computational model of children’s semantic memory. In Proceedings of the 26th Annual Meeting of the Cognitive Science Society, pages 297–302, Mahwah, NJ. Lawrence Erlbaum Associates. find
  11. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics, pages 350–356, Geneva, Switzerland. COLING. find our card
  12. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874. find
  13. Samuel Fernando and Mark Stevenson. 2008. A semantic similarity approach to paraphrase detection. Technology. find
  14. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. In WWW, pages 406–414. find
  15. Peter Foltz, Walter Kintsch, and Thomas Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse Process, 15:285– 307. find
  16. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011a. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1394–1404, Edinburgh, Scotland. find
  17. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011b. Experimenting with transitive verbs in a DisCoCat. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 62–66, Edinburgh, UK. find
  18. Gregory Grefenstette. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Publishers, Norwell, MA. find
  19. Thomas L. Griffiths, Mark Steyvers, and Joshua B. Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114(2):211–244. find
  20. Aminul Islam and Diana Inkpen. 2007. Semantic similarity of short texts. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, Borovets, Bulgaria. find
  21. Walter Kintsch. 2001. Predication. Cognitive Science, 25(2):173–202. find
  22. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. find our card
  23. T. K. Landauer and S. T. Dumais. 1997. A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 104(2):211–240. find
  24. Scott McDonald. 2000. Environmental Determinants of Lexical Processing Effort. Ph.D. thesis, University of Edinburgh. find
  25. Rada Mihalcea, Courtney Corley, and Carlo Strapparava, 2006. Corpus-based and knowledge-based measures of text semantic similarity, pages 775–780. AAAI Press. find
  26. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 38(8):1388–1429. find
  27. Jeff Mitchell. 2011. Composition in distributional models of semantics. Ph.D. thesis, University of Edinburgh. find
  28. Tony A. Plate. 1995. Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3):623–641. find
  29. Long Qiu, Min-Yen Kan, and Tat-Seng Chua. 2006. Paraphrase recognition via dissimilarity significance classification. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 18–26, Sydney, Australia. find
  30. Vasile Rus, Philip M. McCarthy, Mihai C. Lintean, Danielle S. McNamara, and Arthur C. Graesser. 2008. Paraphrase identification with lexico-syntactic graph subsumption. In David Wilson and H. Chad Lane, editors, Florida Artificial Intelligence Research Society Conference, pages 201–206. AAAI Press. find
  31. Hinrich Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–124. find our card
  32. Paul Smolensky. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46:159– 216. find
  33. Richard Socher, Eric H. Huang, Jeffrey Pennin, Andrew Y. Ng, and Christopher D. Manning. 2011a. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Advances in Neural Information Processing Systems 24, pages 801– 809. Granada, Spain. find
  34. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 151–161, Edinburgh, Scot… find our card
  35. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394, Uppsala, Sweden. find our card
  36. Peter D. Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. find our card
  37. Stephen Wan, Mark Dras, Robert Dale, and Cecile Paris. 2006. Using dependency-based features to take the “para-farce” out of paraphrase. In Proceedings of the Australasian Language Technology Workshop 2006, pages 131–138, Sydney, Australia. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blacoe-2012-comparison-vector-based-representations-semantic/