Elamur Research⌕
☰
home / papers / 2008 / A Structured Vector Space Model for Word Meaning in Context

A Structured Vector Space Model for Word Meaning in Context

Summary

Standard semantic spaces give each lemma a single vector that averages over all its uses, so they cannot show how a word's sense shifts in context. Earlier composition approaches merge a target and its context word into one vector. These approaches ignore the syntactic relation between the two words (for example, "a horse draws" versus "draw a horse") and produce one shared result rather than a separate meaning for each word. Erk and Padó propose a structured vector space model (SVS). In SVS, a word is represented by a lexical vector plus one vector per syntactic relation, each encoding the word's selectional preferences (and inverse preferences) for that argument slot. To get a word's meaning in context, its lexical vector is combined with the context word's preference vector for the relation linking them.

The authors evaluate SVS on judging whether paraphrases fit a given context, using the M&L dataset and a subset of the SemEval Lexical Substitution data. SVS beats the Mitchell & Lapata (M&L) model on both. The gain is significant on the more realistic Lexical Substitution data, which suggests that syntax-aware expectations improve context-sensitive word meaning.

Key points

  • Each word gets a lexical vector plus relation-specific preference vectors, such as subj, obj, and inverse relations like obj⁻¹, subj⁻¹, and comp⁻¹. Selectional preferences are computed as vectors from previously seen arguments.
  • Experiment 2 uses three subsets of the Lexical Substitution data:
    • V-SUBJ: verbs with subjects.
    • V-OBJ: verbs with objects.
    • N-OBJ: nouns as objects, 102 sentences.
  • The task is framed as ranking pooled paraphrases and scored with SemEval's "out of ten" precision (P_OOT).
  • SELPREF-POW (n=30) results:
    • It scores 63.1, 55.8, and 56.9 P_OOT on the three subsets, compared with M&L at 50.3, 52.0, and 53.4.
    • It is significantly better than M&L (p = 0.05) and highly significantly better than the target-only baseline (p < 0.01).
  • Baselines: target-only scores 47–50 and a random baseline scores 22%. A "selpref only" baseline already beats M&L on V-SUBJ and N-OBJ.
  • In the SYN space, vectors with raw frequency components modeled the data, but PMI-weighted vectors did not. On the M&L dataset, SVS was better than M&L but not significantly.

Interesting ideas

  • I1. Relation-indexed expectation vectors. Storing a separate "typical filler" vector per syntactic relation lets the same word pair be interpreted differently depending on its dependency relation. This approach could transfer to dependency-parsed corpora, such as UD treebanks for Ukrainian.
  • I2. Separate in-context vectors for each word. The model produces an adapted vector for each word rather than one merged phrase vector. Sentence structure can then stay in an explicit parse while the vectors carry only lexical meaning, which is a useful separation for entailment or inference pipelines.
  • I3. Ablation baselines that isolate each signal. "Target only" uses no context and "selpref only" uses the context word's expectations alone. Comparing against these shows exactly where the gains come from, and the same diagnostic design suits any context-modeling evaluation.
  • I4. Lenient top-10 evaluation for sparse gold data. Paraphrase annotations are incomplete, so scoring the model's ten best guesses weighted by annotator frequency avoids penalizing valid but unlisted answers. This is relevant whenever gold sets for substitution or synonym tasks are crowd-sourced and patchy.
  • I5. Paraphrase fit as a test of inference rules. Judging whether a paraphrase suits a context can double as a check on when a paraphrase-based inference rule applies. This links lexical semantics evaluation to downstream Textual Entailment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

142 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{erk2008structured,
  title = {{A Structured Vector Space Model for Word Meaning in Context}},
  author = {Erk, Katrin and Padó, Sebastian},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  doi = {10.3115/1613715.1613831},
  url = {https://doi.org/10.3115/1613715.1613831}
}

References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Brockmann, M. Lapata. 2003. Evaluating and combining approaches to selectional preference acquisition. In Proceedings of EACL, 27–34. find
  2. I. Dagan, O. Glickman, B. Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges, Lecture Notes in Computer Science, 177–190. Springer. find
  3. K. Erk. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of ACL, 216–223. find
  4. T. Ferretti, C. Gagné, K. McRae. 2003. Thematic role focusing by participle inflections: evidence form conceptual combination. Journal of Experimental Psychology, 29(1):118–127. find
  5. D. Gildea, D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245– 288. find our card
  6. D. Hindle, M. Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103– 120. find
  7. M. Jones, D. Mewhort. 2007. Representing word meaning and order information in a composite holographic lexicon. Psychological review, 114:1–37. find
  8. J. J. Katz, J. A. Fodor. 1964. The structure of a semantic theory. In The Structure of Language. Prentice-Hall. find
  9. A. Kilgarriff. 1997. I don’t believe in word senses. Computers and the Humanities, 31(2):91–113. find our card
  10. W. Kintsch. 2001. Predication. Cognitive Science, 25:173–202. find
  11. T. Landauer, S. Dumais. 1997. A solution to Platos problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  12. D. Lin. 1993. Principle-based parsing without overgeneration. In Proceedings of ACL, 112–120. find
  13. D. Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of COLING-ACL, 768–774. find our card
  14. K. Lund, C. Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, and Computers, 28:203—208. find
  15. C. D. Manning, P. Raghavan, H. Schütze. 2008. Introduction to Information Retrieval. CUP. find
  16. D. McCarthy, J. Carroll. 2003. Disambiguating nouns, verbs, and adjectives using automatically acquired selectional preferences. Computational Linguistics, 29(4):639–654. find
  17. D. McCarthy, R. Navigli. 2007. SemEval-2007 Task 10: English Lexical Substitution Task. In Proceedings of SemEval, 48–53. find our card
  18. S. McDonald, C. Brew. 2004. A distributional model of semantic context effects in lexical processing. In Proceedings of ACL, 17–24. find
  19. S. McDonald, M. Ramscar. 2001. Testing the distributional hypothesis: The influence of context on judgements of semantic similarity. In Proceedings of CogSci, 611–616. find
  20. K. McRae, M. Spivey-Knowlton, M. Tanenhaus. 1998. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38:283–312. find
  21. K. McRae, M. Hare, J. Elman, T. Ferretti. 2005. A basis for generating expectancies for verbs from nouns. Memory and Cognition, 33(7):1174–1184. find
  22. J. Mitchell, M. Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL, 236– 244. find
  23. A. Moschitti, S. Quarteroni. 2008. Kernels on linguistic structures for answer extraction. In Proceedings of ACL, 113–116, Columbus, OH. find
  24. S. Narayanan, D. Jurafsky. 2002. A Bayesian model predicts human parse preference and reading time in sentence processing. In Proceedings of NIPS, 59–65. find
  25. S. Padó, M. Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  26. U. Padó, F. Keller, M. W. Crocker. 2006. Combining syntax and thematic fit in a probabilistic model of sentence processing. In Proceedings of CogSci, 657–662. find
  27. S. Padó, U. Padó, K. Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP/CoNLL, 400–409. find
  28. P. Resnik. 1996. Selectional constraints: An information-theoretic model and its computational realization. Cognition, 61:127–159. find
  29. G. Salton, A. Wang, C. Yang. 1975. A vector-space model for information retrieval. Journal of the American Society for Information Science, 18:613–620. find
  30. H. Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–124. find
  31. S. Sharoff. 2006. Open-source corpora: Using the net to fish for linguistic data. International Journal of Corpus Linguistics, 11(4):435–462. find
  32. P. Smolensky. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46:159–216. find
  33. I. Szpektor, I. Dagan, R. Bar-Haim, J. Goldberger. 2008. Contextual preferences. In Proceedings of ACL, 683– 691, Columbus, OH. find
  34. Y. Wilks. 1975. Preference semantics. In Formal Semantics of Natural Language. CUP. find
  35. A. Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceeedings of COLING, 947–953. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/erk-2008-structured-vector-space-model-word/