Elamur Research
☰
home / papers / 2007 / Dependency-Based Construction of Semantic Space Models

Dependency-Based Construction of Semantic Space Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J07-2002.pdf

Summary

Classic vector-space models of word meaning count which words appear near a target word, treating every neighbour within a window as equally informative. Padó and Lapata argue this ignores syntax, even though argument structure closely reflects lexical meaning. They propose a general, formalized framework that builds semantic spaces from dependency-parsed text. Context is defined by paths in the dependency graph rather than by linear proximity. Three parameters control construction: which syntactic paths count as context, how different relations are weighted, and how dimensions are represented (plain words, parts of speech, or relation labels). The framework subsumes earlier syntax-aware models such as those of Grefenstette and Lin. The authors test it on semantic priming, TOEFL synonym questions, and unsupervised detection of a word's predominant sense, building on McCarthy et al. (2004). In every task, the dependency-based spaces beat word-window baselines and match or exceed the state of the art. This shows that linguistic structure can enrich distributional models while keeping them simple.

Key points

  • Three new construction parameters:
    • a context selection function that picks the relevant dependency paths;
    • a path value function that weights relations;
    • a basis mapping function that defines what the dimensions are.
  • Context selection settings: contexts are specified as non-lexicalized path templates made of POS1:relation:POS2 edge labels (e.g., V:obj:N, N:nn:N). The appendix lists a "Minimum" set of single-edge paths and a "Medium" set that adds multi-edge paths. These templates ship with the authors' released software package.
  • Evaluation tasks:
    • semantic priming, a cognitive-modelling benchmark;
    • TOEFL synonymy questions;
    • unsupervised predominant-sense ranking for WSD, following McCarthy et al. (2004).
  • Baselines: word-based co-occurrence models in the style of HAL and Lowe & McDonald, which typically use the 100–2,000 most frequent words as dimensions and cosine similarity. LSA is also discussed as a reference point.
  • Future directions: testing the effect of SVD, parser quality, and the choice of similarity measure, and applying the models to languages with freer word order (German, Czech). The paper predicts the gains over window-based models will be larger there.
  • Missing details: the excerpt omits the paper's middle sections, so exact corpus, parser and score figures are not reproduced here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

211 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 206 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2007.33.2.161

@article{pado2007dependency,
  title = {{Dependency-Based Construction of Semantic Space Models}},
  author = {Padó, Sebastian and Lapata, Mirella},
  journal = {Computational Linguistics},
  year = {2007},
  doi = {10.1162/coli.2007.33.2.161},
  url = {https://doi.org/10.1162/coli.2007.33.2.161}
}

References

81 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko and German Rigau. 1996. Word sense disambiguation using conceptual density. In Proceedings of the 16th International Conference on Computational Linguistics, pages 16–22, Copenhagen, Denmark. find
  2. Banerjee, Satanjeev and Ted Pedersen. 2003. Extended gloss overlaps as a measure of semantic relatedness. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, pages 805–810, Acapulco, Mexico. find
  3. Bannard, Colin, Timothy Baldwin, and Alex Lascarides. 2003. A statistical approach to the semantics of verb-particles. In Proceedings of the ACL Workshop on Multiword Expressions: Analysis, Acquisition and Treatment, pages 65–72, Sapporo, Japan. find
  4. Barzilay, Regina. 2003. Information Fusion for Multi-Document Summarization: Paraphrasing and Generation.Ph.D. thesis, Columbia University, New York. find
  5. Berry, Michael W., Susan T. Dumais, and Gavin W. O’Brien. 1994. Using linear algebra for intelligent information retrieval. SIAM Review, 37(4):573–595. find
  6. Briscoe, Ted and John Carroll. 2002. Robust accurate statistical annotation of general text. In Proceedings of the 3rd International Conference on Language Resources and Evaluation, pages 1499–1504, Las Palmas, Canary Islands. find
  7. Budanitsky, Alexander and Graeme Hirst. 2001. Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In Proceedings of ACL Workshop on WordNet and Other Lexical Resources, pages 29–34, Pittsburgh, PA. find
  8. Burnard, Lou, 1995. Users Guide for the British National Corpus. British National Corpus Consortium, Oxford University Computing Service, Oxford, UK. find
  9. Choi, Freddy, Peter Wiemer-Hastings, and Johanna Moore. 2001. Latent Semantic Analysis for text segmentation. In Proceedings of the 6th Conference on Empirical Methods in Natural Language Processing, pages 109–117, Seattle, WA. find
  10. Curran, James R. 2004. From Distributional to Semantic Similarity. Ph.D. thesis, University of Edinburgh. find
  11. Curran, James R. and Marc Moens. 2002. Scaling context space. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 231–238, Philadelphia, PA. find
  12. Dunning, Ted. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find our card
  13. Erkan, Günes and Dragomir R. Radev. 2004. Lexrank: Graph-based centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  14. Fillmore, Charles. 1965. Indirect Object Constructions and the Ordering of Transformations. Mouton, The Hague. find
  15. Fodor, Janet Dean. 1995. Comprehending sentence structure. In Lila R. Gleitman and Mark Liberman, editors, Invitation to Cognitive Science, volume 1. MIT Press, Cambridge, MA, pages 209–246. find
  16. Foltz, Peter W., Walter Kintsch, and Thomas K. Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse Process, 15:285–307. find
  17. Goldberg, Adele. 1995. Constructions. Chicago University Press, Chicago. find
  18. Golub, Gene H. and Charles F. Van Loan. 1989. Matrix Computations. Johns Hopkins Series in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, 3rd edition. find
  19. Green, Georgia. 1974. Semantics and Syntactic Regularity. Indiana University Press, Bloomington. find
  20. Grefenstette, Gregory. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Publishers, Dordrecht. find
  21. Gropen, Jess, Steven Pinker, Michelle Hollander, Richard Goldberg, and Ronald Wilson. 1989. The learnability and acquisition of the dative alternation. Language, 65(2):203–257. find
  22. Harris, Zellig. 1968. Mathematical Structures of Language. Wiley, New York. find
  23. Henderson, James, Paola Merlo, Ivan Petroff, and Gerold Schneider. 2002. Using syntactic analysis to increase efficiency in visualizing text collections. In Proceedings of the 19th International Conference on Computational Linguistics, pages 335–341, Taipei, Taiwan. find
  24. Higgins, Derrick. 2004. Which statistics reflect semantics? Rethinking synonymy and word similarity. In Proceedings of the International Conference on Linguistic Evidence, pages 265–284, Tübingen, Germany. find
  25. Hirst, Graeme and David St-Onge. 1998. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, pages 305–332. find
  26. Hodgson, James M. 1991. Informational constraints on pre-lexical priming. Language and Cognitive Processes, 6:169–205. find
  27. Hoste, Véronique, Iris Hendrickx, Walter Daelemans, and Antal van den Bosch. 2002. Parameter optimization for machine-learning of word sense disambiguation. Language Engineering, 8(4):311–325. find
  28. Howell, David C. 2002. Statistical Methods for Psychology. Duxbury, Pacific Grove, CA, 5th edition. find
  29. Jackendoff, Ray. 1983. Semantic and Cognition. The MIT Press, Cambridge, MA. find
  30. Jiang, Jay J. and David W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th International Conference on Research in Computational Linguistics, pages 19–33, Taipei, Taiwan. find
  31. Jones, Michael P. and James H. Martin. 1997. Contextual spelling correction using Latent Semantic Analysis. In Proceedings of the 5th Conference on Applied Natural Language Processing, pages 166–173, Washington, DC. find
  32. Kanejiya, Dharmendra, Arun Kumar, and Surendra Prasad. 2003. Automatic evaluation of students’ answers using syntactically enhanced LSA. In Proceedings of the HLT-NAACL Workshop on Building Educational Applications Using Natural Language Processing, pages 53–60, Edmonton, Canada. find
  33. Keenan, Edward and Bernard Comrie. 1977. Noun phrase accessibility and universal grammar. Linguistic Inquiry, 8:62–100. find
  34. Kilgarriff, Adam. 2001. Comparing corpora. International Journal of Corpus Linguistics, 6(1):97–133. find
  35. Koeling, Rob, Diana McCarthy, and John Carroll. 2005. Domain-specific sense distributions and predominant sense acquisition. In Proceedings of the Joint Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 419–426, Vancouver, Canada. find
  36. Landauer, Thomas and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  37. Leacock, Claudia and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, pages 265–283. find
  38. Lee, Lillian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 25–32, College Park, MA. find
  39. Lesk, Michael. 1986. Automatic sense disambiguation: How to tell a pine cone from an ice cream cone. In Proceedings of the 1986 Special Interest Group in Documentation, pages 24–26, New York. find
  40. Levin, Beth. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. find
  41. Levy, Joseph P. and John A. Bullinaria. 2001. Learning lexical properties from word usage patterns. In Robert M. French and Jacques P. Sougné, editors, Connectionist Models of Learning, Development, and Evolution. Springer, pages 273–282. find
  42. Lin, Dekang. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of the Joint Annual Meeting of the Association for Computational Linguistics and International Conference on Computational Linguistics, pages 768–774, Montréal, Canada. find
  43. Lin, Dekang. 1998b. Dependency-based evaluation of MINIPAR. In Proceedings of the LREC Workshop on the Evaluation of Parsing Systems, pages 234–241, Granada, Spain. find
  44. Lin, Dekang. 1999. Automatic identification of non-compositional phrases. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 317–324, College Park, MA. find
  45. Lin, Dekang. 2001. LaTaT: Language and text analysis tools. In Proceedings of the 1st Human Language Technology Conference, pages 222–227, San Francisco, CA. find
  46. Lin, Dekang and Patrick Pantel. 2001. Discovery of inference rules for question answering. Natural Language Engineering, 7(4):342–360. find
  47. Lowe, Will. 2001. Towards a theory of semantic space. In Proceedings of the 23rd Annual Conference of the Cognitive Science Society, pages 576–581, Edinburgh, UK. find
  48. Lowe, Will and Scott McDonald. 2000. The direct route: Mediated priming in semantic space. In Proceedings of the 22nd Annual Conference of the Cognitive Science Society, pages 675–680, Philadelphia, PA. find
  49. Lund, Kevin and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, and Computers, 28:203–208. find
  50. Manber, Udi and Sun Wu. 1994. GLIMPSE: a tool to search through entire file systems. In Proceedings of the USENIX Winter 1994 Technical Conference, pages 23–32, San Francisco, CA. find
  51. Manning, Chris and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA. find
  52. McCarthy, Diana, Bill Keller, and John Carroll. 2003. Detecting a continuum of compositionality in phrasal verbs. In Proceedings of the ACL Workshop on Multiword Expressions: Analysis, Acquisition and Treatment, pages 73–80, Sapporo, Japan. find
  53. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant senses in untagged text. In Proceedings of the 42th Annual Meeting of the Association for Computational Linguistics, pages 280–287, Barcelona, Spain. find our card
  54. McDonald, Scott. 2000. Environmental Determinants of Lexical Processing Effort. Ph.D. thesis, University of Edinburgh. find
  55. McDonald, Scott and Chris Brew. 2004. A distributional model of semantic context effects in lexical processing. In Proceedings of the 42th Annual Meeting of the Association for Computational Linguistics, pages 17–24, Barcelona, Spain. find
  56. Mihalcea, Rada and Phil Edmonds, editors. 2004. Proceedings of Senseval-3: The Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, Barcelona, Spain. find
  57. Miltsakaki, Eleni. 2003. The Syntax-Discourse Interface: Effects of the Main-Subordinate Distinction on Attention Structure. Ph.D. thesis, University of Pennsylvania. find
  58. Morris, Robin K. 1994. Lexical and message-level sentence context effects on fixation times in reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, (20):92–103. find
  59. Neville, Helen, Janet L. Nichol, Andrew Barss, Kenneth I. Forster, and Merrill F. Garrett. 1991. Syntactically based sentence prosessing classes: Evidence form event-related brain potentials. Journal of Cognitive Neuroscience, 3:151–165. find
  60. Patel, Malti, John A. Bullinaria, and Joseph P. Levy. 1998. Extracting semantic representations from large text corpora. In Proceedings of the 4th Neural Computation and Psychology Workshop: Connectionist Representations, pages 199–212, London. find
  61. Pedersen, Ted, Siddharth Patwardhan, and Jason Michelizzi. 2004. WordNet::Similarity—measuring the relatedness of concepts. In Proceedings of the Joint Human Language Technology Conference and Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 38–41,… find
  62. Pinker, Steven. 1989. Learnability and Cognition: The Acquisition of Argument Structure. The MIT Press, Cambridge, MA. find
  63. Rada, Roy, Hafedh Mili, and Ellen Bicknell. 1989. Development and application of a metric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics, 19(1):17–30. find
  64. Resnik, Philip. 1995. Using information content to evaluate semantic similarity. In Proceedings of 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montréal, Canada. find
  65. Rubenstein, Herbert and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  66. Sahlgren, Magnus. 2006. The Word-Space Model: Using Distributional Analysis to Represent Syntagmatic and Paradigmatic Relations Between Words in High-Dimensional Vector Spaces. Ph.D. thesis, Stockholm University. find
  67. Salton, Gerard and Michael J. McGill. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York. find
  68. Salton, Gerard and Maria Smith. 1989. On the application of syntactic methodologies in automatic text indexing. In Proceedings of the 12th ACM SIGIR Conference, pages 137–150, Cambridge, MA. find
  69. Salton, G., A. Wang, and C. Yang. 1975. A vector-space model for information retrieval. Journal of the American Society for Information Science, 18:613–620. find
  70. Sampson, Geoffrey R. 1995. English for the Computer. Oxford University Press, Oxford. find
  71. Schütze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–124. find our card
  72. Strzalkowski, Tomek, editor. 1999. Natural Language Information Retrieval. Kluwer Academic Publishers, Dordrecht. find
  73. Talmy, L. 1985. Lexicalisation patterns: Semantic structure in lexical forms. In T. Shopen, editor, Language Typology and Syntactic Description III: Grammatical Categories and the Lexicon. Cambrige University Press, Cambridge, pages 57–149. find
  74. Tesnière, Lucien. 1959. Elements de syntaxe structurale. Klincksieck, Paris. find
  75. Turney, Peter D. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of the 12th European Conference on Machine Learning, pages 491–502, Freiburg, Germany. find
  76. Voorhees, Ellen M. 1999. Natural language processing and information retrieval. In 2nd School on Information Extraction (SCIE99), pages 32–48, Frascati (Rome), Italy. find
  77. Weeds, Julie. 2003. Measures and Applications of Lexical Distributional Similarity. Ph.D. thesis, University of Sussex, UK. find
  78. West, R. F. and K. E. Stanovich. 1986. Robust effects of syntactic structure on visual word processing. Journal of Memory and Cognition, 14:104–112. find
  79. Widdows, Dominic. 2003. Unsupervised methods for developing taxonomies by combining syntactic and statistical information. In Proceedings of the Joint Human Language Technology Conference and Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 197–204… find
  80. Wiemer-Hastings, Peter and Iraide Zipitria. 2001. Rules for syntax, vectors for semantics. In Proceedings of the 23rd Annual Conference of the Cognitive Science Society, pages 1140–1145, Edinburgh, UK. find
  81. Yarowsky, David and Radu Florian. 2002. Evaluating sense disambiguation across diverse parameter spaces. Natural Language Engineering, 9(4):293–310. find