Elamur Research
☰
home / papers / 2010 / Distributional Memory: A General Framework for Corpus-Based Semantics

Distributional Memory: A General Framework for Corpus-Based Semantics

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J10-4006.pdf

Summary

Distributional semantic models (DSMs) are usually built one task at a time. Each problem, from synonym detection to relation classification, gets its own corpus statistics, representation and algorithm. Baroni and Lenci argue that this fragmentation is inefficient, prone to overfitting, and unconvincing as a model of human semantic memory. Their alternative is Distributional Memory (DM), a single reusable resource. Corpus co-occurrences are extracted only once and stored as weighted word–link–word tuples in a third-order tensor. For each problem, the tensor is turned into a suitable matrix, and its rows and columns provide the semantic space, for example for words or for word pairs. One DM model is tested on an unusually wide set of tasks, from similarity judgments and analogies to qualia harvesting and verb alternations. Without task-specific tuning, the best DM implementation competes with specialized algorithms and with the authors' reimplementations of state-of-the-art methods. It sometimes trails them, usually by small margins. The work shows that a general distributional resource, similar in spirit to WordNet, is achievable.

Key points

  • The core data structure is a third-order tensor of weighted word–link–word tuples. Task-specific matrices are derived from it as different views of the same corpus data, so the corpus never has to be reprocessed.
  • The evaluation covers nine task families:
    • word similarity judgments
    • synonym discovery
    • concept categorization
    • verb selectional preferences
    • analogy solving
    • relation classification between word pairs
    • qualia structure harvesting
    • prediction of typical concept properties
    • verb alternation classification
  • Comparisons include task-specific results from the literature and the authors' own reimplementations of state-of-the-art methods. Turney's PairClass is discussed as a competing general approach that requires keeping the full corpus available.
  • The paper separates three components: an unsupervised acquisition phase (tuple extraction and weighting), a declarative memory (the tensor), and procedural task solvers that may be supervised, such as the nearest-centroid method used for most non-attributional tasks.
  • DM triples resemble semantic-network and RDF representations. This could ease integration with other knowledge sources and with cognitive models such as Rogers and McClelland (2004). Compositionality is identified as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

220 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 214 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00016

@article{baroni2010distributional,
  title = {{Distributional Memory: A General Framework for Corpus-Based Semantics}},
  author = {Baroni, Marco and Lenci, Alessandro},
  journal = {Computational Linguistics},
  year = {2010},
  doi = {10.1162/coli\_a\_00016},
  url = {https://doi.org/10.1162/coli_a_00016}
}

References

116 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aerts, Diederik and Marek Czachor. 2004. Quantum aspects of semantic analysis and symbolic artificial intelligence. Journal of Physics A: Mathematical and General, 37:123–132. find
  2. Alishahi, Afra and Suzanne Stevenson. 2008. A distributional account of the semantics of multiword expressions. Italian Journal of Linguistics, 20(1):157–179. find
  3. Almuhareb, Abdulrahman. 2006. Attributes in Lexical Acquisition. Ph.D. thesis, University of Essex. find
  4. Almuhareb, Abdulrahman and Massimo Poesio. 2004. Attribute-based and value-based clustering: An evaluation. In Proceedings of EMNLP, pages 158–165, Barcelona. find
  5. Almuhareb, Abdulrahman and Massimo Poesio. 2005. Concept learning and categorization from the web. In Proceedings of CogSci, pages 103–108, Stresa. find
  6. Baldwin, Timothy, Valia Kordoni, and Aline Villavicencio. 2009. Prepositions in applications: A survey and introduction to the special issue. Computational Linguistics, 35(2):119–149. find
  7. Baroni, Marco, Eduard Barbu, Brian Murphy, and Massimo Poesio. 2010. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222–254. find
  8. Baroni, Marco, Stefan Evert, and Alessandro Lenci, editors. 2008. Bridging the Gap between Semantic Theory and Computational Simulations: Proceedings of the ESSLLI Workshop on Distributional Lexical Semantic. FOLLI, Hamburg. find
  9. Baroni, Marco and Alessandro Lenci. 2008. Concepts and properties in word spaces. Italian Journal of Linguistics, 20(1):55–88. find
  10. Baroni, Marco and Alessandro Lenci. 2009. One distributional memory, many semantic tasks. In Proceedings of the EACL GEMS Workshop, pages 1–8, Athens. find
  11. Biciçi, Ergun and Deniz Yuret. 2006. Clustering word pairs to answer analogy questions. In Proceedings of the Fifteenth Turkish Symposium on Artificial Intelligence and Neural Networks, pages 277–284, Muǧla. find
  12. Bruza, Peter and Richard Cole. 2005. Quantum logic of semantic space: An exploratory investigation of context effects in practical reasoning. In Sergei Artemov, Howard Barringer, Arthur d’Avila Garcez, Luis C. Lamb, and John Woods, editors, We Will Show Them: Essays in Honour of Dov Gabbay, volume o… find
  13. Buitelaar, Paul, Philipp Cimiano, and Bernardo Magnini. 2005. Ontology Learning from Text. IOS Press, Amsterdam. find
  14. Bullinaria, John and Joseph Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39:510–526. find
  15. Chen, Hsin-Hsi, Ming-Shun Lin, and Yu-Chuan Wei. 2006. Novel association measures using Web search with double checking. In Proceedings of COLING-ACL, pages 1009–1016, Sydney. find
  16. Church, Kenneth and Peter Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29. find our card
  17. Cimiano, Philipp and Johanna Wenderoth. 2007. Automatic acquisition of ranked qualia structures from the Web. In Proceedings of ACL, pages 888–895, Prague. find
  18. Clark, Stephen and Stephen Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the AAAI Spring Symposium on Quantum Interaction, pages 52–55, Stanford, CA. find
  19. Curran, James and Marc Moens. 2002. Improvements in automatic thesaurus extraction. In Proceedings of the ACL Workshop on Unsupervised Lexical Acquisition, pages 59–66, Philadelphia, PA. find
  20. Davidov, Dmitry and Ari Rappoport. 2008a. Classification of semantic relationships between nominals using pattern clusters. In Proceedings of ACL, pages 227–235, Columbus, OH. find
  21. Davidov, Dmitry and Ari Rappoport. 2008b. Unsupervised discovery of generic relationships using pattern clusters and its evaluation by automatically generated SAT analogy questions. In Proceedings of ACL, pages 692–700, Columbus, OH. find
  22. Dietterich, Thomas. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1924. find
  23. Dowty, David. 1977. Word Meaning and Montague Grammar. Kluwer, Dordrecht. find
  24. Dunning, Ted. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find our card
  25. Efron, Bradley and Robert Tibshirani. 1994. An Introduction to the Bootstrap. Chapman and Hall, Boca Raton, FL. find
  26. Erk, Katrin. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of ACL, pages 216–223, Prague. find
  27. Erk, Katrin and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In Proceedings of EMNLP, pages 897–906, Honolulu, HI. find
  28. Erk, Katrin and Sebastian Padó. 2009. Paraphrase assessment in structured vector space: Exploring parameters and datasets. In Proceedings of the EACL GEMS Workshop, pages 57–65, Athens. find
  29. Evert, Stefan. 2005. The Statistics of Word Cooccurrences. Ph.D. dissertation, Stuttgart University. find
  30. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
  31. Garrard, Peter, Matthew Lambon Ralph, John Hodges, and Karalyn Patterson. 2001. Prototypicality, distinctiveness, and intercorrelation: Analyses of the semantic attributes of living and nonliving concepts. Cognitive Neuropsychology, 18(2):25–174. find
  32. Geeraerts, Dirk. 2010. Theories of Lexical Semantics. Oxford University Press, Oxford. find
  33. Girju, Roxana, Adriana Badulescu, and Dan Moldovan. 2006. Automatic discovery of part-whole relations. Computational Linguistics, 32(1):83–135. find
  34. Girju, Roxana, Preslav Nakov, Vivi Nastase, Stan Szpakowicz, Peter Turney, and Deniz Yuret. 2007. SemEval-2007 task 04: Classification of semantic relations between nominals. In Proceedings of SemEval 2007, pages 13–18, Prague. find
  35. Grefenstette, Gregory. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer, Boston, MA. find
  36. Griffiths, Tom, Mark Steyvers, and Josh Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114:211–244. find
  37. Harris, Zellig. 1954. Distributional structure. Word, 10(2-3):1456–1162. find
  38. Hearst, Marti. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of COLING, pages 539–545, Nantes. find
  39. Hearst, Marti. 1998. Automated discovery of WordNet relations. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, pages 131–151. find
  40. Herdaǧdelen, Amaç and Marco Baroni. 2009. BagPack: A general framework to represent semantic relations. In Proceedings of the EACL GEMS Workshop, pages 33–40, Athens. find
  41. Herdaǧdelen, Amaç, Katrin Erk, and Marco Baroni. 2009. Measuring semantic relatedness with vector space models and random walks. In Proceedings of TextGraphs-4, pages 50–53, Singapore. find
  42. Heylen, Kris, Yves Peirsman, Dirk Geeraerts, and Dirk Speelman. 2008. Modelling word similarity: An evaluation of automatic synonymy extraction algorithms. In Proceedings of LREC, pages 3243–3249, Marrakech. find
  43. Hou, Yuexian and Dawei Song. 2009. Characterizing pure high-order entanglements in lexical semantic spaces via information geometry. In Peter Bruza, Donald Sofge, William Lawless, and C. J. van Rijsbergen, editors, Quantum Interaction: Third International Symposium, QI 2009. Springer, Berlin, pages… find
  44. Jackendoff, Ray. 1990. Semantic Structures. MIT Press, Cambridge, MA. find
  45. Joanis, Eric, Suzanne Stevenson, and David James. 2008. A general feature space for automatic verb classification. Natural Language Engineering, 14(3):337–367. find
  46. Jones, Michael and Douglas Mewhort. 2007. Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114:1–37. find
  47. Karypis, George. 2003. CLUTO: A clustering toolkit. Technical Report 02-017, University of Minnesota Department of Computer Science, Minneapolis. find
  48. Kilgarriff, Adam, Pavel Rychly, Pavel Smrz, and David Tugwell. 2004. The Sketch Engine. In Proceedings of Euralex, pages 105–116, Lorient. find
  49. Kolda, Tamara. 2006. Multilinear operators for higher-order decompositions. Technical Report 2081, SANDIA, Albuquerque, NM. find
  50. Kolda, Tamara and Brett Bader. 2009. Tensor decompositions and applications. SIAM Review, 51(3):455–500. find
  51. Kolda, Tamara and Jimeng Sun. 2008. Scalable tensor decompositions for multi-aspect data mining. In Proceedings of ICDM, pages 94–101, Pisa. find
  52. Landauer, Thomas and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  53. Lenci, Alessandro. 2008. Distributional approaches in linguistic and cognitive research. Italian Journal of Linguistics, 20(1):1–31. find
  54. Lenci, Alessandro. 2010. The life cycle of knowledge. In Chu-Ren Huang, Nicoletta Calzolari, Aldo Gangemi, Alessandro Lenci, Alessandro Oltramari, and Laurent Prévot, editors, Ontology and the Lexicon. A Natural Language Processing Perspective. Cambridge University Press, Cambridge, UK, pages 241–2… find
  55. Levin, Beth. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago, IL. find
  56. Levin, Beth and Malka Rappaport-Hovav. 2005. Argument Realization. Cambridge University Press, Cambridge, UK. find
  57. Lin, Dekang. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of COLING-ACL, pages 768–774, Montreal. find
  58. Lin, Dekang. 1998b. An information-theoretic definition of similarity. In Proceedings of ICML, pages 296–304, Madison, WI. find
  59. Liu, Hugo and Push Singh. 2004. ConceptNet: A practical commonsense reasoning toolkit. BT Technology Journal, pages 211–226. find
  60. Lowe, Will. 2001. Towards a theory of semantic space. In Proceedings of CogSci, pages 576–581, Edinburgh, UK. find
  61. Lund, Kevin and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28:203–208. find
  62. Manning, Chris and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA. find
  63. Markman, Arthur B. 1999. Knowledge Representation. Psychology Press, New York, NY. find
  64. Matveeva, Irina, Gina-Anne Levow, Ayman Farahat, and Christian Royer. 2005. Generalized latent semantic analysis for term representation. In Proceedings of RANLP, pages 60–68, Borovets. find
  65. McRae, Ken, George Cree, Mark Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4):547–559. find
  66. McRae, Ken, Michael Spivey-Knowlton, and Michael Tanenhaus. 1998. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38:283–312. find
  67. Merlo, Paola and Suzanne Stevenson. 2001. Automatic verb classification based on statistical distributions of argument structure. Computational Linguistics, 27(3):373–408. find
  68. Meyer, Carl. 2000. Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia, PA. find
  69. Miller, George and Walter Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6:1–28. find
  70. Minkov, Einat and William Cohen. 2007. Learning to rank typed graph walks: Local and global approaches. In Proceedings of WebKDD/SNA-KDD, pages 1–8, San José, CA. find
  71. Minkov, Einat and William Cohen. 2008. Learning graph walk based similarity measures for parsed text. In Proceedings of EMNLP, pages 907–916, Honolulu, HI. find
  72. Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL, pages 236–244, Columbus, OH. find
  73. Murphy, Gregory. 2002. The Big Book of Concepts. MIT Press, Cambridge, MA. find
  74. Nastase, Vivi and Stan Szpakowicz. 2003. Exploring noun-modifier semantic relations. In Proceedings of the Fifth International Workshop on Computational Semantics, pages 285–301, Tilburg, The Netherlands. find
  75. Ó Séaghdha, Diarmuid and Ann Copestake. 2009. Using lexical and relational similarity to classify semantic relations. In Proceedings of EACL, pages 621–629, Athens. find
  76. Padó, Sebastian and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  77. Padó, Ulrike. 2007. The Integration of Syntax and Semantic Plausibility in a Wide-Coverage Model of Sentence Processing. Ph.D. dissertation, Saarland University, Saarbrücken. find
  78. Padó, Ulrike, Sebastian Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP, pages 400–409, Prague. find
  79. Pantel, Patrick and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of COLING-ACL, pages 113–120, Sydney. find
  80. Peirsman, Yves and Dirk Speelman. 2009. Word space models of lexical variation. In Proceedings of the EACL GEMS Workshop, pages 9–16, Athens. find
  81. Pustejovsky, James. 1995. The Generative Lexicon. MIT Press, Cambridge, MA. find
  82. Quesada, Jose, Praful Mangalath, and Walter Kintsch. 2004. Analogy-making as predication using relational information and LSA vectors. In Proceedings of CogSci, page 1623, Chicago, IL. find
  83. Raghunathan, Trivellore. 2003. An approximate test for homogeneity of correlated correlation coefficients. Quality & Quantity, 37:99–110. find
  84. Rapp, Reinhard. 2003. Word sense discovery based on sense descriptor dissimilarity. In Proceedings of the 9th MT Summit, pages 315–322, New Orleans, LA. find
  85. Rapp, Reinhard. 2004. A freely available automatically generated thesaurus of related words. In Proceedings of LREC, pages 395–398, Lisbon. find
  86. Rogers, Timothy and James McClelland. 2004. Semantic Cognition: A Parallel Distributed Processing Approach. MIT Press, Cambridge, MA. find
  87. Rothenhäusler, Klaus and Hinrich Schütze. 2009. Unsupervised classification with dependency based word spaces. In Proceedings of the EACL GEMS Workshop, pages 17–24, Athens, Greece. find
  88. Rubenstein, Herbert and John Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  89. Ruiz-Casado, Maria, Enrique Alfonseca, and Pablo Castells. 2005. Using context-window overlapping in synonym discovery and ontology extension. In Proceedings of RANLP, pages 1–7, Borovets. find
  90. Sagi, Eyal, Stefan Kaufmann, and Brady Clark. 2009. Semantic density analysis: Comparing word meaning across time and phonetic space. In Proceedings of the EACL GEMS Workshop, pages 104–111, Athens. find
  91. Sahlgren, Magnus. 2005. An introduction to random indexing. http://www.sics.se/∼ mange/papers/RI intro.pdf. ↗
  92. Sahlgren, Magnus. 2006. The Word-Space Model. Ph.D. dissertation, Stockholm University. find
  93. Schulte im Walde, Sabine. 2006. Experiments on the automatic induction of German semantic verb classes. Computational Linguistics, 32:159–194. find our card
  94. Schütze, Hinrich. 1997. Ambiguity Resolution in Natural Language Learning. CSLI Publications, Stanford, CA. find
  95. Smolensky, Paul. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46:159–216. find
  96. Smolensky, Paul and Geraldine Legendre. 2006. The Harmonic Mind. From Neural Computation to Optimality-theoretic Grammar. MIT Press, Cambridge, MA. find
  97. Terra, Egidio and Charles Clarke. 2003. Frequency estimates for statistical word similarity measures. In Proceedings of HLT-NAACL, pages 244–251, Edmonton. find
  98. Turney, Peter. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of ECML, pages 491–502, Freiburg. find
  99. Turney, Peter. 2006a. Expressing implicit semantic relations without supervision. In Proceedings of COLING-ACL, pages 313–320, Sydney. find
  100. Turney, Peter. 2006b. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. find
  101. Turney, Peter. 2007. Empirical evaluation of four tensor decomposition algorithms. Technical Report ERB-1152, NRC, Ottawa. find
  102. Turney, Peter. 2008. A uniform approach to analogies, synonyms, antonyms and associations. In Proceedings of COLING, pages 905–912, Manchester. find
  103. Turney, Peter and Michael Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1-3):251–278. find
  104. Turney, Peter and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  105. Van de Cruys, Tim. 2009. A non-negative tensor factorization model for selectional preference induction. In Proceedings of the EACL GEMS Workshop, pages 83–90, Athens. find
  106. Van Overschelde, James, Katherine Rawson, and John Dunlosky. 2004. Category norms: An updated and expanded version of the Battig and Montague (1969) norms. Journal of Memory and Language, 50:289–335. find
  107. Van Rijsbergen, C. J. 2004. The Geometry of Information Retrieval. Cambridge University Press, Cambridge, UK. find
  108. Veale, Tony and Yanfen Hao. 2008. Acquiring naturalistic concept descriptions from the Web. In Proceedings of LREC, pages 1121–1124, Marrakech. find
  109. Vinson, David and Gabriella Vigliocco. 2008. Semantic feature production norms for a large set of objects and events. Behavior Research Methods, 40(1):183–190. find
  110. Widdows, Dominic. 2004. Geometry and Meaning. CSLI Publications, Stanford, CA. find
  111. Widdows, Dominic. 2008. Semantic vector products: Some initial investigations. In Proceedings of the Second AAAI Symposium on Quantum Interaction, pages 1–8, Oxford. find
  112. Widdows, Dominic and Beate Dorow. 2002. A graph model for unsupervised lexical acquisition. In Proceedings of ICCL, pages 1–7, Taipei. find
  113. Widdows, Dominic and Stanley Peters. 2003. Word vectors and quantum logic. In Proceedings of the Eighth Mathematics of Language Conference, pages 1–14, Bloomington, IN. find
  114. Zarcone, Alessandra and Alessandro Lenci. 2008. Computational models of event type classification in context. In Proceedings of LREC, pages 1232–1238, Marrakech. find
  115. Zhao, Ying and George Karypis. 2003. Criterion functions for document clustering: Experiments and analysis. Technical Report 01-40, University of Minnesota Department of Computer Science, Minneapolis. find
  116. Zhitomirsky-Geffet, Maayan and Ido Dagan. 2009. Bootstrapping distributional feature vector quality. Computational Linguistics, 35(3):435–461. find