Elamur Research⌕
☰
home / papers / 2006 / Similarity of Semantic Relations

Similarity of Semantic Relations

Summary

Turney tackles automatic measurement of relational similarity: how closely the relation linking one word pair matches the relation in another (e.g., mason:stone vs. carpenter:wood). This differs from attributional similarity between single words, which underlies synonymy. He proposes Latent Relational Analysis (LRA), an unsupervised, corpus-driven method that builds on the Vector Space Model (VSM) approach of Turney and Littman (2005). That earlier approach represented a word pair by counts of a fixed set of hand-written connecting patterns. LRA makes three changes. It mines connecting patterns from the corpus automatically. It smooths the pair-by-pattern frequency matrix with Singular Value Decomposition (SVD). It also expands each pair with thesaurus-derived synonym variants. Using the WMTS corpus (roughly 5×10¹⁰ English words), LRA was tested on SAT multiple-choice analogy questions and on noun-modifier relation classification. On both tasks it clearly outperformed VSM. On the analogies, it reached the level of an average college-bound human test taker. This suggests relational similarity can be measured well enough to support work on information extraction, word sense disambiguation, retrieval and metaphor understanding.

Key points

  • On 374 SAT analogy questions, LRA scores about 56% accuracy, compared with 47% for VSM. Average human performance is about 57%, which is statistically equivalent to LRA. With the WMTS corpus, LRA's F is 56.5% versus 40.3% for VSM.
  • The baseline VSM relied on 64 hand-built patterns. LRA instead generates patterns from the corpus and works on large matrices, e.g., 17,232 rows × 8,000 columns.
  • On noun-modifier relation classification, LRA was used as the distance measure in 1-nearest-neighbour classification of 600 pairs from Nastase and Szpakowicz (2003). It scored 39.8% on the 30-class scheme (VSM: 27.8%) and 58.0% on the 5-class scheme (VSM: 45.7%).
  • The author also tried NMF, PLSA, KPCA and Iterative Scaling as alternatives to SVD. None was clearly better, and none scaled to the full matrix size.
  • Stated limitations include slow runtime, sensitivity to corpus size, and scarce labelled data (about 20 examples per class). Proposed future work includes hybrid systems that combine LRA with lexicon-based approaches such as Veale (2004).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🏛️National Research Council CanadaOttawa · Canada36 works · 16 873 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2006.32.3.379

@article{turney2006similarity,
  title = {{Similarity of Semantic Relations}},
  author = {Turney, Peter D.},
  journal = {Computational Linguistics},
  year = {2006},
  doi = {10.1162/coli.2006.32.3.379},
  url = {https://doi.org/10.1162/coli.2006.32.3.379}
}

References

77 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agresti, Alan. 1990. Categorical Data Analysis. Wiley, Hoboken, NJ. find
  2. Ando, Rie Kubota. 2000. Latent semantic space: Iterative scaling improves precision of inter-document similarity measurement. In Proceedings of the 23rd Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR-2000), pages 216–223, Athens, Greece. find
  3. Baeza-Yates, Ricardo A. and Berthier A. Ribeiro-Neto. 1999. Modern Information Retrieval. ACM Press, New York, NY. find
  4. Banerjee, Satanjeev and Ted Pedersen. 2003. Extended gloss overlaps as a measure of semantic relatedness. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI-03), pages 805–810, Acapulco, Mexico. find our card
  5. Barker, Ken and Stan Szpakowicz. 1998. Semi-automatic recognition of noun modifier relationships. In Christian Boitet and Pete Whitelock, editors, Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics and Seventeenth International Conference on Computational… find
  6. Berland, Matthew and Eugene Charniak. 1999. Finding parts in very large corpora. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL ’99), pages 57–64, New Brunswick, NJ. find our card
  7. Berry, Michael W. 1992. Large scale singular value computations. International Journal of Supercomputer Applications, 6(1):13–49. find
  8. Budanitsky, Alexander and Graeme Hirst. 2001. Semantic distance in wordnet: An experimental, application-oriented evaluation of five measures. In Proceedings of the Workshop on WordNet and Other Lexical Resources, Second Meeting of the North American Chapter of the Association for Computational Ling… find
  9. Chiarello, Christine, Curt Burgess, Lorie Richards, and Alma Pollock. 1990. Semantic and associative priming in the cerebral hemispheres: Some words do, some words don’t ... sometimes, some places. Brain and Language, 38:75–104. find
  10. Claman, Cathy. 2000. 10 Real SATs. College Entrance Examination Board. find
  11. Clarke, Charles L. A., Gordon V. Cormack, and Christopher R. Palmer. 1998. An overview of multitext. ACM SIGIR Forum, 32(2):14–15. find
  12. Daganzo, Carlos F. 1994. The cell transmission model: A dynamic representation of highway traffic consistent with the hydrodynamic theory. Transportation Research Part B: Methodological, 28(4):269–287. find
  13. Deerwester, Scott C., Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science (JASIS), 41(6):391–407. find
  14. Dolan, William B. 1995. Metaphor as an emergent property of machine-readable dictionaries. In Proceedings of the AAAI 1995 Spring Symposium Series: Representation and Acquisition of Lexical Knowledge: Polysemy, Ambiguity and Generativity, pages 27–32, Menlo Park, CA. find
  15. Dumais, Susan T. 1990. Enhancing performance in latent semantic indexing (LSI) retrieval. Technical Report TM-ARH-017527, Bellcore, Morristown, NJ. find
  16. Dumais, Susan T. 1993. Latent semantic indexing (LSI) and TREC-2. In D. K. Harman, editor, Proceedings of the Second Text REtrieval Conference (TREC-2), pages 105–115. National Institute of Standards and Technology, Gaithersburg, MD. find
  17. Falkenhainer, Brian. 1990. Analogical interpretation in context. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, pages 69–76. Lawrence Erlbaum Associates, Mahwah, NJ. find
  18. Falkenhainer, Brian, Kenneth D. Forbus, and Dedre Gentner. 1989. The structure-mapping engine: Algorithm and examples. Artificial Intelligence, 41(1):1–63. find
  19. Federici, Stefano, Simonetta Montemagni, and Vito Pirrelli. 1997. Inferring semantic similarity from distributional evidence: An analogy-based approach to word sense disambiguation. In Proceedings of the ACL/EACL Workshop on Automatic Information Extraction and Building of Lexical Semantic Resources… find
  20. Feelders, Ad and William Verkooijen. 1995. Which method learns the most from data? Methodological issues in the analysis of comparative studies. In Fifth International Workshop on Artificial Intelligence and Statistics, pages 219–225, Ft. Lauderdale, FL. find
  21. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
  22. French, Robert M. 2002. The computational modeling of analogy-making. Trends in Cognitive Sciences, 6(5):200–205. find
  23. Gentner, Dedre. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155–170. find
  24. Gentner, Dedre, Brian Bowdle, Phillip Wolff, and Consuelo Boronat. 2001. Metaphor is like analogy. In Dedre Gentner, Keith J. Holyoak, and Boicho N. Kokinov, editors, The Analogical Mind: Perspectives from Cognitive Science. MIT Press, Cambridge, MA, pages 199–253. find
  25. Gildea, Daniel and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  26. Girju, Roxana, Adriana Badulescu, and Dan I. Moldovan. 2003. Learning semantic constraints for the automatic discovery of part-whole relations. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2003),… find
  27. Goldenberg, David. 2005. The emperor’s new clothes: Undressing the new and unimproved SAT. Gelf Magazine, March. http://www.gelf-magazine.com/mt/archives/the emperors new clothes.html. ↗
  28. Golub, Gene H. and Charles F. Van Loan. 1996. Matrix Computations. 3rd ed. Johns Hopkins University Press, Baltimore, MD. find
  29. Harman, Donna. 1986. An experimental study of factors important in document ranking. In Proceedings of the Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’86), pages 186–193, Pisa, Italy. find
  30. Hearst, Marti A. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the Fourteenth International Conference on Computational Linguistics, pages 539–545, Nantes, France. find
  31. Hirst, Graeme and David St-Onge. 1998. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database, pages 305–332. MIT Press, Cambridge, MA. find
  32. Hofmann, Thomas. 1999. Probabilistic latent semantic indexing. In Proceedings of the 22nd Annual ACM Conference on Research and Development in Information Retrieval (SIGIR ’99), pages 50–57, Berkeley, CA, August. find
  33. Hofstadter, Douglas and the Fluid Analogies Research Group. 1995. Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. Basic Books, New York. find
  34. Jarmasz, Mario and Stan Szpakowicz. 2003. Roget’s thesaurus and semantic similarity. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), pages 212–219, Borovets, Bulgaria. find
  35. Jiang, Jay J. and David W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the International Conference on Research in Computational Linguistics (ROCLING X), pages 19–33, Tapei, Taiwan. find
  36. Kurtz, Stanley. 2002. Testing debate. National Review Magazine, August. http://www.nationalreview.com/kurtz/kurtz082102.asp. ↗
  37. Lakoff, George and Mark Johnson. 1980. Metaphors We Live By. University of Chicago Press, Chicago, IL. find
  38. Landauer, Thomas K. and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  39. Lapata, Mirella and Frank Keller. 2004. The web as a baseline: Evaluating the performance of unsupervised web-based models for a range of NLP tasks. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 20… find
  40. Lauer, Mark. 1995. Designing Statistical Language Learners: Experiments on Compound Nouns. Ph.D. thesis, Macquarie University, Sydney. find
  41. Leacock, Claudia and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database, pages 265–283. MIT Press, Cambridge, MA. find
  42. Lee, Daniel D. and H. Sebastian Seung. 1999. Learning the parts of objects by nonnegative matrix factorization. Nature, 401:788–791. find
  43. Lesk, Michael E. 1969. Word-word associations in document retrieval systems. American Documentation, 20(1):27–38. find
  44. Lesk, Michael E. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of ACM SIGDOC ’86, pages 24–26, New York, NY. find
  45. Lewis, David D. 1991. Evaluating text categorization. In Proceedings of the Speech and Natural Language Workshop, pages 312–318, Asilomar, CA. Morgan Kaufmann, San Francisco, CA. find
  46. Lin, Dekang. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING-ACL ’98), pages 768–774, Montreal, Canada. find our card
  47. Lin, Dekang. 1998b. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning (ICML ’98), pages 296–304. Morgan Kaufmann, San Francisco, CA. find
  48. Marx, Zvika, Ido Dagan, Joachim Buhmann, and Eli Shamir. 2002. Coupled clustering: A method for detecting structural correspondence. Journal of Machine Learning Research, 3:747–780. find
  49. Medin, Douglas L., Robert L. Goldstone, and Dedre Gentner. 1990. Similarity involving attributes and relations: Judgments of similarity and difference are not inverses. Psychological Science, 1(1):64–69. find
  50. Moldovan, Dan, Adriana Badulescu, Marta Tatu, Daniel Antohe, and Roxana Girju. 2004. Models for the semantic classification of noun phrases. In Proceedings of the Computational Lexical Semantics Workshop at HLT-NAACL 2004, pages 60–67, Boston, MA. find
  51. Morris, Jane and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics, 17(1):21–48. find our card
  52. Nastase, Vivi and Stan Szpakowicz. 2003. Exploring noun–modifier semantic relations. In Fifth International Workshop on Computational Semantics (IWCS-5), pages 285–301, Tilburg, the Netherlands. find
  53. Pantel, Patrick and Dekang Lin. 2002. Discovering word senses from text. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 613–619, New York, NY. find
  54. Rada, Roy, Hafedh Mili, Ellen Bicknell, and Maria Blettner. 1989. Development and application of a metric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics, 19(1):17–30. find
  55. Rehder, Bob, M. E. Schreiner, Michael B. W. Wolfe, Darrell Laham, Thomas K. Landauer, and Walter Kintsch. 1998. Using latent semantic analysis to assess knowledge: Some technical considerations. Discourse Processes, 25:337–354. find
  56. Reitman, Walter R. 1965. Cognition and Thought: An Information Processing Approach. John Wiley and Sons, New York, NY. find
  57. Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95), pages 448–453, San Mateo, CA. Morgan Kaufmann, San Francisco, CA. find our card
  58. Riloff, Ellen and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), pages 474–479, Menlo Park, CA. find
  59. Rosario, Barbara and Marti Hearst. 2001. Classifying the semantic relations in noun-compounds via a domain-specific lexical hierarchy. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP-01), pages 82–90, Pittsburgh, PA. find our card
  60. Rosario, Barbara, Marti Hearst, and Charles Fillmore. 2002. The descent of hierarchy, and selection in relational semantics. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL ’02), pages 417–424, Philadelphia, PA. find
  61. Ruge, Gerda. 1992. Experiments on linguistically-based term associations. Information Processing and Management, 28(3):317–332. find
  62. Salton, Gerard. 1989. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley, Reading, MA. find
  63. Salton, Gerard and Chris Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5):513–523. find
  64. Salton, Gerard and Michael J. McGill. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York, NY. find
  65. Scholkopf, Bernhard, Alexander J. Smola, and Klaus-Robert Muller. 1997. Kernel principal component analysis. In Proceedings of the International Conference on Artificial Neural Networks (ICANN-1997), pages 583–588, Berlin. find
  66. Terra, Egidio and Charles L. A. Clarke. 2003. Frequency estimates for statistical word similarity measures. In Proceedings of the Human Language Technology and North American Chapter of Association of Computational Linguistics Conference 2003 (HLT/NAACL 2003), pages 244–251, Edmonton, Canada. find
  67. Turney, Peter D. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of the Twelfth European Conference on Machine Learning, pages 491–502, Springer, Berlin. find
  68. Turney, Peter D. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02), pages 417–424, Philadelphia, PA. find our card
  69. Turney, Peter D. 2005. Measuring semantic similarity by latent relational analysis. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI-05), pages 1136–1141, Edinburgh, Scotland. find
  70. Turney, Peter D. and Michael L. Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1–3):251–278. find
  71. Turney, Peter D., Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), pages 482–489, Borovets,… find
  72. Vanderwende, Lucy. 1994. Algorithm for automatic interpretation of noun sequences. In Proceedings of the Fifteenth International Conference on Computational Linguistics, pages 782–788, Kyoto, Japan. find
  73. Veale, Tony. 2003. The analogical thesaurus. In Proceedings of the 15th Innovative Applications of Artificial Intelligence Conference (IAAI 2003), pages 137–142, Acapulco, Mexico. find
  74. Veale, Tony. 2004. WordNet sits the SAT: A knowledge-based approach to lexical analogy. In Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004), pages 606–612, Valencia, Spain. find
  75. Yangarber, Roman. 2003. Counter-training in discovery of semantic patterns. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), pages 343–350, Sapporo, Japan. find
  76. Yarowsky, David. 1993. One sense per collocation. In Proceedings of the ARPA Human Language Technology Workshop, pages 266–271, Princeton, NJ. find our card
  77. Zelenko, Dmitry, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of Machine Learning Research, 3:1083–1106. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/turney-2006-similarity-semantic-relations/