Elamur Research⌕
☰
home / papers / 2009 / A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

Summary

The paper compares two families of methods for scoring how alike two words are: methods built on WordNet and methods learned from corpora. It also examines how they differ on similarity data versus relatedness data. The WordNet side runs personalized PageRank over a graph of synsets and words, once for each word. The two resulting probability distributions are then compared with cosine. The distributional side builds vector space models from a crawled Web corpus of about 1.6 Terawords, using bag-of-words, context windows, or syntactic dependency templates, with χ² weighting and cosine. Each approach is the strongest in its own class on RG and WordSim353. An SVM that combines them reaches the best published scores on both datasets. The authors also run what they describe as the first cross-lingual (English–Spanish) evaluation of such measures, with only small losses. This matters because it shows Web-scale distributional statistics can match curated lexical resources, and because the two families turn out to be complementary.

Key points

  • WordNet graphs used: MCR16 (Multilingual Central Repository with WordNet 1.6), WN30, and WN30g (WordNet 3.0 plus disambiguated gloss relations). PageRank used the default damping of 0.85 and 30 iterations, with no tuning.
  • On WordSim353, WN30g scores 0.66 Spearman, compared with 0.53 for MCR16 and 0.56 for WN30. The authors credit the disambiguated glosses for most of this gain, mainly because they model relatedness better.
  • The Web corpus started from about four billion English documents crawled in August 2008. Computing all test-pair similarities with MapReduce takes about 15 minutes on 2,000 cores.
  • Context windows suit similarity best: a window of 4 gives 0.89 on RG. Bag-of-words suits relatedness best.
  • A supervised SVM combination, run in Weka with 10-fold cross-validation, reaches 0.78 Spearman on WordSim353. This is above Gabrilovich and Markovitch's Wikipedia-based 0.75, and the improvement on RG is statistically significant.

Interesting ideas

  • I1. Cross-lingual similarity through a shared concept graph: Wordnets aligned via an InterLingual Index let words from two languages be mapped onto one set of concepts. A Ukrainian–English word pair could therefore be compared on a common graph, provided an aligned Ukrainian wordnet exists.
  • I2. Personalized PageRank as a word signature: Starting all random-jump mass at the target word yields a distribution over concepts. This captures paths through the graph, not just shortest distances, and the idea carries over to any knowledge graph.
  • I3. Distributional fallback for missing words: Corpus-based similarity was used to cover items absent from WordNet, which gave the best unsupervised results. A curated resource can be extended cheaply in the same way.
  • I4. Report confidence intervals and use Spearman: The authors report 95% intervals and show that many published RG/WordSim353 differences cannot be told apart. They prefer Spearman because it needs no dataset-specific output mapping. Both habits would improve evaluation of any small benchmark.
  • I5. Separate similarity from relatedness: Different context types serve each notion better, so the window or feature design should match the target task, for example synonym finding versus topical association.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

257 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 240 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{agirre2009study,
  title = {{A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches}},
  author = {Agirre, Eneko and Alfonseca, Enrique and Hall, Keith and Kravalova, Jana and Paşca, Marius and Soroa, Aitor},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  doi = {10.3115/1620754.1620758},
  url = {https://doi.org/10.3115/1620754.1620758}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Agirre and A. Soroa. 2009. Personalizing pagerank for word sense disambiguation. In Proc. of EACL 2009, Athens, Greece. find
  2. M.A. Alvarez and S.J. Lim. 2007. A Graph Modeling of Semantic Similarity between Words. Proc. of the Conference on Semantic Computing, pages 355–362. find
  3. J. Atserias, L. Villarejo, G. Rigau, E. Agirre, J. Carroll, B. Magnini, and P. Vossen. 2004. The meaning multilingual central repository. In Proc. of Global WordNet Conference, Brno, Czech Republic. find
  4. D. Bollegala, Matsuo Y., and M. Ishizuka. 2007. Measuring semantic similarity between words using web search engines. In Proceedings of WWW’2007. find
  5. A. Budanitsky and G. Hirst. 2006. Evaluating WordNet-based Measures of Lexical Semantic Relatedness. Computational Linguistics, 32(1):13–47. find our card
  6. H. Chen, M. Lin, and Y. Wei. 2006. Novel association measures using web search with double checking. In Proceedings of COCLING/ACL 2006. find
  7. J. Daude, L. Padro, and G. Rigau. 2000. Mapping Word-Nets using structural information. In Proceedings of ACL’2000, Hong Kong. find
  8. C. Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database and Some of its Applications. MIT Press, Cambridge, Mass. find
  9. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. 2002. Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  10. E. Gabrilovich and S. Markovitch. 2007. Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis. Proc of IJCAI, pages 6–12. find
  11. T. H. Haveliwala. 2002. Topic-sensitive pagerank. In WWW ’02: Proceedings of the 11th international conference on World Wide Web, pages 517–526. find
  12. T. Hughes and D. Ramage. 2007. Lexical semantic relatedness with random graph walks. In Proceedings of EMNLP-CoNLL-2007, pages 581–589. find
  13. M. Jarmasz. 2003. Roget’s Thesuarus as a lexical resource for Natural Language Processing. find
  14. J.J. Jiang and D.W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics, volume 33. Taiwan. find
  15. C. Leacock and M. Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. WordNet: An Electronic Lexical Database, 49(2):265–283. find
  16. D. Lin. 1998a. An information-theoretic definition of similarity. In Proc. of ICML, pages 296–304, Wisconsin, USA. find
  17. D. Lin. 1998b. Automatic Retrieval and Clustering of Similar Words. In Proceedings of ACL-98. find our card
  18. G.A. Miller and W.G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  19. J. Nivre. 2006. Inductive Dependency Parsing, volume 34 of Text, Speech and Language Technology. Springer. find
  20. S. Pado and M. Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  21. S. Patwardhan and T. Pedersen. 2006. Using WordNet-based Context Vectors to Estimate the Semantic Relatedness of Concepts. In Proceedings of the EACL Workshop on Making Sense of Sense: Bringing Computational Linguistics and Pycholinguistics Together, pages 1–8, Trento, Italy. find
  22. P. Resnik. 1995. Using Information Content to Evaluate Semantic Similarity in a Taxonomy. Proc. of IJCAI, 14:448–453. find our card
  23. H. Rubenstein and J.B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  24. M Ruiz-Casado, E. Alfonseca, and P. Castells. 2005. Using context-window overlapping in Synonym Discovery and Ontology Extension. In Proceedings of RANLP-2005, Borovets, Bulgaria,. find
  25. M. Sahami and T.D. Heilman. 2006. A web-based kernel function for measuring the similarity of short text snippets. Proc. of WWW, pages 377–386. find
  26. M. Strube and S.P. Ponzetto. 2006. WikiRelate! Computing Semantic Relatedness Using Wikipedia. In Proceedings of the AAAI-2006, pages 1419–1424. find
  27. P.D. Turney. 2001. Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL. Lecture Notes in Computer Science, 2167:491–502. find
  28. P. Vossen, editor. 1998. EuroWordNet: A Multilingual Database with Lexical Semantic Networks. Kluwer Academic Publishers. find
  29. Z. Wu and M. Palmer. 1994. Verb semantics and lexical selection. In Proc. of ACL, pages 133–138, Las Cruces, New Mexico. find our card
  30. D. Yang and D.M.W. Powers. 2005. Measuring semantic similarity in the taxonomy of WordNet. Proceedings of the Australasian conference on Computer Science. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/