Elamur Research⌕
☰
home / papers / 2005 / Co-occurrence Retrieval: A Flexible Framework for Lexical Distributional Similarity

Co-occurrence Retrieval: A Flexible Framework for Lexical Distributional Similarity

Summary

Many NLP methods rely on words being distributionally similar, but there is no single agreed definition of this similarity, so many competing measures exist. Weeds and Weir propose co-occurrence retrieval (CR), a parameterized framework that treats finding similar words like document retrieval. Each word's co-occurrence features are "retrieved" by another word, and the overlap is scored with precision and recall. Context is defined through grammatical relations that RASP extracted from the BNC.

The authors use the framework to test three questions:

  • whether similarity should be asymmetric,
  • whether some contexts deserve more weight,
  • whether differences in how strongly each word occurs in a context matter.

They evaluate on two tasks, automatic thesaurus generation (WordNet prediction) and pseudo-disambiguation, using 2,000 nouns. Tuned CR settings significantly beat existing measures on both tasks. The two tasks favour opposite balances of precision and recall, so no single fixed measure is likely to win on both. The authors attribute this to an asymmetry built into lexical substitutability. The framework also explains why familiar measures succeed or fail on each task.

Key points

  • Sparse data: in the RASP/BNC data, 14% of noun–verb direct-object tokens and 49% of types in one half were absent from the other half.
  • Existing measures as special cases: several known measures fall out of particular CR settings.
    • Dice and Jaccard correspond exactly to γ = 1 in the additive type-based model.
    • Hindle's Measure corresponds to γ = 0, β = 0 in the additive MI-based model.
    • The L1 Norm corresponds to the difference-weighted token-based model.
    • Lin's Measure closely approximates a harmonic-mean combination of MI-based precision and recall.
    • α-skew divergence resembles high-recall t-test settings.
  • Opposite task preferences: WordNet prediction benefits from high recall, while pseudo-disambiguation needs high precision. This explains Lin's weakness on low-frequency nouns and α-skew's weakness on pseudo-disambiguation.
  • Feature weighting: weighting features with the t-test outperformed MI. Difference-weighting gave no gains for the best models.
  • Rare words: results for low-frequency nouns were not significantly worse than for high-frequency ones. This suggests distributional methods remain usable when data is limited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Information retrieval & search

Institutions

🎓University of SussexBrighton · United Kingdom15 works · 3,064 citations

Cite

Cite as Julie Weeds and David Weir. 2005. Co-occurrence Retrieval: A Flexible Framework for Lexical Distributional Similarity. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/089120105775299122

@article{weeds2005co,
  title = {{Co-occurrence Retrieval: A Flexible Framework for Lexical Distributional Similarity}},
  author = {Julie Weeds and David Weir},
  journal = {Computational Linguistics},
  year = {2005},
  doi = {10.1162/089120105775299122},
  url = {https://doi.org/10.1162/089120105775299122}
}

References

60 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bernard, John R. L., editor. 1990. The Macquarie Encyclopedic Thesaurus. The Macquarie Library, Sydney, Australia. find
  2. Briscoe, Edward and John Carroll. 1995. Developing and evaluating a probabilistic LR parser of part-of-speech and punctuation labels. In Proceedings of the 4th ACL/SIGDAT International Workshop on Parsing Technologies, pages 48–58, Cambridge, MA. find
  3. Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  4. Budanitsky, Alexander. 1999. Lexical Semantic Relatedness and its Application in Natural Language Processing. Ph.D. thesis, University of Toronto, Ontario. find
  5. Budanitsky, Alexander and Graeme Hirst. 2001. Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In Proceedings of the NAACL-01 Workshop on WordNet and Other Lexical Resources, Pittsburgh, PA. find
  6. Caraballo, Sharon. 1999. Automatic construction of a hypernym-labelled noun hierarchy from text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), pages 120–126, College Park, MA. find
  7. Carroll, John and Edward Briscoe. 1996. Apportioning development effort in a probabilistic LR parsing system through evaluation. In Proceedings of the ACL/ SIGDAT Conference on Empirical Methods in Natural Language Processing (EMNLP96), pages 92–100, Santa Cruz, CA. find
  8. Church, Kenneth W., William Gale, Patrick Hanks, Donald Hindle, and Rosamund Moon. 1994. Lexical substitutability. In B. T. S. Atkins and A. Zampolli, editors. Computational Approaches to the Lexicon. Oxford University Press, Oxford, pages 153–177. find
  9. Church, Kenneth W. and Patrick Hanks. 1989. Word association norms, mutual information and lexicography. In Proceedings of the 27th Annual Conference of the Association for Computational Linguistics (ACL-89), pages 76–82, Vancouver, British Columbia. our card
  10. Clark, Stephen and David Weir. 2000. A class-based probabilistic approach to structural disambiguation. In Proceedings of the 18th International Conference on Computational Linguistics (COLING-00), pages 194–200, Saarbrucken, Germany. find
  11. Clark, Stephen and David Weir. 2002. Class-based probability estimation using a semantic hierarchy. Computational Linguistics, 28(2):187–206. find
  12. Cover, T. M. and J. A. Thomas. 1991. Elements of Information Theory. Wiley, New York. find
  13. Curran, James R. and Marc Moens. 2002. Improvements in automatic thesaurus extraction. In Proceedings of the ACL-SIGLEX Workshop on Unsupervised Lexical Acquisition, pages 59–67, Philadelphia, PA. find
  14. Dagan, Ido, Lillian Lee, and Fernando Pereira. 1999. Similarity-based models of word co-occurrence probabilities. Machine Learning Journal, 34(1–3):43–69. find
  15. Dagan, Ido, S. Marcus, and S. Markovitch. 1993. Contextual word similarity and estimation from sparse data. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL-93), pages 56–63, Columbus, OH. find
  16. Even-Zohar, Yair and Dan Roth. 2000. A classification approach to word prediction. find
  17. In Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-00), pages 124–131, Pittsburg, PA. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. find
  18. Firth, John Rupert. 1957. A synopsis of linguistic theory 1930-1955. In Studies in Linguistic Analysis, pages 1–32, Philogical Society, Oxford. Reprinted in Palmer, F. (ed.), 1968 Selected Papers of J. R. Firth, Longman, Harlow. find
  19. Fontenelle, Thierry, Walter Bruls, Luc Thomas, Tom Vanallemeersch, and Jacques Jansen. 1994. DECIDE, MLAP-Project 93-19, deliverable D-1a: a survey of collocation extraction tools. Technical report, University of Liege, Belgium. find
  20. Frakes, W. B. and R. Baeza-Yates, editors. 1992. Information Retrieval, Data Structures and Algorithms. Prentice Hall, New York. find
  21. Fung, Pascale and Kathleen McKeown. 1997. A technical word- and term-translation aid using noisy parallel corpora across language groups. Machine Translation, 12(1–2):53–87. find
  22. Gale, William, Kenneth W. Church, and David Yarowsky. 1992. Work on statistical methods for word sense disambiguation. In Working notes of the AAAI symposium on probabilistic approaches to natural language, pages 54–60, Menlo Park, CA. find
  23. Geffet, Maayan and Ido Dagan. 2004. Feature vector quality and distributional similarity. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), pages 247–253, Geneva, Switzerland. find
  24. Golding, Andrew R. and Dan Roth. 1999. A winnow-based approach to context-sensitive spelling correction. Machine Learning, 34(1–3):182–190. find
  25. Grefenstette, Gregory. 1994. Corpus-derived first-, second- and third-order word affinities. In Proceedings of Euralex, pages 279–290, Amsterdam, Holland. find
  26. Harris, Zelig S. 1968. Mathematical Structures of Language. John Wiley, New York. find
  27. Hindle, Donald. 1990. Noun classification from predicate-argument structures. In Proceedings of the 28th Annual Meeting of the Association for Computational Linguistics (ACL-1990), pages 268–275, Pittsburgh, PA. our card
  28. Hindle, Donald and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120. our card
  29. Jiang, Jay J. and David W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the International Conference on Research in Computational Linguistics (ROCLING X), Taiwan. find
  30. Kaufman, Leonard and Peter J. Rousseeuw. 1990. Finding Groups in Data: An Introduction to Cluster Analysis. John Wiley, New York. find
  31. Kilgarriff, Adam. 2003. Thesauruses for natural language processing. In Proceedings of the Joint Conference on Natural Language Processing and Knowledge Engineering, pages 5–13, Beijing, China. find
  32. Kilgarriff, Adam and Colin Yallop. 2000. What’s in a thesaurus. In Second Conference on Language Resources and Evaluation (LREC-00), pages 1371–1379, Athens. find
  33. Kullback, S. and R.A. Leibler. 1951. On information and sufficiency. Annals of Mathematical Statistics, 22:79–86. find
  34. Lee, Lillian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-1999), pages 23–32, College Park, MA. our card
  35. Lee, Lillian. 2001. On the effectiveness of the skew divergence for statistical language analysis. Artificial Intelligence and Statistics, pages 65–72. find
  36. Li, Hang. 2002. Word clustering and disambiguation based on co-occurrence data. Natural Language Engineering, 8(1):25–42. find
  37. Li, Hang and Naoki Abe. 1998. Generalizing case frames using a thesaurus and the MDL principle. Computational Linguistics, 24(2):217–244. our card
  38. Lin, Dekang. 1997. Using syntactic dependency as local context to resolve word sense ambiguity. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics (ACL-97), pages 64–71… find
  39. Lin, Dekang. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING-ACL ’98), pages 768–774, Montreal, Quebec. our card
  40. Lin, Dekang. 1998b. An information-theoretic definition of similarity. In Proceedings of International Conference on Machine Learning, Madison, WI. find
  41. Lin, Dekang, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI-03), pages 1492–1493. find
  42. Manning, Christopher D. and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA. find
  43. McCarthy, Diana, Rob Koeling, and Julie Weeds. 2004. Ranking WordNet senses. Technical Report 569, Department of Informatics, University of Sussex, Brighton. find
  44. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 280–287, Barcelona, Spain. our card
  45. Miller, G., M. Chodorow, S. Landes, C. Leacock, and R. Thomas. 1994. Using a semantic concordance for sense identification. In Proceedings of the ARPA Human Language Technology Workshop, Plainsboro, NJ. our card
  46. Pantel, Patrick and Dekang Lin. 2000. Word-for-word glossing of contextually similar words. In Proceedings of the Conference on Applied Natural Language Processing / 1st Meeting of the North American Chapter of the Association for Computational Linguistics (ANLP-NAACL-00), pages 78–85, Seattle, WA. find
  47. Patwardhan, Siddharth, Satanjeev Banerjee, and Ted Pedersen. 2003. Using measures of semantic relatedness for word sense disambiguation. In Proceedings of the 4th International Conference on Intelligent Text Processing and Computational Linguistics, pages 241–257, Mexico City. find
  48. Pereira, Fernando, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of similar words. In Proceedings of the 30th Annual Meeting of the Association for Computational Linguistics (ACL-93), pages 183–190, Columbus, OH. find
  49. Pustejovsky, James. 1995. The generative lexicon. MIT Press, Cambridge, MA. find
  50. Resnik, Philip. 1993. Selection and Information: A Class-Based Approach to Lexical Relationships. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA. find
  51. Roget, Peter. 1911. Thesaurus of English Words and Phrases. Longmans, Green and Co., London. find
  52. Rooth, Mats, Stefan Riezler, Detlef Prescher, Glenn Carroll, and Franz Beil. 1999. Inducing a semantically annotated lexicon via EM-based clustering. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), pages 104–111, College Park, MA. find
  53. Salton, Gerald and M. J. McGill. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York. find
  54. Schütze, Hinrich. 1992. Dimensions of meaning. In Proceedings of Conference on Supercomputing, pages 787–796, Minneapolis, MN. find
  55. Schütze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–124. van Rijsbergen, C. J. 1979. Information Retrieval, second edition. Butterworths, London. find
  56. Weeds, Julie. 2003. Measures and Applications of Lexical Distributional Similarity. Ph.D. thesis, University of Sussex, Brighton. find
  57. Weeds, Julie and David Weir. 2003a. Finding and evaluating sets of nearest neighbours. In Proceedings of the 2nd International Conference on Corpus Linguistics, pages 879–888, Lancaster, UK. find
  58. Weeds, Julie and David Weir. 2003b. A general framework for distributional similarity. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-03), pages 81–88, Sapporo, Japan. our card
  59. Weeds, Julie, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), pages 1015–1021, Geneva, Switzerland. find
  60. Xu, Jinxi and Bruce Croft. 1996. Query expansion using local and global disambiguation. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR-96), pages 4–11, Zurich, Switzerland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weeds-2005-co-occurrence-retrieval-flexible-framework/