Elamur Research⌕
☰
home / papers / 2002 / Improvements in Automatic Thesaurus Extraction

Improvements in Automatic Thesaurus Extraction

Paper · 2002 · James R. Curran and Marc Moens

Summary

Hand-built thesauri are expensive to produce, quickly go out of date, and cover new subdomains poorly. Curran and Moens study how to extract thesauri automatically from distributional evidence. A shallow extractor in the style of Grefenstette turns the British National Corpus into (term, relation, word) context tuples, and terms are compared through their weighted attribute vectors. The authors split similarity into a separate measure function and weight function. They compare existing options (from Lin, Grefenstette, and IR-style set measures) with new weights adapted from collocation statistics, such as the t-test, χ², likelihood ratio and mutual information. Evaluation covers 70 sampled nouns against a gold standard that merges three thesauri. The t-test-style weighting combined with a Jaccard measure performs best among the functions tested. To get around the quadratic cost of pairwise comparison, they add an approximation: a cheap coarse check on a small "canonical" set of attributes comes first, and the full fine-grained comparison runs only when that check passes. This cuts runtime substantially at a small accuracy cost, making thesaurus extraction from large corpora practical.

Key points

  • Corpus and processing:
    • The BNC contains 114M words in 6.2M sentences.
    • Tagging parts of speech and chunking the text required 159 minutes, after which extracting relations needed roughly 7.5 minutes further.
    • The output was 28M relation occurrences over 10M distinct relations, lemmatised with the Sussex morphological analyser.
  • Evaluation set:
    • The 70 nouns were drawn from WordNet to vary in frequency, sense count, hierarchy depth and subtree distribution.
    • Each term received 200 candidate synonyms.
    • The gold standard is the union of synonyms from Macquarie, Roget's and Moby, with the topic-ordered thesauri converted to head-ordered format.
  • Metrics: direct matches (DIRECT), inverse rank (INVR), and precision at 1, 5 and 10.
  • Best configuration: JACCARD with TTEST weighting, a minimum cutoff of 30, and a canonical vector of 100 attributes weighted by TTESTLOG. Averaged over the 70 terms, it scored DIRECT 26, P(1) 76%, P(5) 52%, P(10) 44% and INVR 2.08.
  • Approximation trade-off: naive extraction is O(n²m). With a maximum cutoff of 10,000 terms per attribute and 70 canonical attributes, total DIRECT was 1841. That is 3.9% below full extraction, with execution time reduced by 89%.

Interesting ideas

  • I1. Separate the measure from the weight. Treating vector comparison and attribute weighting as independent, swappable parts turns the design into a grid you can search systematically. This is useful for any distributional-similarity work, including building Ukrainian word-similarity resources.
  • I2. Reuse collocation statistics as feature weights. The authors assume an informative context forms a strong collocation with its term, so they weight attributes with t-test or χ² scores. An optional log-frequency factor favours more frequent attributes. Test statistics like these could weight features in other sparse co-occurrence models.
  • I3. Filter cheaply, then compare in full. A short "canonical" attribute list acts as a gate, and only pairs that share a canonical attribute get the full comparison. The canonical vector size directly controls the speed/accuracy trade-off. The same two-stage pattern fits large-scale nearest-neighbour search in general.
  • I4. Choose canonical features by kind and spread. The strongest-weighted features alone turned out too idiomatic, as with pants, which was too often paired with smarty. Keeping only subject and object relations worked better. Attributes shared by too many terms, such as (direct-obj, get) with 8769 terms, are dropped through a maximum cutoff. Both filters transfer to any blocking or indexing scheme.
  • I5. Combine gold standards and use several metrics. Merging several thesauri reduces coverage gaps. Reporting DIRECT, INVR and P(n) together gives a fuller picture than exact matches alone, and comparing the gold standard against itself (MAX) shows the upper bound.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as James R. Curran and Marc Moens. 2002. Improvements in Automatic Thesaurus Extraction. Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1118627.1118635

@inproceedings{curran2002improvements,
  title = {{Improvements in Automatic Thesaurus Extraction}},
  author = {James R. Curran and Marc Moens},
  booktitle = {Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition},
  year = {2002},
  doi = {10.3115/1118627.1118635},
  url = {https://doi.org/10.3115/1118627.1118635}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Douglas Baker and Andrew McCallum. 1998. Distributional clustering of words for text classification. In Proceedings of the 21st annual international ACM SI-GIR conference on Research and Development in Information Retrieval, pages 96–103, Melbourne, Australia, 24–28 August. find
  2. John R. L. Bernard, editor. 1990. The Macquarie Encyclopedic Thesaurus. The Macquarie Library, Sydney, Australia. find
  3. Peter F. Brown, Vincent J. Della Pietra, Peter V. deSouza, Jennifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479, December. find
  4. Sharon A. Caraballo. 1999. Automatic construction of a hypernym-labeled noun hierarchy from text. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 120–126, College Park, MD USA, 20–26 June.
  5. Stephen Clark and David Weir. 2001. Class-based probability estimation using a semantic hierarchy. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics, pages 95–102, Pittsburgh, PA USA, 2–7 June. find
  6. Carolyn J. Crouch. 1988. Construction of a dynamic thesaurus and its use for associated information retrieval. In Proceedings of the eleventh international conference on Research and Development in Information Retrieval, pages 309–320, Grenoble, France, 13–15 June. find
  7. James R. Curran and Marc Moens. 2002. Scaling context space. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA USA, 7–12 July. (to appear). find
  8. Cristiane Fellbaum, editor. 1998. WordNet: an electronic lexical database. The MIT Press, Cambridge, MA USA. find
  9. Gregory Grefenstette. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Publishers, Boston, USA. find
  10. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th international conference on Computational Linguistics, pages 539–545, Nantes, France, 23–28 July. find
  11. Dekang Lin. 1998a. Automatic retrieval and clustering of similar words. In Proceedings of the 17th International Conference on Computational Linguistics and of the 36th Annual Meeting of the Association for Computational Linguistics, pages 768–774, Montréal, Québec, Canada, 10–14 August.
  12. Dekang Lin. 1998b. An information-theoretic definition of similarity. In Proceedings of the Fifteen International Conference on Machine Learning, pages 296– 304, Madison, WI USA, 24–27 July. find
  13. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA USA. find
  14. Scott McDonald. 2000. Environmental determinants of lexical processing effort. Ph.D. thesis, University of Edinburgh. find
  15. Guido Minnen, John Carroll, and Darren Pearce. 2000. Robust applied morphological generation. In In Proceedings of the First International Natural Language Generation Conference, pages 201–208, 12–16 June.
  16. Darren Pearce. 2001. Synonymy in collocation extraction. In Workshop on WordNet and Other Lexical Resources: Applications, Extensions and Customizations, (NAACL 2001), pages 41–46, Pittsburgh, PA USA, 2–7 June. find
  17. Fernando Pereira, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of English words. In Proceedings of the 31st annual meeting of the Association for Computational Linguistics, pages 183–190, Columbus, Ohio USA, 22–26 June.
  18. Peter Roget. 1911. Thesaurus of English words and phrases. Longmans, Green and Co., London, UK. find
  19. Gerda Ruge. 1997. Automatic detection of thesaurus relations for information retrieval applications. In Foundations of Computer Science: Potential - Theory - Cognition, Lecture Notes in Computer Science, volume LNCS 1337, pages 499–506. Springer Verlag, Berlin, Germany. find
  20. Mark Sanderson and Bruce Croft. 1999. Deriving concept hierarchies from text. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and Development in Information Retrieval, pages 206–213, Berkeley, CA USA, 15–19 August. find
  21. Grady Ward. 1996. Moby Thesaurus. Moby Project. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/curran-2002-improvements-automatic-thesaurus-extraction/