Elamur Research⌕
☰
home / papers / 2004 / Automatically Labeling Semantic Classes

Automatically Labeling Semantic Classes

Paper · 2004 · Patrick Pantel and Deepak Ravichandran

Summary

Clustering systems such as CBC can group words into semantic classes, for example diseases, hockey goalies or colors, but they leave these classes unnamed. Pantel and Ravichandran present a top-down method that assigns names to such classes and then uses those names to produce is-a (hyponym) pairs. Each word is described by grammatical contexts weighted with discounted pointwise mutual information. For every class, a small "committee" of highly similar core members is chosen, and their vectors are averaged into a grammatical signature. The system then searches this signature for simple syntactic patterns, such as apposition, that point to a class name, and outputs a ranked list of candidate labels. Every instance in a labeled class becomes a hyponym of the label. The method named 98.5% of 1,432 CBC noun concepts, with an MRR of 77.1% in human evaluation. It extracted 159,000 hyponyms at 68% precision, rising to 81.5% for proper names. The work matters because it turns unnamed clusters into usable taxonomic knowledge that helps question answering.

Key points

  • The labeling algorithm accepts word clusters from any source. The experiments use CBC output, and the result is a ranked list of names for each class.
  • The pipeline has three phases:
    • word feature vectors built from grammatical contexts, weighted by PMI with a discount that counters PMI's bias toward rare items;
    • committee selection through group-average clustering, scored by cluster size × average pairwise similarity;
    • pattern search over the averaged signature, e.g. the N:appo:N relation, which yields goaltender/goalie for a cluster of NHL goalies.
  • Results on 1,432 noun concepts:
    • 98.5% received a label, with an MRR of 77.1%;
    • 159,000 hyponyms were extracted at 68% precision;
    • 65,000 proper-name hyponyms reached 81.5% precision.
  • On 50 TREC-2003 definition questions, the system's top-5 answers were correct 60% of the time (strict) and 62% (lenient). WordNet scored 38%, and Fleischman et al. scored 42% and 44%.
  • Semantic indexing of the TREC-2002 collection with the extracted hyponyms helped on 179 TREC-2003 questions that have an explicit answer type:
    • correct top-1 passages rose from 36 to 43, roughly a 20% gain;
    • correct top-100 passages rose from 131 to 134.

Interesting ideas

  • I1. Label the class, not each word. Searching a pooled signature for a whole class avoids the sparse and ambiguous evidence of individual words. A rare instance can inherit a hypernym simply by belonging to the class, which could help low-resource languages where many words appear only a few times.
  • I2. Represent a class by its committee. Averaging only the tight core of a cluster gives a cleaner profile than averaging every member, because noisy or polysemous members are left out. The same idea could work as a general denoising step for prototype vectors in other clustering tasks.
  • I3. Score candidate clusters by size × average similarity. This simple metric rewards groups that are both large and coherent. It is a cheap heuristic that could be reused for selecting seed sets elsewhere.
  • I4. Discount PMI by frequency. Multiplying PMI by a frequency-based factor damps the inflated scores of rare words and features. This is useful in any distributional pipeline built on co-occurrence counts.
  • I5. Measure recall indirectly through QA tasks. Recall for knowledge extraction is hard to measure directly. Running definition questions and passage retrieval with and without the resource gives a practical, application-based proxy for coverage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Patrick Pantel and Deepak Ravichandran. 2004. Automatically Labeling Semantic Classes. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pantel2004automatically,
  title = {{Automatically Labeling Semantic Classes}},
  author = {Patrick Pantel and Deepak Ravichandran},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1041/}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barwise, J. and Perry, J. 1985. Semantic innocence and uncompromising situations. In: Martinich, A. P. (ed.) The Philosophy of Language. New York: Oxford University Press. pp. 401–413. find
  2. Berland, M. and E. Charniak, 1999. Finding parts in very large corpora. In ACL-1999. pp. 57–64. College Park, MD.
  3. Church, K. and Hanks, P. 1989. Word association norms, mutual information, and lexicography. In Proceedings of ACL-89. pp. 76–83. Vancouver, Canada.
  4. Fleischman, M.; Hovy, E.; and Echihabi, A. 2003. Offline strategies for online question answering: Answering questions before they are asked. In Proceedings of ACL-03. pp. 1–7. Sapporo, Japan. find
  5. Girju, R.; Badulescu, A.; and Moldovan, D. 2003. Learning semantic constraints for the automatic discovery of part-whole relations. In Proceedings of HLT/NAACL-03. pp. 80–87. Edmonton, Canada.
  6. Han, J. and Kamber, M. 2001. Data Mining – Concepts and Techniques. Morgan Kaufmann. find
  7. Hearst, M. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING-92. pp. 539–545. Nantes, France. find
  8. Hindle, D. 1990. Noun classification from predicate-argument structures. In Proceedings of ACL-90. pp. 268–275. Pittsburgh, PA.
  9. Leacock, C.; Chodorow, M.; and Miller; G. A. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147–165.
  10. Lenat, D. 1995. CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):33–38. find
  11. Lin, D. 1994. Principar - an efficient, broad-coverage, principle-based parser. Proceedings of COLING-94. pp. 42–48. Kyoto, Japan. find
  12. Lin, D. 1998. Automatic retrieval and clustering of similar words. In Proceedings of COLING/ACL-98. pp. 768–774. Montreal, Canada.
  13. Lin, D. and Pantel, P. 2001. Induction of semantic classes from natural language text. In Proceedings of SIGKDD-01. pp. 317–322. San Francisco, CA. find
  14. Mann, G. S. 2002. Fine-Grained Proper Noun Ontologies for Question Answering. SemaNet’ 02: Building and Using Semantic Networks, Taipei, Taiwan. find
  15. Miller, G. 1990. WordNet: An online lexical database. International Journal of Lexicography, 3(4). find
  16. Pasca, M. and Harabagiu, S. 2001. The informative role of WordNet in Open-Domain Question Answering. In Proceedings of NAACL-01 Workshop on WordNet and Other Lexical Resources. pp. 138–143. Pittsburgh, PA. find
  17. Pantel, P. and Lin, D. 2002. Discovering Word Senses from Text. In Proceedings of SIGKDD-02. pp. 613–619. Edmonton, Canada. find
  18. Riloff, E. and Shepherd, J. 1997. A corpus-based approach for building semantic lexicons. In Proceedings of EMNLP-1997.
  19. Riloff, E. and Jones, R. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Proceedings of AAAI-99. pp. 474–479. Orlando, FL. find
  20. Salton, G. and McGill, M. J. 1983. Introduction to Modern Information Retrieval. find
  21. McGraw Hill Schank, R. and Abelson, R. 1977. Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Lawrence Erlbaum Associates. find
  22. Siegel, S. and Castellan Jr., N. J. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill. find
  23. Voorhees, E. 2003. Overview of the question answering track. To appear in Proceedings of TREC-12 Conference. NIST, Gaithersburg, MD. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pantel-2004-automatically-labeling-semantic-classes/