Elamur Research⌕
☰
home / papers / 2006 / Espresso: Leveraging Generic Patterns for Automatically Harvesting Semantic Relations

Espresso: Leveraging Generic Patterns for Automatically Harvesting Semantic Relations

Summary

Espresso extracts binary semantic relations from raw text with very little supervision: it needs only a handful of seed pairs. It follows Hearst-style bootstrapping and repeats three steps. First, it learns surface patterns from the seeds, using a modified Ravichandran and Hovy method. Second, it ranks the patterns and keeps the strongest. Third, it collects new instances with them.

The novelty is how it treats generic patterns such as "X of Y". These cover many pairs but are very noisy. Earlier systems either dropped them or filtered them with expensive manual annotation. Espresso instead checks candidates on the Web: a pair that truly holds the relation should also co-occur with precise, low-coverage patterns. A PMI-based reliability score for patterns and instances drives this filtering.

The authors tested it on a small domain-specific textbook corpus (CHEM) and a larger news corpus (TREC). They extracted is-a, part-of, succession, reaction and production relations. Using generic patterns (ESP+) raised recall by one to two orders of magnitude, while average precision fell by less than 10%. This makes broad-coverage relation harvesting practical even for small corpora and less common relation types.

Key points

  • The authors compare against RH02 (Ravichandran and Hovy) and PR04 (Pantel et al.).
    • RH02 returns many more relations, but its precision is much lower because it applies generic patterns without any filtering.
    • PR04 reaches only 0.23 relative recall against ESP-.
    • GI03 (Girju et al.) was not reproduced. Its authors reported 83% precision and 72% recall on a TREC-9 sample, but the method relies on heavy supervision.
  • Bootstrapping stops after τ1 = 5 patterns, or when the average pattern score falls by more than τ2 = 50%. The top m = 200 instances seed the next round, and the pattern set may grow by one pattern per iteration.
  • Generic patterns were found for every relation except succession (between one and five per relation). On one part-of setting (Table 4), recall rose 577× while precision dropped about 10%.
  • Candidate terms can span several words, following a modified Justeson term definition. For small corpora, two expansion steps add instances:
    • Web expansion queries Google with the learned patterns.
    • Syntactic expansion shortens terms to their heads.
  • The authors built manual gold standards for 100 sampled instances per generic pattern (22 patterns in total). They estimated confidence intervals with bootstrap resampling, at roughly ±10–15%.

Interesting ideas

  • I1. Use precise patterns to check noisy ones. High-recall, low-precision patterns can be kept if each of their outputs is confirmed by co-occurrence with precise patterns in a huge corpus. This transfers to any weakly supervised extractor, for example harvesting Ukrainian relations from web-scale text.
  • I2. Rank patterns by reliability rather than raw frequency. A PMI-based reliability score lets rare but accurate patterns survive. Simple frequency cutoffs would throw such patterns away.
  • I3. Abstract surrounding terms to fight sparsity. Other terminological expressions in a sentence are replaced with a generic TR label before patterns are learned. This makes patterns reusable on small corpora, and the later filtering step absorbs the extra noise.
  • I4. Expand small corpora on demand. When too few instances are found, the system adds Web-query expansion and head-based term shortening, and the same filter removes the noise these introduce. This is useful for low-resource domains where the local corpus lacks redundancy.
  • I5. Analyse each pattern with top-K curves. Plotting precision, recall and F-score over the score-ranked outputs of each generic pattern showed that a single global threshold was far from optimal. The same diagnostic would help tune per-pattern or per-class thresholds in other pipelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

151 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 148 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1220175.1220190

@inproceedings{pantel2006espresso,
  title = {{Espresso: Leveraging Generic Patterns for Automatically Harvesting Semantic Relations}},
  author = {Pantel, Patrick and Pennacchiotti, Marco},
  booktitle = {Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
  doi = {10.3115/1220175.1220190},
  url = {https://doi.org/10.3115/1220175.1220190}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berland, M. and E. Charniak, 1999. Finding parts in very large corpora. In Proceedings of ACL-1999. pp. 57-64. College Park, MD. find our card
  2. Brown, T.L.; LeMay, H.E.; Bursten, B.E.; and Burdge, J.R. 2003. Chemistry: The Central Science, Ninth Edition. Prentice Hall. find
  3. Caraballo, S. 1999. Automatic acquisition of a hypernym-labeled noun hierarchy from text. In Proceedings of ACL-99. pp 120-126, Baltimore, MD. find
  4. Cover, T.M. and Thomas, J.A. 1991. Elements of Information Theory. John Wiley & Sons. find
  5. Day, D.; Aberdeen, J.; Hirschman, L.; Kozierok, R.; Robinson, P.; and Vilain, M. 1997. Mixed-initiative development of language processing systems. In Proceedings of ANLP-97. Washington D.C. find
  6. Downey, D.; Etzioni, O.; and Soderland, S. 2005. A Probabilistic model of redundancy in information extraction. In Proceedings of IJCAI-05. pp. 1034-1041. Edinburgh, Scotland. find
  7. Etzioni, O.; Cafarella, M.J.; Downey, D.; Popescu, A.-M.; Shaked, T.; Soderland, S.; Weld, D.S.; and Yates, A. 2005. Unsupervised named-entity extraction from the Web: An experimental study. Artificial Intelligence, 165(1): 91-134. find
  8. Fellbaum, C. 1998. WordNet: An Electronic Lexical Database. MIT Press. find
  9. Geffet, M. and Dagan, I. 2005. The Distributional Inclusion Hypotheses and Lexical Entailment. In Proceedings of ACL-2005. Ann Arbor, MI. find our card
  10. Girju, R.; Badulescu, A.; and Moldovan, D. 2006. Automatic Discovery of Part-Whole Relations. Computational Linguistics, 32(1): 83-135. find
  11. Hearst, M. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of COLING-92. pp. 539-545. Nantes, France. find
  12. Hindle, D. 1990. Noun classification from predicate-argument structures. In Proceedings of ACL-90. pp. 268– 275. Pittsburgh, PA. find our card
  13. Justeson J.S. and Katz S.M. 1995. Technical Terminology: some linguistic properties and algorithms for identification in text. In Proceedings of ICCL-95. pp.539-545. Nantes, France. find
  14. Lin, C.-Y. and Hovy, E.H.. 2000. The Automated acquisition of topic signatures for text summarization. In Proceedings of COLING-00. pp. 495-501. Saarbrücken, Germany. find our card
  15. Lin, D. and Pantel, P. 2002. Concept discovery from text. In Proceedings of COLING-02. pp. 577-583. Taipei, Taiwan. find
  16. Mann, G. S. 2002. Fine-Grained Proper Noun Ontologies for Question Answering. In Proceedings of SemaNet’ 02: Building and Using Semantic Networks, Taipei, Taiwan. find
  17. Pantel, P. and Ravichandran, D. 2004. Automatically labeling semantic classes. In Proceedings of HLT/NAACL-04. pp. 321-328. Boston, MA. find
  18. Pantel, P.; Ravichandran, D.; Hovy, E.H. 2004. Towards terascale knowledge acquisition. In Proceedings of COLING-04. pp. 771-777. Geneva, Switzerland. find
  19. Pasca, M. and Harabagiu, S. 2001. The informative role of WordNet in Open-Domain Question Answering. In Proceedings of NAACL-01 Workshop on WordNet and Other Lexical Resources. pp. 138-143. Pittsburgh, PA. find
  20. Ravichandran, D. and Hovy, E.H. 2002. Learning surface text patterns for a question answering system. In Proceedings of ACL-2002. pp. 41-47. Philadelphia, PA. find our card
  21. Riloff, E. and Shepherd, J. 1997. A corpus-based approach for building semantic lexicons. In Proceedings of EMNLP-97. find
  22. Siegel, S. and Castellan Jr., N. J. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill. find
  23. Szpektor, I.; Tanev, H.; Dagan, I.; and Coppola, B. 2004. Scaling web-based acquisition of entailment relations. In Proceedings of EMNLP-04. Barcelona, Spain. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pantel-2006-espresso-leveraging-generic-patterns-automatically/