Elamur Research⌕
☰
home / papers / 2009 / Distant supervision for relation extraction without labeled data

Distant supervision for relation extraction without labeled data

Paper · 2009 · Mike Mintz, Steven Bills, Rion Snow et al.

Summary

The paper addresses the automatic extraction of semantic relations between entities from text. Classical methods require a manually annotated corpus (e.g., ACE), so data are scarce and performance degrades outside the subject domain. The authors propose a distant supervision scheme in which the Freebase knowledge base serves as the teacher. If two entities are linked by some relation in Freebase, then every sentence in which both occur is treated as a likely example of that relation. Entities in 1.2 million Wikipedia articles are identified with an NER tagger. Features from all sentences in which the same pair occurs are combined into a single vector, and a multiclass logistic regression is then trained on lexical and syntactic (dependency) features. The system extracted 10,000 new facts for 102 relations with a precision of 67.6%. The approach requires no manual annotation, yields relations with the canonical names of the knowledge base, and scales to corpora of any size. It subsequently became the baseline for an entire line of research in relation extraction.

Key points

  • Data: the July 2008 Freebase dump contained 116 million facts, 7,300 relations, and 9 million entities. After filtering, 1.8 million instances of 102 relations among 940 thousand entities remained. For comparison, the ACE corpora contain about 17 thousand manually annotated instances.
  • The model is a multiclass logistic regression. It combines about 400,000 noisy features collected from all sentences in which an entity pair occurs.
  • Performance was evaluated in two ways: automatically, on a held-out portion of Freebase, and manually, by Amazon Mechanical Turk annotators. Among the top 100 results for each of the 10 most frequent relations, the average precision is 0.67 for syntactic features, 0.66 for lexical features, and 0.69 for both combined.
  • Syntactic features help most with ambiguous relations (director–film, writer–film). They are also useful when the entities in a sentence are separated by many words but are adjacent in the dependency tree.
  • Limitations: evaluation on held-out Freebase data underestimates performance because of false negatives. As future work, the authors mention chunk features instead of full parsing, as well as coreference resolution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Information extraction · 167 worksKnowledge graphs & ontologies · 192 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Natural language processing · 131 worksInformation extraction · 50 works

Cite

Cite as Mike Mintz, Steven Bills, Rion Snow and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mintz2009distant,
  title = {{Distant supervision for relation extraction without labeled data}},
  author = {Mike Mintz, Steven Bills, Rion Snow and Daniel Jurafsky},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  doi = {10.3115/1690219.1690287},
  url = {https://doi.org/10.3115/1690219.1690287}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the 5th ACM International Conference on Digital Libraries. find
  2. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Manuela M Veloso, editor, IJCAI-07, pages 2670– 2676.
  3. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD ’08, pages 1247– 1250, New York, NY. ACM. find
  4. Sergei Brin. 1998. Extracting patterns and relations from the World Wide Web. In Proceedings World Wide Web and Databases International Workshop, Number 1590 in LNCS, pages 172–183. Springer. find
  5. Razvan Bunescu and Raymond Mooney. 2007. Learning to extract relations from the web using minimal supervision. In ACL-07, pages 576–583, Prague, Czech Republic, June.
  6. Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Thomas Lengauer, Reinhard Schneider, Peer Bork, Douglas L. Brutlag, Janice I. Glasgow, Hans W. Mewes, and Ralf Zimmer, editors, ISMB, pages 77–86. AAAI. find
  7. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) Program–Tasks, Data, and Evaluation. LREC-04, pages 837–840.
  8. Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. Artificial Intelligence, 165(1):91–134. find
  9. Jenny R. Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In ACL-05, pages 363–370, Ann Arbor, MI.
  10. Roxana Girju, Adriana Badulescu, and Dan Moldovan. 2003. Learning semantic constraints for the automatic discovery of part-whole relations. In HLT-NAACL-03, pages 1–8, Edmonton, Canada.
  11. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING-92, Nantes, France.
  12. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question-answering. Natural Language Engineering, 7(4):343–360. find
  13. Dekang Lin. 1998. Dependency-based evaluation of minipar. In Workshop on the Evaluation of Parsing Systems. find
  14. Metaweb. 2008. Freebase data dumps. http://download.freebase.com/datadumps/. ↗
  15. Alexander A. Morgan, Lynette Hirschman, Marc Colosimo, Alexander S. Yeh, and Jeff B. Colombe. 2004. Gene name identification and normalization using a model organism database. J. of Biomedical Informatics, 37(6):396–410. find
  16. Patrick Pantel and Marco Pennacchiotti. 2006. Espresso: leveraging generic patterns for automatically harvesting semantic relations. In COL-ING/ACL 2006, pages 113–120, Sydney, Australia.
  17. Marco Pennacchiotti and Patrick Pantel. 2006. A bootstrapping algorithm for automatically harvesting semantic relations. In in Proceedings of Inference in Computational Semantics (ICoS-06), pages 87–96. find
  18. Deepak Ravichandran and Eduard H. Hovy. 2002. Learning surface text patterns for a question answering system. In ACL-02, pages 41–47, Philadelphia, PA.
  19. Ellen Riloff and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI-99, pages 474–479. find
  20. Benjamin Rozenfeld and Ronen Feldman. 2008. Self-supervised relation extraction from the web. Knowledge and Information Systems, 17(1):17–33. find
  21. Yusuke Shinyama and Satoshi Sekine. 2006. Preemptive information extraction using unrestricted relation discovery. In HLT-NAACL-06, pages 304–311, New York, NY.
  22. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Lawrence K. Saul, Yair Weiss, and Léon Bottou, editors, NIPS 17, pages 1297–1304. MIT Press. find
  23. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP 2008, pages 254–263, Honolulu, HI.
  24. Mihai Surdeanu and Massimiliano Ciaramita. 2007. Robust information extraction with perceptrons. In Proceedings of the NIST 2007 Automatic Content Extraction Workshop (ACE07), March. find
  25. Fei Wu and Daniel S. Weld. 2007. Autonomously se-mantifying wikipedia. In CIKM ’07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 41– 50, Lisbon, Portugal. find
  26. Guodong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In ACL-05, pages 427–434, Ann Arbor, MI.
  27. Guodong Zhou, Min Zhang, Donghong Ji, and Qiaoming Zhu. 2007. Tree kernel-based relation extraction with context-sensitive structured parse tree information. In EMNLP/CoNLL 2007. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mintz-2009-distant-supervision-relation-extraction-without/