Distant Supervision for Relation Extraction with an Incomplete Knowledge Base
Summary
Distant supervision trains relation extractors by aligning a knowledge base with raw text. Any entity pair the KB does not list is usually treated as a negative example. The authors show this assumption is badly flawed because KBs such as Freebase are far from complete. Many supposed negatives actually express a target relation in text, and these hidden positives outnumber the real positive pairs several times over.
To handle this, they reframe the task as learning from positive and unlabeled entity-pair bags. Their model, MIML-semi, extends the MIML model of Surdeanu et al. (2012) with an extra layer of latent "true" bag labels that sits between the observed KB labels and the mention-level classifiers. A global constraint fixes the expected share of positive bags, and training uses hard EM.
On the human-assessed KBP dataset, MIML-semi beats MIML, MultiR, and Mintz++ across nearly the whole precision-recall curve. The work matters because it identifies KB incompleteness as a central source of noise in distant supervision and offers a principled way to train despite it.
Key points
- Freebase gaps: For PERSON entities, 93.8% have no place of birth, 78.5% no nationality, and 98.8% no parents recorded.
- False negatives: The authors hand-annotated 200 unlabeled bags each from the Riedel and KBP distant-supervision datasets. About 8.5% (Riedel) and 11.5% (KBP) were false negatives. Given the positive:unlabeled ratios (1:134 and 1:37), false negatives outnumber true positives roughly 11.4× and 4×.
- Model: It has four layers: mention-level classifier, MIML aggregation, latent bag labels ℓ, and observed P/U labels. The parameter θ, the expected fraction of positive bags, is enforced through a Gaussian penalty with a large constant k.
- Experimental setup:
- Data: KBP training bags built by mapping Wikipedia infoboxes onto 1.5M KBP documents plus a Wikipedia snapshot.
- Tuning and evaluation: 40 queries for dev and 160 for evaluation, with θ = 0.25 and T = 8 EM iterations.
- Classifiers: L2-regularized logistic regression from Stanford CoreNLP.
- Result: MIML-semi outperforms MIML, MultiR (Hoffmann), and Mintz++ at almost all recall levels.
Interesting ideas
- I1. Unlabeled is not negative: Model "absent from the KB" as unlabeled rather than negative, with a hidden true label behind it. This applies to any weak labeling from an incomplete resource, such as gazetteers for NER or dictionaries for term extraction.
- I2. Global prior on the positive rate: A single prior on the overall share of positives, tuned on a small dev set, controls how many unlabeled items get flipped to positive. It is a cheap, interpretable way to regularize positive-unlabeled learning.
- I3. Greedy E-step: In each round, the unlabeled pairs the current model scores as most likely positive are relabeled as positive until the target fraction is reached. This is a simple self-training-style step that fits neatly inside hard EM.
- I4. Audit the weak labels first: Before designing a model, hand-annotate a small random sample of auto-generated negatives to measure the false-negative rate. It quickly shows whether label noise is the real bottleneck.
- I5. Evaluate with human judgments: Held-out evaluation against the same incomplete KB penalizes correct extractions that the KB happens to lack. Human-assessed ground truth gives a more honest measurement.
Institutions
New York · United StatesCite
@inproceedings{min2013distant,
title = {{Distant Supervision for Relation Extraction with an Incomplete Knowledge Base}},
author = {Min, Bonan and Grishman, Ralph and Wan, Li and Wang, Chang and Gondek, David},
booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
year = {2013},
url = {https://aclanthology.org/N13-1095/}
}References
- Razvan Bunescu and Raymond Mooney. 2007. Learning to extract relations from the web using minimal supervision. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. find
- Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology. find
- Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
- Heng Ji, Ralph Grishman, and Hoa T. Dang. 2011. Overview of the TAC 2011 knowledge base population track. In Proceedings of the Text Analytics Conference. find
- Jing Jiang and ChengXiang Zhai. 2007. A systematic exploration of the feature space for relation extraction. In Proceedings of HLT-NAACL-2007. find
- Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for information extraction. In Proceedings of ACL-2004. find
- Scott Miller, Heidi Fox, Lance Ramshaw, and Ralph Weischedel. 2000. A novel use of statistical parsing to extract information from text. In Proceedings of NAACL-2000. find
- Bonan Min, Shuming Shi, Ralph Grishman and Chin-Yew Lin. 2012a. Ensemble Semantics for Large-scale Unsupervised Relation Extraction. In Proceedings of EMNLP-CoNLL 2012. find
- Bonan Min, Xiang Li, Ralph Grishman and Ang Sun. 2012b. find
- New York University 2012 System for KBP Slot Filling. In Proceedings of the Text Analysis Conference (TAC) 2012. find
- Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics. find
- Truc Vien T. Nguyen and Alessandro Moschitti. 2011. End-to-end relation extraction using distant supervision from external semantic repositories. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find
- Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 10). find
- Ang Sun, Ralph Grishman, Wei Xu, and Bonan Min. 2011. find
- New York University 2011 system for KBP slot filling. In Proceedings of the Text Analytics Conference. find
- Mihai Surdeanu, Sonal Gupta, John Bauer, David Mc-Closky, Angel X. Chang, Valentin I. Spitkovsky, and Christopher D. Manning. 2011. Stanfords distantly-supervised slot-filling system. In Proceedings of the Text Analytics Conference. find
- Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, Christopher D. Manning. 2012. Multi-instance Multi-label Learning for Relation Extraction. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing and Natural Language Learning. find
- TAC KBP 2011 task definition. 2011. http://nlp.cs.qc.cuny.edu/kbp/2011/KBP2011 TaskDefinition.pdf Shingo Takamatsu, Issei Sato, Hiroshi Nakagawa. 2012. Reducing Wrong Labels in Distant Supervision for Relation Extraction. In Proceedings of 50th Annual Meeting of the Association for Computational Li… ↗
- Fei Wu and Daniel S. Weld. 2007. Autonomously semantifying wikipedia. In Proceedings of the International Conference on Information and Knowledge Management (CIKM-2007). find
- Guodong Zhou, Jian Su, Jie Zhang and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of ACL-2005. find our card
This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2013-distant-supervision-relation-extraction-incomplete/