📄 TextRunner: Open Information Extraction on the Web

Summary

This short demonstration paper presents TextRunner, a working system for Open Information Extraction (OIE). Classic extraction systems target a fixed list of relations and depend on labelled examples or hand-written patterns. TextRunner instead reads a whole corpus once, with no human input, and collects every relational triple it can find. It chunks sentences into noun phrases and checks nearby pairs of noun phrases with a lightweight classifier. That classifier is trained on examples labelled automatically by running a full parser and applying heuristics to a small sample. A separate component, Resolver, groups strings that refer to the same entity or relation, and an inverted index spread over a cluster supports fast interactive queries. The public demo covers 117 million web pages. On a 9-million-document test set, the system produced 7.8 million well-formed tuples, about 80% of which human judges accepted as correct. It also made fewer errors than the relation-specific KnowItAll system. The work shows that relation-independent extraction can run at Web scale.

Key points

• Extractions are triples of the form (entity, relation, entity), for example (Edison, invented, light bulbs).
• On a 9M-document subset (133 million sentences), extraction used under 68 CPU hours, or about 4 hours on the cluster.
• The test corpus yielded 7.8M well-formed tuples, and 80.4% of a random sample of 400 were judged correct.
• Against KnowItAll on ten frequent relations, both systems found similar numbers of correct extractions (11,631 vs. 11,476). TextRunner's error rate was 12% versus 18%, a one-third reduction.
• Resolver applies an unsupervised probabilistic model to decide whether two strings co-refer, based on the tuples they appear in.

Interesting ideas

• I1. Expensive tool as teacher, cheap model at runtime. A full parser labels training data offline, and a fast POS-based classifier then does the extraction at scale. This pattern could serve any low-resource language where a good parser exists but is too slow for large corpora.
• I2. Heuristic self-labelling instead of manual annotation. Rule-based checks on parsed sentences produce positive and negative examples automatically. This avoids annotation costs and makes it easy to adapt the system to a new domain or language.
• I3. Single-pass design as a hard constraint. Limiting processing to one linear pass, unlike clustering methods that scale quadratically with document count, keeps the system feasible on Web-sized data. This is a useful design principle for large crawls.
• I4. Distributional synonym merging. Two strings are treated as likely synonyms when they appear with similar argument tuples. This turns noisy, open-ended relation phrases into usable clusters without a predefined schema.
• I5. Indexing extractions for querying. Storing the tuples in an inverted index, with optional expansion to synonyms, makes the extracted knowledge immediately browsable. This also helps with quick manual quality checks.