🧲 Information extraction

About 16 minutes to read · every paper named here has its own page in this library

What it is

Suppose you have a million news articles and want a table of every company acquisition, with buyer, target, price and date. Reading them by hand is impossible, and keyword search only returns documents, not rows of a table. Information extraction (IE) is the branch of natural language processing that fills such tables automatically. It reads sentences and writes out facts in a fixed format. It differs from information retrieval, which finds relevant documents, and from text summarization, which produces shorter prose. The output of IE is data: records, triples, graphs.

IE is usually broken into a set of subtasks that build on one another:

• Named entity recognition finds the spans that name people, organizations, places, dates and other typed things.
• Coreference resolution groups mentions that refer to the same entity, such as "Jane Doe", "Doe" and "she".
• Relation extraction decides which semantic relation, if any, holds between two entity mentions.
• Event extraction finds a word that signals an event (the trigger), assigns an event type, and attaches arguments in named roles.
• Template filling, the scenario-template task of the early evaluations, merges these pieces into a domain record with fixed fields. "Slot filling" names related but distinct tasks. In knowledge base population it means filling attributes of a given entity (Angeli et al. 2015). In spoken-language understanding it means tagging the words of a user query (Goo et al. 2018).
• Entity linking maps each mention to an entry in a knowledge base, which is close to word sense disambiguation applied to names.

Take one invented sentence: "On Monday, Acme Corp. named Jane Doe its chief executive, replacing John Roe, who retired after a decade in Boston." The table shows the records an IE system would aim to produce. The event types follow ACE, while the relation names are illustrative and not taken from a particular schema.

The populated records feed a knowledge graph, a database, a search index or a question answering system. IE is a central tool of text mining and of knowledge representation. The schema of types and slots is usually designed with methods from ontology engineering.

How it works

A classic IE system is a pipeline. Text is tokenized and tagged for part of speech, often parsed with dependency parsing. Entities are then recognized, and candidate pairs or triggers are classified. Coreferent mentions are merged, and the results are mapped onto a template. Each stage passes its output to the next. That makes the system modular, but errors made early cannot be repaired later.

A careful stage-by-stage study of ACE event extraction showed the effect. Each stage could be fed either its own predictions or gold annotation, meaning the human-made reference labels. Replacing gold arguments with predicted ones cost the most. Predicted triggers (Ahn's "anchors") cost the next most, and attributes and coreference mattered far less (Ahn 2006).

Much of the field can be read as a search for ways around this cascade. One approach is joint inference, which decides triggers and arguments together, or entities and relations together, as a single structure (Li et al. 2013; Miwa & Sasaki 2014). A second widens the context beyond the sentence, to the document or to clusters of related documents (Ji & Grishman 2008; Liao & Grishman 2010). A third, more recent, uses generation: a single sequence-to-sequence model writes the whole structure as text (Lu et al. 2021).

Whatever the architecture, systems are scored against human annotation with precision, recall and their harmonic mean:

P  = TP / (TP + FP)
R  = TP / (TP + FN)
F1 = 2 · P · R / (P + R)

TP counts system outputs that match the gold annotation, FP counts system outputs that do not, and FN counts the gold items the system missed. P is the share of the system's output that is correct, R is the share of the gold facts that were found, and F1 balances the two.

The meaning of "match" matters as much as the formula. It may require exact span boundaries, the correct type, correct roles, or the correct full triple. For chemical formulas, strict matching is necessary. "Bi2Te3" and "Bi2Se3" are different compounds, yet string-similarity measures rate them as nearly identical, so fuzzy matching would accept one for the other. The same strictness rejects legitimate variants such as "Si" written for "silicon", which only an expert-judged score accepts (Dagdelen et al. 2024). Scores can also be averaged per class (macro), as on SemEval-2010 Task 8 (Zeng et al. 2014), or pooled over all decisions (micro).

History

The Message Understanding Conferences

Modern IE took shape in the Message Understanding Conferences (MUC), a series of evaluations funded by the US government. Participating systems had to read news or military messages and fill templates about a scenario, such as terrorist incidents or management succession. Grishman and Sundheim's short history of MUC-6 records how the evaluation came to be split into separable tasks: named entity tagging, coreference, template elements and full scenario templates (Grishman & Sundheim 1996). This decomposition is still the basic vocabulary of IE.

Two other MUC habits outlived the series: scoring by precision and recall against an answer key, and shared tasks run on common data. Systems of that era were mostly cascades of finite-state patterns written by hand. Writing patterns for a new domain could take roughly ten person-days (Surdeanu et al. 2003).

ACE, TAC-KBP and the domain challenges

The Automatic Content Extraction (ACE) programme followed MUC. It defined entity, relation and event annotations that became the standard testbeds of the 2000s and 2010s. ACE relation work used 5 top-level relation types and 24 subtypes (Culotta & Sorensen 2004; Zhou et al. 2005). ACE 2005 events used 8 types and 33 subtypes (Li et al. 2013).

The Text Analysis Conference then reframed the goal as knowledge base population. Its tasks asked systems to link names to a Wikipedia-derived knowledge base (Dredze et al. 2010). They also asked systems to fill attribute slots about given entities, as in the TAC-KBP 2013 slot-filling evaluation (Angeli et al. 2015).

Biomedicine ran its own shared tasks. BioNLP'09 moved the field past simple protein-interaction pairs to nested molecular events, and 24 teams submitted final results (Kim et al. 2009).

From patterns to learning to language models

Over three decades, the main approach shifted from handwritten rules to statistical learning with engineered features and kernels. It moved to neural networks around the middle of the 2010s. It then turned to pretrained encoders: BERT appeared in late 2018 and spread through IE during 2019. Most recently it has turned to generative large language models (Xu et al. 2023).

In parallel, the field asked how much human supervision is really needed. Answers ranged from fully labeled corpora to distant supervision from knowledge bases (Mintz et al. 2009) and to open extraction with no fixed schema at all (Banko et al. 2007).

Methods

Patterns and lexico-syntactic rules

The simplest extractors match surface patterns. Hearst (1992) showed that phrases like "X such as Y" or "Y and other X" reliably signal that Y is a kind of X. Hyponym pairs can therefore be harvested from large corpora with little effort (Hearst 1992). These "Hearst patterns" are still used to bootstrap taxonomies and to collect hypernymy pairs for resources in lexical semantics. Pattern systems are precise and transparent, but their recall is limited and they are expensive to port.

Surdeanu and colleagues offered a cheaper route. They mapped automatically labeled predicate-argument structures, of the kind produced by semantic role labeling, onto template slots. The result reached about 90% of a cascaded finite-state system's F-measure after only a couple of hours of mapping per domain (Surdeanu et al. 2003).

Sequence labeling for entities

Entity recognition is usually framed as tagging each token. The common label schemes are BIO (begin, inside, outside) and BILOU, which adds tags for the last token of a name and for single-token names. Conditional random fields (CRFs) dominated this task for years.

Finkel, Grenager and Manning kept a trained CRF and replaced Viterbi decoding with Gibbs sampling. This let them add long-range constraints, for example that repeated names should receive the same label. Against state-of-the-art systems, it gave a relative error reduction of up to 9%. The scores were per-entity F1 on CoNLL 2003 NER and per-token F1 on seminar-announcement templates. The price was much slower inference (Finkel et al. 2005). The paper is the usual citation for the Stanford NER tagger. The tagger's core is a linear-chain CRF, and the Gibbs-sampling layer is an optional extension.

Supervised relation extraction: features and kernels

Supervised relation extraction treats each candidate entity pair as a classification problem. Feature-based systems combine words, entity types, chunks, parse paths and resources such as WordNet. A systematic study on ACE found that shallow chunking supplied most of the syntactic benefit and full parsing added little, because most relations link mentions that stand close together. Of 9,830 training relations, 4,163 had no words at all between the two mentions (Zhou et al. 2005).

Kernel methods avoid explicit feature vectors and compare structures directly. Zelenko, Aone and Richardella defined tree kernels over shallow parses for SVMs and the voted perceptron (Zelenko et al. 2002). Culotta and Sorensen extended the idea to dependency subtrees (Culotta & Sorensen 2004). Bunescu and Mooney argued that the useful syntax is concentrated along the shortest route connecting the entity pair in the dependency graph. Their path kernel outperformed the earlier tree kernel on ACE, with 52.5 against 45.8 F in the two-stage setup (Bunescu & Mooney 2005).

Neural relation classification and joint extraction

Neural models replaced handcrafted features with learned word embeddings. They added position features that encode each token's distance to the two entities. A convolutional network reached 82.7 macro-F1 on SemEval-2010 Task 8 (Zeng et al. 2014). A feature-free CNN gave comparable results and was also applied to the harder, imbalanced ACE setting (Nguyen & Grishman 2015). An attention-based bidirectional LSTM scored 84.0 using word vectors alone (Zhou et al. 2016). Multi-level attention CNNs reported 88.0 (Wang et al. 2016).

Syntax returned in neural form. Graph convolution over pruned dependency trees set a new state of the art on TACRED, the TAC Relation Extraction Dataset, and trained much faster than tree-recursive models (Zhang et al. 2018). Attention-guided GCNs learned a "soft pruning" of the full tree instead of fixed rules (Guo et al. 2019).

End-to-end systems must find the entities themselves. Miwa and Sasaki cast joint extraction as filling a word-by-word table, with entity tags on the diagonal and relation labels elsewhere (Miwa & Sasaki 2014). Miwa and Bansal stacked a tree LSTM over the dependency path on top of a sequence tagger and shared parameters between the two tasks (Miwa & Bansal 2016).

Overlapping triples, where several facts share an entity, motivated GraphRel's relation-weighted graphs (Fu et al. 2019). They also motivated CasRel's per-relation object taggers, which improved F1 by 17.5 points on NYT and 30.2 on WebNLG over the strongest baselines (Wei et al. 2020).

Span-based multi-task models such as DyGIE (Luan et al. 2019) and DyGIE++ (Wadden et al. 2019) enumerate candidate spans and refine them across coreference and relation graphs. They were evaluated on news, scientific and biomedical corpora: ACE, SciERC, the Wet Lab Protocol Corpus, and GENIA, a corpus of annotated biomedical abstracts (Luan et al. 2019; Wadden et al. 2019).

Pretrained encoders then simplified things further. A plain BERT model with typed entity masks reached 67.8 F1 on TACRED (Shi & Lin 2019). A carefully built pipeline, PURE, outperformed joint models that used the same encoders by 1.7–2.8 relation F1 (Zhong & Chen 2020). This reopened the question of whether joint modeling is needed at all.

Event extraction

Event extraction adds triggers and roles to the problem. Joint structured prediction with global features, trained by beam-search perceptron, beat staged classifiers on ACE 2005 (Li et al. 2013). Document-level inference improved scores without new labels. One method enforced consistency across related documents (Ji & Grishman 2008), and another used the co-occurrence of different event types (Liao & Grishman 2010).

Neural approaches followed. DMCNN split max-pooling at the trigger and the argument. This matters because 27.3% of ACE sentences describe more than one event (Chen et al. 2015). JRNN predicted triggers and arguments jointly with a recurrent network, reaching 69.3 trigger F1 and 55.4 argument F1 (Nguyen et al. 2016). OneIE decoded entities, relations and events as one information graph scored with global features (Lin et al. 2020).

Distant supervision and weak signals

Labeling relations by hand is slow. Mintz and colleagues proposed distant supervision. When a knowledge base says two entities are related, sentences mentioning both are treated as noisy positive examples. Relation extractors can then be trained without hand-labeled data (Mintz et al. 2009).

The same trick supplied training data for many later systems. One was a drug–gene–mutation extractor over PubMed, whose cross-sentence extraction roughly tripled the yield (Peng et al. 2017). Another was a large biological relation benchmark built from a curated database (Verga et al. 2018). Automatic data generation was also applied to events, through language-model rewriting of existing examples (Yang et al. 2019).

Open information extraction

Open IE drops the fixed schema. TextRunner, described in 2007, made one pass over about 9 million web pages. It extracted (argument, relation phrase, argument) triples for any relation and made a third fewer errors than KnowItAll at comparable recall (Banko et al. 2007).

WOE trained an open extractor from Wikipedia infoboxes aligned with article sentences. Its parser-based variant improved F-measure over TextRunner by 72–91% but ran roughly 30 times slower (Wu & Weld 2010).

Angeli, Johnson Premkumar and Manning split sentences into short clauses and shortened them with natural logic. This step deletes words only when the original clause still entails the reduced one. A handful of patterns then turned the clauses into triples. Their system extracted about 42.7 million entity-pair triples against about 12.3 million for Ollie, and it improved downstream slot filling (Angeli et al. 2015).

Entity linking

Entity linking resolves names to knowledge-base entries. Dredze and colleagues used high-recall candidate selection followed by a supervised ranker that treats "not in the KB" (NIL) as one more candidate. They reached roughly 80% accuracy on TAC-KBP 2009 against a knowledge base of about 818,000 entries (Dredze et al. 2010).

Babelfy unified linking with word sense disambiguation over BabelNet. It was tested on KORE50, a small test set of short, highly ambiguous sentences. There, partial matching put the right entity into the candidate list in 98.6% of cases, compared with 42.4% for exact string matching (Moro et al. 2014).

BLINK used a BERT bi-encoder for dense retrieval and a cross-encoder for re-ranking. It links against 5.9 million Wikipedia entities in about 2 ms per mention (Wu et al. 2020).

Extraction as reading comprehension and generation

Another line of work recasts IE as question answering. Entity-relation extraction became multi-turn QA, with entities found in earlier turns inserted into later questions (Li et al. 2019). Event arguments became answers to role questions (Du & Cardie 2020). Reading-comprehension pretraining made event extraction work with very little data. With only 1% of the training data, argument F1 was 49.8, compared with 2.2 for a classification baseline (Liu et al. 2020).

Generation removes token-level labeling entirely. Text2Event linearized event records as bracketed strings and trained T5 to produce them, using decoding constrained by the event schema (Lu et al. 2021). A fine-tuned GPT-3 or Llama-2, trained on a few hundred annotated passages, can emit linked materials-science records as JSON (Dagdelen et al. 2024).

A survey of generative IE with large language models reviews the techniques now in use: prompt engineering, fine-tuning and few-shot learning (Xu et al. 2023). These systems build on the Transformer architecture and on pretrained language models.

Datasets and benchmarks

Benchmarks have defined IE as much as methods have. The table groups the most common ones by subtask. The evaluation and benchmarks topic covers measurement issues across NLP.

SemEval-2010 Task 8 contains 10,717 sentences with 9 directed relation types plus an Other class (Zhou et al. 2016). Apart from the Other class, each sentence carries one of the typed relations, so the task is mostly classification rather than detection.

ACE is harder. Fewer than 15% of candidate pairs there express a relation (Culotta & Sorensen 2004), and detection, not type assignment, accounts for most errors (Zhou et al. 2005).

TACRED came out of work on slot filling with position-aware attention and large supervised data (Zhang et al. 2017). It became the main benchmark for sentence-level neural RE (Zhang et al. 2018).

Document-level benchmarks such as DocRED require chaining evidence across sentences. LSR, which learns a latent document graph, reported 59.05 F1 there (Nan et al. 2020). In biomedicine, about 30% of CDR relations cross sentence boundaries (Verga et al. 2018). The generative-IE survey catalogues 50 benchmarks: 32 for NER, 11 for relations and 7 for events (Xu et al. 2023).

Small protocol differences change the numbers. Many event papers assume gold entity mentions. When predicted candidates replace them, argument F1 can fall by about 11 points (Li et al. 2013). Evaluation conventions also vary between papers. One BERT-based event extractor reported its results under a stricter protocol than earlier work, so its scores cannot be compared one-to-one with older tables (Yang et al. 2019).

Relation models can also exploit entity names instead of context. On SemEval, a model shown only the two entities still reached 65.1 development F1. Unmasked models lose much of their score when the entities change (Zhang et al. 2018).

Spoken-language slot filling, as in ATIS and Snips, uses related token-tagging metrics alongside intent accuracy (Goo et al. 2018).

People and institutions

Ralph Grishman, at New York University, links the MUC era to the neural one. He co-wrote the history of MUC-6 (Grishman & Sundheim 1996). He showed how cross-document and cross-event inference improve ACE event extraction (Ji & Grishman 2008; Liao & Grishman 2010). He also co-authored early CNN and RNN extractors (Nguyen & Grishman 2015; Nguyen et al. 2016).

Heng Ji, now at the University of Illinois Urbana-Champaign, worked on cross-document refinement (Ji & Grishman 2008) and on joint structured event extraction (Li et al. 2013). She later co-developed the OneIE graph model, released with English, Spanish and Chinese models (Lin et al. 2020).

Christopher Manning, at Stanford University, co-authored the Gibbs-sampling CRF for non-local constraints (Finkel et al. 2005) and the clause-based open IE system (Angeli et al. 2015). His other IE work includes TACRED-era slot filling (Zhang et al. 2017) and graph convolution over dependency trees (Zhang et al. 2018).

Hannaneh Hajishirzi, at the University of Washington and the Allen Institute for Artificial Intelligence, and Yi Luan built the span-graph DyGIE family. It was tested among others on the SciERC scientific corpus (Luan et al. 2019; Wadden et al. 2019).

Makoto Miwa, at the Toyota Technological Institute in Nagoya, proposed the table-filling and LSTM-based joint entity-relation models (Miwa & Sasaki 2014; Miwa & Bansal 2016). Kang Liu, of the Chinese Academy of Sciences, co-authored three works: the convolutional relation classifier (Zeng et al. 2014), DMCNN (Chen et al. 2015) and event extraction as reading comprehension (Liu et al. 2020).

Zhiyuan Liu, at Tsinghua University, co-authored multi-level attention CNNs (Wang et al. 2016). GuoDong Zhou led the feature-rich ACE relation study (Zhou et al. 2005). Wei Lu co-authored attention-guided GCNs and latent structure refinement for document-level RE (Guo et al. 2019; Nan et al. 2020). Claire Cardie, at Cornell University, co-authored event extraction by question answering (Du & Cardie 2020).

Open IE grew out of Oren Etzioni's group. Its first system was TextRunner, described by Michele Banko and colleagues (Banko et al. 2007). Fei Wu and Daniel Weld later built WOE (Wu & Weld 2010).

Open questions

Pipelines or joint models? The field moved toward joint decoding to stop errors from propagating (Li et al. 2013; Lin et al. 2020). Yet a well-built pipeline with strong encoders beat joint models, and attempts to correct propagated errors did not help much (Zhong & Chen 2020). It remains unclear when the interaction between subtasks really matters.

Do models read the context or memorize the names? Relation classifiers can rely on entity identity (Zhang et al. 2018). This calls into question how far benchmark gains will carry over to new entities and domains.

Beyond the sentence. Many facts are spread across sentences and documents. This holds in biomedicine (Peng et al. 2017; Verga et al. 2018) and in encyclopedic text (Nan et al. 2020). Multi-hop reasoning over long inputs, and reconciling extractions across documents, remain largely unsolved.

Supervision and noise. Distant supervision scales cheaply but labels many sentences that do not express the relation (Mintz et al. 2009). Open IE avoids schemas, but its originators noted the work it leaves unfinished: relation synonymy, argument typing and merging triples into a graph (Banko et al. 2007).

Generative extraction with large language models. Instruction-tuned and fine-tuned language models lower the barrier for domain experts (Dagdelen et al. 2024). They also bring new problems (Xu et al. 2023):

• outputs that do not fit the schema;
• hallucinated facts;
• sensitivity to prompts;
• high compute costs;
• difficulty updating stored knowledge.

Constrained decoding is one partial remedy (Lu et al. 2021).

Evaluation itself. Strict string matching rejects valid variants, while similarity-based matching would accept a different compound with a near-identical formula (Dagdelen et al. 2024). Gold-entity assumptions inflate event scores (Li et al. 2013). Scoring free-form generated records fairly is still an open problem.

So is extending IE to low-resource languages and to multilingual NLP settings. Work there includes low-resource and few-shot generative methods (Xu et al. 2023) and language-independent joint models trained on several languages (Lin et al. 2020).

Further reading is gathered on the separate topic hubs for information extraction and knowledge graphs.

The library in figures