📄 Using Predicate-Argument Structures for Information Extraction

Summary

The paper asks whether event-level information extraction (IE) can avoid hand-built, domain-specific pattern grammars. The authors propose to build it on automatically recognized predicate-argument structures instead. In their design, a domain is described by two things: the predicates that matter for it and a mapping from those predicates' arguments to templette slots. Argument labeling is learned from PropBank, the Wall Street Journal part of Penn Treebank 2 annotated with verb arguments. The authors replace the backoff-lattice statistical labeler of Gildea and Palmer (2002) with C5 decision trees. These trees can take a much larger feature set, including content words, part-of-speech tags, named-entity information and phrasal-verb statistics. The new labeler beats the statistical one by about 17% F-score, and this improvement carries over to up to 14% better extraction. They test on two HUB Event-99 domains, "market change" and "death". There the predicate-argument approach reaches roughly 90% of a cascaded FSA system's F-measure. It needs only a couple of hours of slot mapping per domain, compared with about ten person-days of pattern writing. This makes IE much cheaper to port to new domains.

Key points

• The argument labeler is trained on PropBank, a one-million-word corpus. Its numbered roles run from Arg0 to Arg5, and it also has ArgM functional tags such as ArgM-LOC, ArgM-TMP and ArgM-DIR.
• The method uses two C5 classifiers, one that finds argument constituents and one that assigns their roles. It reaches over 88% F-measure on identification and over 83% accuracy on role assignment. Predicate lexical features drive fewer than 5% of the tree's branching decisions.
• Templette F-measure, comparing the statistical labeler, the inductive labeler and the FSA system:
  • Market change: 68.9%, 82.8% and 91.3%.
  • Death: 58.4%, 67.0% and 72.7%.
• Correctly matched event structures were 26, 33 and 38 for the three systems, and missed ones were 16, 9 and 4. For the comparison, coreference and templette merging were switched off in both architectures.
• The "death" domain relies on verbs that are rare in training; for example, "die" occurs only 5 times in PropBank. The inductive model still held up there.

Interesting ideas

• I1. Separating language analysis from the domain. A generic semantic-role layer is paired with a thin, per-domain table that maps arguments to slots. This turns domain porting into a small mapping task that could be learned automatically. The same split could work for any extraction schema, such as trade events or geographic relations.
• I2. Choosing a learner that makes feature testing easy. The authors moved away from a backoff lattice, whose structure grows awkward with each added feature. With decision trees they could quickly try many features and see how each one mattered. This is a useful reminder that model choice can speed up research iteration, not only change final accuracy.
• I3. A "content word" next to the syntactic head. Simple rules pick the informative word inside a phrase, for example the noun inside a PP or the verb inside an SBAR or infinitive VP. Heads like prepositions or "that" carry little meaning. The trick could transfer to other parsing-based features where function-word heads hide the semantics.
• I4. Backing off sparse lexical features to coarser ones. The authors noticed that head-word and content-word features were rarely used because of sparsity. Their fix was to add the POS tags of those words and named-entity flags as generalizations. This is cheap and applicable to low-resource settings.
• I5. Evaluating in two kinds of domains on purpose. They compared a domain whose verbs are common in training with one whose verbs are rare. They also reported both slot-level F-measure and counts of matched events. This shows how a component generalizes to unseen predicates, not just its average score.