📄 The stages of event extraction

Summary

The paper looks at automatic extraction of ACE 2005 events. These are rich structures: an anchor word, typed arguments, four attributes, and coreference links between mentions. Ahn builds a deliberately simple, modular pipeline that splits the problem into four classification stages: anchor identification, argument identification, attribute assignment and event coreference. Each stage is trained with TiMBL (memory-based) and MegaM (maximum entropy), and MegaM alone is used for coreference because that step needs probabilities. Following the ACE diagnostic setup, gold entities, timexes and values are given as input. Every stage is first evaluated on its own with precision, recall, F-measure or accuracy. Then gold and learned outputs are swapped stage by stage to see how much each one affects the overall ACE value. With all learned components, the system reaches an ACE value of 22.3%, which falls inside the range reported for the 2005 diagnostic task. The ablation shows that argument and anchor errors cost the most, while attributes and coreference matter much less. This gives a clear guide to where research effort pays off.

Key points

• Data: the ACE 2005 training corpus (599 documents), split by document into 539 for training and 60 for testing. Another 60 training documents are held out as a development set for MegaM tuning. Sentences are parsed with the Charniak parser and the parses are converted to dependencies.
• Anchor identification is framed as word-level classification into 34 classes (33 event types plus None). This works because over 95% of anchors are a single word. The class skew is extreme: 197,261 of 202,135 training instances are None, and the largest real class, Conflict:Attack, has only 1,410.
• Modality, polarity and genericity barely beat the majority baseline. For tense, MegaM reaches 0.633 accuracy against a baseline of 0.535. The event coreference pair classifier gets F = 0.658, while a same-type baseline gets 0.286.
• In the ACE value ablation, replacing gold arguments with learned ones costs 35.6 points (81.1% to 45.5%). Learned anchors cost about 21–22 points. Learned attributes cost 4.8 points relative to gold attributes.
• Perfect coreference adds 7.5 points over no coreference when the other stages use gold data (81.1% to 88.6%). The learned coreference classifier adds only 0.6 points in the fully learned pipeline.

Interesting ideas

• I1. Extending gold-input diagnostics to each stage. Feeding every component the gold output of the stages before it shows each step's true difficulty, separate from propagated errors. This could help profile any multi-stage NLP pipeline, such as parsing followed by NER followed by relation extraction.
• I2. A gold/learned swap matrix scored with the end-task metric. Running the full system under many gold/learned/majority combinations shows which stage limits final performance. Dependencies between stages mean not every combination is valid. The result helps decide where to spend annotation or modelling effort.
• I3. Majority-class components as a sanity check. Putting a trivial classifier in a slot can reveal that a "learned" module adds almost nothing to the final score, as happened here with the attributes. This is a cheap test worth running before investing in a component.
• I4. Layered fallback for aligning mentions to parses. The aligner first looks for an exact constituent, then a sequence of constituents, then a word matching the mention's start or end, and finally a word containing the whole mention. This makes independently produced annotations usable with a parser's output, which is common when combining corpora with different tokenisation.
• I5. Coreference features built from argument overlap. Two event mentions are compared through the arguments they share, the arguments they don't share, and shared arguments that appear in different roles. This structural similarity signal could carry over to linking other predicate-argument structures, for example relations or frames.