📄 Using Document Level Cross-Event Inference to Improve Event Extraction

Summary

Most ACE event extraction systems decide about triggers and arguments one sentence at a time, and local context often cannot resolve the ambiguity. For example, "left" can be a Transport event or an End-Position event. Liao and Grishman use evidence from the whole document, including events of other types, not just repeated events of the same type. Their two-stage approach works as follows:

1. A sentence-level MaxEnt baseline tags events.
2. Only high-confidence decisions are kept in a document-level table of confident events, with a separate table for conflicting role assignments.
3. Two extra MaxEnt classifiers, one for triggers and one for arguments, use features describing which other event types and entity roles appear elsewhere in the document. They re-tag words and entity mentions that were left uncertain.

Training used ACE 2005 newswire data. On a blind test set, the method improves F-score by 9.0 points for trigger classification, 9.0 for argument classification and 8.1 for role classification over the baseline. It also clearly beats a reproduction of Ji and Grishman's (2008) within-event-type consistency rules. The work shows that typical co-occurrence of event types is a cheap but strong signal for information extraction.

Key points

• Task: ACE 2005 event extraction with 8 types and 33 subtypes. The subtypes are treated as flat, independent classes, and event mention coreference is ignored.

• Data split:

  • 549 ACE 2005 training documents train both the baseline and the document-level taggers.
  • 10 newswire texts form the development set.
  • 40 newswire texts form the blind test set, following Ji and Grishman (2008).

• Trigger F-score: 59.74 for the baseline, 61.43 with within-event-type rules, and 68.79 with the cross-event model.

• Argument and role F-scores:

  System	Argument F	Role F
  Baseline	41.29	36.46
  Within-event-type rules	47.35	42.21
  Cross-event model	50.28	44.55

  Gains come mostly from recall, though precision also rises.

• Human comparison: two annotators were evaluated on 28 of the test texts.

  • Their trigger F-scores were 59.3 and 72.0, so the system's 68.79 falls between them.
  • Their role F-scores were 55.3 and 62.4, which remain above the system's 44.55.
  • This low agreement highlights how difficult the task is.

Interesting ideas

• I1. Tag easy cases first, then use them as context. Unambiguous triggers such as "appointment" or "founded" can help decide hard ones like "stepped" (down). This pattern suits any document-level labeling task where some decisions are much more certain than others.
• I2. Learn inter-type statistics instead of hand-writing them. Co-occurrence of event types and role correspondences (e.g., an entity filling the Victim role in a Die event frequently also fills the Target role in an Attack) can be estimated from the training corpus and fed in as classifier features. This avoids costly hand-built inference rules and could transfer to relation or frame extraction in other languages.
• I3. Build features from conjunctions. Each feature combines the event type being predicted, another event type in the document, and the role the same entity plays for that other type. Combined with coreference, this lets an easily found argument in one sentence support a hard, distant one elsewhere.
• I4. Keep only confident decisions and track conflicts. Low-confidence tags are filtered out, and a separate table resolves an entity that receives contradictory roles. This limits error propagation when first-pass outputs become features for a second pass.
• I5. Report human agreement next to system scores. Scoring two individual annotators against the adjudicated key gives a realistic ceiling and shows how hard the task is. This is useful when creating new annotated resources, for example Ukrainian event corpora.