📄 The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation

Summary

This paper describes the ACE program, a NIST/LDC/BBN effort that began with a 1999 pilot. ACE builds and benchmarks technology for pulling structured content out of language data: the entities mentioned, the relations stated between them, and the events they take part in. Unlike MUC, which scored strings on the page, ACE defines its targets as abstract objects. Systems must therefore find an entity across names, descriptions and pronouns, which makes coreference resolution central.

The paper covers four things:

• Tasks: EDT, RDC, and the newly added VDC for events, along with the supporting LNK coreference annotation.
• Data format: an XML format called "apf."
• Annotation: LDC's multi-pass annotation of English, Chinese and Arabic news, plus ASR and OCR output, with dual annotation used to measure agreement.
• Scoring: a value-based metric.

Inter-annotator agreement is high for entities but much lower for relations, which shows how hard the deeper tasks are. The program matters because it shaped the task definitions, corpora and evaluation practice for later entity, relation and event extraction research.

Key points

• Entities: EDT covers seven entity types: Person, Organization, Location, Facility, Weapon, Vehicle, and GPE. Each type has subtypes. Mentions are labeled as named, nominal or pronominal, with full extent and head marked. Each entity also gets a class, such as specific or generic.
• Relations: RDC uses five relation types (Role, Part, At/Located, Near, Social), which expand to 24 types and subtypes. Annotators separate explicit relations from those that rely on contextual inference, and they exclude relations that depend on world knowledge.
• Events: VDC, planned for the 2004 evaluation, targets five event types: Interaction, Movement, Transfer, Creation, and Destruction. Annotation covers anchors, participant roles (agent, object, source, target) and modality.
• Corpora: The data comes from broadcast news, newswire and newspapers, drawing on sources such as the TDT corpora, Chinese Treebank and Arabic Treebank. Released sets include:
  • ACE-2 (LDC2003T11): 180K training words.
  • ACE 2003 (LDC2004T09): 100K training and 50K evaluation words in English, Chinese and Arabic.
  • ACE 2004: planned at 300K training words.
• Agreement: In 2002, English EDT scored 86 and English RDC only 35. In 2003, English EDT agreement climbed to 88, with Chinese at 87 and Arabic at 74; RDC also improved, reaching 52 in English and 45 in Chinese. These figures come from dual annotation of 5–10% of the data.

Interesting ideas

• I1. Score referents, not strings. Evaluating whether a system found the entity, rather than whether it tagged the right words, makes coreference a built-in part of the task. This could help any extraction benchmark where downstream use depends on knowing who or what is meant.
• I2. Value-based scoring with an optimal mapping. System objects are first aligned to reference objects in the way that maximizes the score. The system then earns credit for correct detections and attributes and loses value for false alarms and attribute errors. A single synthetic "application value" like this can be tuned with weights to reflect what a real user cares about.
• I3. Coreference discount. A mention that is valid but attached to the wrong entity receives a reduced penalty, because the entity-level error has already been charged. This avoids counting the same mistake twice and is worth copying in any hierarchical metric.
• I4. Events as generalized relations. Events are modeled as n-ary relations with role-labeled participants, and the same scoring structure is reused for both. One shared schema and scorer keeps the tasks consistent and makes it easier to add new ones.
• I5. Dual annotation plus adjudication on a small sample. Re-annotating 5–10% of the data from scratch is a cheap way to measure agreement per task and language. In ACE it exposed specific trouble spots, such as metonymy, generic entities, and the explicit-versus-implicit distinction for relations, which then guided guideline revisions.