Cross-Document Event Coreference: Annotations, Experiments, and Observations
Summary
Bagga and Baldwin extend their earlier cross-document person coreference system so that it handles events. Given only a coarse event class, such as resignations, elections or espionage, the system groups documents into equivalence classes. Two documents share a class when they describe the same real-world event, meaning the same actors, organization and time. The authors argue this differs from Information Extraction and TDT because the input is minimal: candidate documents are found simply by matching a keyword substring.
CAMP does not build within-document chains for events. So the system creates an event "summary" from each document's sentences that contain the event verb or its nominalizations. Optionally it also uses a few chosen synonyms. A vector space model then compares these summaries and merges those whose similarity exceeds a threshold. Output is scored with both the MUC scorer and the authors' B-CUBED metric. F-measures range from about 45% to 90%. Elections proved hardest because separate elections share places, offices, people and issues. A two-annotator study showed high human agreement, which suggests the task is well defined and gives an upper bound for systems.
Key points
- Pipeline: within-document coreference from Penn's CAMP, then a SentenceExtractor that builds per-document summaries, then VSM-Disambiguate, which applies vector space similarity with a threshold.
- Data sets: the earlier "John Smith" person data set, plus event data sets for "resign", "elections" and "espionage".
- Results: F-measure is roughly 45–90% across data sets and scorers.
- Adding synonym sentences improved the elections results by about 3 F-points.
- One chain about the 1996 US elections covered 36 of the 73 election articles.
- Scoring: the authors map cross-document scoring onto a within-document problem using a "meta document". They also introduce the B-CUBED scorer to address weaknesses of the MUC scorer.
- Interannotator agreement on elections:
- MUC scorer: 87% precision and recall when annotated independently, rising to 95% after adjudication.
- B-CUBED scorer: 80% independent, 93% after adjudication.
Cited by
- Detecting Subevent Structure for Event Coreference Resolution
- Unsupervised Event Coreference Resolution with Rich Linguistic Features
- Unsupervised Event Coreference Resolution
- Graph-based Event Coreference Resolution
- A Pairwise Event Coreference Model, Feature Impact and Evaluation for Event Coreference Resolution
- Resolving Event Coreference with Supervised Representation Learning and Clustering-Oriented Regularization
- NPs for Events: Experiments in Coreference Annotation
- Joint Entity and Event Coreference Resolution across Documents
- Supervised Within-Document Event Coreference using Information Propagation
- Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles
- Event Detection and Co-reference with Minimal Supervision
- Revisiting the Evaluation for Cross Document Event Coreference
- A Discriminative Hierarchical Model for Fast Coreference at Large Scale
Cite
@inproceedings{bagga1999cross,
title = {{Cross-Document Event Coreference: Annotations, Experiments, and Observations}},
author = {Bagga, Amit and Baldwin, Breck},
booktitle = {Coreference and Its Applications},
year = {1999},
url = {https://aclanthology.org/W99-0201/}
}References
- Bagga, Amit, and Breck Baldwin. Algorithms for Scoring Coreference Chains. Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation, May 1998. find our card
- Bagga, Amit, and Breck Baldwin. How Much Processing is Required for Cross-Document Coreference? Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation, May 1998. find
- Bagga, Amit, and Breck Baldwin. Entity-Based Cross-Document Coreferencing Using the Vector Space Model. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Con-ferende on Computational Linguistics (COLING-ACL'98), pp. 79-85, August 19… find
- Bagga, Amit, and Breck Baldwin. Coreference as the Foundations for Link Analysis Over Free Text Databases. In Proceedings of the COLING-A CL'98 Content Visualization and Intermedia Representations Workshop (CVIR'98), pp. 19-24, August 1998. find
- Grishman, Ralph. Whither Written Language Evaluation?, Proceedings of the Human Language Technology Workshop, pp. 120-125, March 1994, San Francisco: Morgan Kaufmann. find
- Salton, G e r a r d . AUtomatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer, 1989, Reading, MA: Addison-Wesley. number of correct elements in the output chain containing entity~ number of elements in the output chain containing entityi Precisioni = number… find
- John Smith resign elections espionage 197 219 135 184 Figure 7: Analysis of the Data Sets Optimal Threshold 0.15 0.15 0.20 0.20 0.08 0.10 0.25 0.25 Data Set Scorer MUC B-CUBED MUC B-CUBED MUC B-CUBED MUC B-CUBED John Smith resign elections espionage Figure 16: Analysis of the Data Sets avg \# of summ… find