Cross-Document Event Coreference: Annotations, Experiments, and Observations

Summary

Bagga and Baldwin extend their earlier cross-document person coreference system so that it handles events. Given only a coarse event class, such as resignations, elections or espionage, the system groups documents into equivalence classes. Two documents share a class when they describe the same real-world event, meaning the same actors, organization and time. The authors argue this differs from Information Extraction and TDT because the input is minimal: candidate documents are found simply by matching a keyword substring.

CAMP does not build within-document chains for events. So the system creates an event "summary" from each document's sentences that contain the event verb or its nominalizations. Optionally it also uses a few chosen synonyms. A vector space model then compares these summaries and merges those whose similarity exceeds a threshold. Output is scored with both the MUC scorer and the authors' B-CUBED metric. F-measures range from about 45% to 90%. Elections proved hardest because separate elections share places, offices, people and issues. A two-annotator study showed high human agreement, which suggests the task is well defined and gives an upper bound for systems.

Key points

Pipeline: within-document coreference from Penn's CAMP, then a SentenceExtractor that builds per-document summaries, then VSM-Disambiguate, which applies vector space similarity with a threshold.
Data sets: the earlier "John Smith" person data set, plus event data sets for "resign", "elections" and "espionage".
Results: F-measure is roughly 45–90% across data sets and scorers.
- Adding synonym sentences improved the elections results by about 3 F-points.
- One chain about the 1996 US elections covered 36 of the 73 election articles.
Scoring: the authors map cross-document scoring onto a within-document problem using a "meta document". They also introduce the B-CUBED scorer to address weaknesses of the MUC scorer.
Interannotator agreement on elections:
- MUC scorer: 87% precision and recall when annotated independently, rising to 95% after adjudication.
- B-CUBED scorer: 80% independent, 93% after adjudication.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bagga1999cross,
  title = {{Cross-Document Event Coreference: Annotations, Experiments, and Observations}},
  author = {Bagga, Amit and Baldwin, Breck},
  booktitle = {Coreference and Its Applications},
  year = {1999},
  url = {https://aclanthology.org/W99-0201/}
}

References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Bagga, Amit, and Breck Baldwin. Algorithms for Scoring Coreference Chains. Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation, May 1998. find our card
Bagga, Amit, and Breck Baldwin. How Much Processing is Required for Cross-Document Coreference? Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation, May 1998. find
Bagga, Amit, and Breck Baldwin. Entity-Based Cross-Document Coreferencing Using the Vector Space Model. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Con-ferende on Computational Linguistics (COLING-ACL'98), pp. 79-85, August 19… find
Bagga, Amit, and Breck Baldwin. Coreference as the Foundations for Link Analysis Over Free Text Databases. In Proceedings of the COLING-A CL'98 Content Visualization and Intermedia Representations Workshop (CVIR'98), pp. 19-24, August 1998. find
Grishman, Ralph. Whither Written Language Evaluation?, Proceedings of the Human Language Technology Workshop, pp. 120-125, March 1994, San Francisco: Morgan Kaufmann. find
Salton, G e r a r d . AUtomatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer, 1989, Reading, MA: Addison-Wesley. number of correct elements in the output chain containing entity~ number of elements in the output chain containing entityi Precisioni = number… find
John Smith resign elections espionage 197 219 135 184 Figure 7: Analysis of the Data Sets Optimal Threshold 0.15 0.15 0.20 0.20 0.08 0.10 0.25 0.25 Data Set Scorer MUC B-CUBED MUC B-CUBED MUC B-CUBED MUC B-CUBED John Smith resign elections espionage Figure 16: Analysis of the Data Sets avg \# of summ… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bagga-1999-cross-document-event-coreference-annotations/

📄 Cross-Document Event Coreference: Annotations, Experiments, and Observations