📄 Description of the UPENN CAMP System as Used for Coreference

Summary

This MUC-7 system description covers two contributions from the University of Pennsylvania team: a critique of coreference evaluation and an update on the CAMP coreference system. The authors walk through the model-theoretic, link-based scorer of Vilain et al. used in MUC-6/7 and identify two weaknesses. It gives no credit for correctly keeping singletons apart, and it penalizes every wrong link equally, even when one error merges far more entities than another. To fix both problems they propose B-CUBED, which computes precision and recall for each entity and then combines them using weights. Equal per-entity weights suit information extraction, while equal per-class weights suit IR or web search.

CAMP was re-implemented after MUC-6 but keeps the same architecture: a pipeline of specialized, mostly high-precision modules, each of which can be scored on its own. The results show that small, focused modules can reach high precision. The authors also observe an apparently linear precision/recall trade-off across systems, so a coreference system can be chosen to match the task.

Key points

• On the paper's toy example, the MUC scorer gives two different erroneous responses the same 90% precision. B-CUBED with equal entity weights separates them: 16/21 (76%) versus 7/12 (58%). With equal class weights the figures are 79.6% versus 75%.
• The pipeline runs in this order: genre-specific rules for oddly formatted text, La Hack for proper nouns (supported by IBM's Textract named entity tool), appositive and predicate-nominal syntax, quoted speech, and CogNIAC for proper nouns, common nouns and pronouns.
• The quoted-speech module reached 2.6% recall and 86.8% precision. CogNIAC common noun resolution reached 7.1% recall and 90.7% precision, up from 10%/48% in MUC-6.
• La Hack did worse on the New York Times data than on the MUC-6 Wall Street Journal corpus. The authors attribute this to less regular use of honorifics and corporate designators.
• CogNIAC pronoun resolution reached 4.5% recall and 70.0% precision, short of the 80% precision target. The authors point to common noun anaphora as a major source of disagreement between annotators.