📄 Evaluation of named entity coreference

Summary

The paper argues that the usual coreference scores can mislead when the real goal is to collect every reference to a named entity, whether by name, pronoun or nominal, as search or information extraction requires. Metrics such as MUC, B3, CEAFe and LEA ignore mention types. They give full credit to chains of pronouns that never connect to a name, and they barely penalise systems that miss whole entities. The authors propose Named Entity Coreference (NEC) metrics. For each gold chain, the best-matching predicted chain must contain a variant of the entity's name. Its F1 is averaged across gold chains, and unmatched chains score zero. Separate diagnostics report missed entities, accuracy per mention type, and wrongly split or merged entities. The authors apply these metrics to five off-the-shelf systems on OntoNotes PER, ORG and GPE entities. Gains in the standard score do not always carry over to NEC. Lee et al. (2018) is nonetheless clearly the strongest system. The metrics expose errors that matter for downstream use.

Key points

• In a toy example, two outputs receive identical MUC, B3, CEAF and LEA scores. NEC F1 gives the output with named mentions 0.83 and the pronoun-only output 0.
• Named PER, ORG and GPE entities form 38% of OntoNotes coreference clusters but account for 54% of mentions that need resolution. Evaluation used the newswire, broadcast news and magazine portions.
• The systems evaluated were the Stanford deterministic, statistical and neural versions, plus the end-to-end models of Lee et al. (2017, 2018). System chains were kept only if a name was the mention head, which was checked with spaCy dependency parses and gold NER tags.
• Moving from the rule-based to the statistical Stanford system raises Coref F1 for PER (0.50→0.56) while NEC F1 falls (0.55→0.46).
• Lee et al. (2018) reaches a PER NEC F1 of 0.80. It misses 7.5% of PER, 15% of ORG and 8% of GPE chains, while other systems often miss 20–40%.