A Model-Theoretic Coreference Scoring Scheme
Summary
The paper defines the scoring metric used for the coreference task of MUC-6, now known as the MUC metric. Coreference chains are viewed as equivalence classes, and scoring counts links rather than individual pairwise decisions. Recall for a key chain is the number of links needed to connect it minus the number of pieces the response splits it into, normalized by the minimal number of links; precision is computed symmetrically with key and response swapped. This makes the score independent of which particular links a system outputs, as long as the resulting partition is the same. The metric is simple and intuitive, but it is known to ignore singleton entities and to reward over-merging, which later metrics (B³, CEAF) were designed to address.
How Mova uses it
coref/src/score.rs: the MUC metric is implemented in Rust as one of the coreference scores, alongside B³, CEAF-e/CEAF-m, and CoNLL F1 (the average of MUC, B³ and CEAF-e).- Mentions are matched by head word (as in the CRAC shared tasks), with and without singletons.
- The scorer is checked against the worked example of Pradhan et al. 2014 (MUC 0.40 / 0.40) and by negative controls: one giant cluster gives MUC recall 1 but CoNLL < 0.7; all-singletons gives MUC 0.
Effectiveness in Mova
It is an evaluation metric, not a modelling idea, so there is no gain to measure. It is a component of the reported results, e.g. on GUM test (UD 2.18, gold trees, with singletons) the default sieve system scores CoNLL F1 73.04 with MUC F1 69.18, B³ 75.26, CEAF-e 74.68.
👨🔬💥
Library summary
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/M95-1005.pdf
Summary
The paper addresses how to score system output for the MUC-6 coreference task. The earlier scorer worked link by link, so its results depended on which particular links annotators happened to write in the key. On some guideline examples it also gave recall values that did not match intuition. The MITRE authors instead compare the equivalence classes that key and response induce, i.e. the models of the IDENT relation. For each key class S, they count how many pieces the response splits it into. Recall is then (|S| − |p(S)|)/(|S| − 1): the share of the minimum set of linking edges that the response actually supplies. Precision mirrors this, with response classes partitioned by the key. Corpus-wide scores sum the numerators and the denominators over all classes. The measure gives the expected scores on every guideline example, including the two the old scorer got wrong. It only needs a union-find pass, not a transitive closure. This scheme became the widely used "MUC score" for coreference evaluation.
Key points
- The core idea is to treat scores as the smallest number of links that must be added to make the classes match. Missing response links count against recall, and links the key would need to cover the response count against precision.
- Each key class S needs |S| − 1 links at minimum, and |p(S)| − 1 of them are missing. Classes are padded with implicit singletons for mentions absent from the other side.
- On the guideline cases, key A-B B-C B-D vs. response A-B C-D gives R = 2/3, P = 2/2. Key A-B B-C vs. response A-C gives R = 1/2, P = 1/1, where the old link-based scorer had failed.
- The paper works through two multi-class examples. One gives R = P = 3/6 (50%), and the other gives R = 2/5 (40%) and P = 2/4 (50%).
- Equivalence classes can be built with Tarjan's UNION-FIND in nearly linear time, which is much cheaper than listing the transitive closure. For now the scheme is defined only for identity links.
Cite
@inproceedings{vilain1995model,
title = {{A Model-Theoretic Coreference Scoring Scheme}},
author = {Vilain, Marc and Burger, John and Aberdeen, John and Connolly, Dennis and Hirschman, Lynette},
booktitle = {Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995},
year = {1995},
doi = {10.3115/1072399.1072405},
url = {https://doi.org/10.3115/1072399.1072405}
}