A Multi-Pass Sieve for Coreference Resolution
Summary
The paper proposes an unsupervised, deterministic coreference system built as a sequence of "sieves": independent rule-based passes applied from the most precise to the least precise. Each pass sees the clusters built by earlier passes, so later, weaker rules can use attributes shared across a whole entity cluster rather than a single mention. Early passes handle high-confidence cases such as exact string match and appositive or predicate-nominative constructions; later passes relax head matching and finally resolve pronouns under agreement constraints. Candidate antecedents are ordered by a syntactic tree traversal in the spirit of Hobbs. The authors report that this simple, modular design was competitive with or better than learned systems of the time on standard benchmarks; it became the basis of the Stanford deterministic coreference system.
How Mova uses it
coref/src/sieve.rs— the overall architecture: an ordered list of sieves from most to least precise (speaker, exact, relaxed, appos, pred, relpron, acronym, reflexive, strict head-match variants, proper_head, pronoun), each attaching only the first mention of its cluster and skipping indefinite mentions except on exact match.- Candidate ordering in the Hobbs style, taken from the description in this paper and its follow-up (Lee et al. 2013): breadth-first left-to-right over the tree, own clause first, then the rest of the sentence, then previous sentences (left-to-right for pronouns, right-to-left for nominals).
- Cluster-level features: a cluster's number, gender, animacy and person are the union of its mentions' features, so pronoun agreement is checked against the whole entity.
- Every link carries the id of the sieve that made it, which
coref explainprints as the reason.
Effectiveness in Mova
Measured on GUM test (UD 2.18), head match, CoNLL F1 with / without singletons, gold trees: the default sieve (12 passes) scores 73.04 / 60.31, versus exact string match only 53.07 / 27.88 and "pronoun → nearest agreeing mention" 57.73 / 32.67; on Mova's own predicted trees 67.90 / 54.03. On dev, Hobbs-style candidate ordering beats pure recency: pronoun precision 64.2% vs 55.4%, CoNLL 73.43 / 63.34 vs 72.23 / 61.33. Two relaxed sieves were disabled because their precision on dev was too low (strict_c 31%, relaxed_head 24%).
👨🔬💥
Library summary
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D10-1048.pdf
Summary
Most coreference systems decide whether two mentions refer to the same entity with one scoring function over many features. As a result, a few highly reliable cues can be drowned out by many weaker ones, and each decision sees only local evidence. The Stanford authors replace this with a "sieve": a sequence of deterministic, unsupervised coreference passes ordered from most to least precise. Each pass extends the clusters built by earlier passes. Mentions in a cluster pool their attributes (number, gender, animacy), so later decisions use information from the whole entity rather than from a single mention pair. Candidate antecedents are ranked by traversing Stanford parser trees. Predicted entity types come from the Stanford NER. The system needs no gold coreference links and no machine learning. On ACE 2004 and MUC-6 benchmarks it matches or beats many supervised and unsupervised systems. Because new passes can be added without retraining or changing existing ones, the design suits rule-based or low-resource pipelines, and the authors suggest using sieves for other information-extraction tasks.
Key points
- Data: ACE2004-ROTH-DEV was used for development (68 docs, 4,536 mentions). Test sets were ACE2004-CULOTTA-TEST (107 docs), ACE2004-NWIRE (128 docs, 11,413 mentions) and MUC6-TEST (30 docs, 2,068 mentions). All experiments use gold mention boundaries.
- Evaluation: pairwise F1, MUC and B³. Gold NE labels are not used; types come from the Stanford NER.
- Error analysis on MUC6-TEST: recall errors outnumber precision errors about eightfold, a consequence of putting precision first. Nominal and proper mentions cause about three times as many errors as pronouns.
- Sampled recall errors: four causes account for 77% of them: longer later mentions, missing semantic knowledge, attribute mismatches and antecedent-ordering mistakes.
- Release: the code is published as Stanford dcoref (nlp.stanford.edu/software/dcoref.shtml).
Cite
@inproceedings{raghunathan2010multi,
title = {{A Multi-Pass Sieve for Coreference Resolution}},
author = {Raghunathan, Karthik and Lee, Heeyoung and Rangarajan, Sudarshan and Chambers, Nathanael and Surdeanu, Mihai and Jurafsky, Dan and Manning, Christopher},
booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
year = {2010},
url = {https://aclanthology.org/D10-1048/}
}References
- B. Amit and B. Baldwin. 1998. Algorithms for scoring coreference chains. In MUC-7. find our card
- E. Bengston and D. Roth. 2008. Understanding the value of features for coreference resolution. In EMNLP. find our card
- S. Bergsma and D. Lin. 2006. Bootstrapping Path-Based Pronoun Resolution. In ACL-COLING. find
- P.F. Brown, V.J. Della Pietra, S.A. Della Pietra, and R.L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19. find
- M. Collins and Y. Singer. 1999. Unsupervised models for named entity classification. In EMNLP-VLC. find
- A. Culotta, M. Wick, R. Hall, and A. McCallum. 2007. First-order probabilistic models for coreference resolution. In NAACL-HLT. find
- M. Elsner and E. Charniak. 2010. The same-head heuristic for coreference. In ACL. find
- J. Finkel, T. Grenager, and C. Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In ACL. find
- J. Finkel and C. Manning. 2008. Enforcing transitivity in coreference resolution. In ACL. find
- B. A. Fox 1993. Discourse structure and anaphora: written and conversational English. Cambridge University Press. find
- J. Ghosh. 2003. Scalable clustering methods for data mining. Handbook of Data Mining, chapter 10, pages 247–277. find
- A. Haghighi and D. Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In EMNLP. find our card
- A. Haghighi and D. Klein. 2010. Coreference resolution in a modular, entity-centered model. In HLT-NAACL. find
- J.R. Hobbs. 1977. Resolving pronoun references. Lingua. find our card
- H. Ji and D. Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person mention detection. In PACLIC. find
- L. Kertz, A. Kehler, and J. Elman. 2006. Grammatical and Coherence-Based Factors in Pronoun Interpretation. In Proceedings of the 28th Annual Conference of the Cognitive Science Society. find
- D. Klein and C. Manning. 2003. Accurate unlexicalized parsing. In ACL. find our card
- X. Luo. 2005. On coreference resolution performance metrics. In HTL-EMNLP. find our card
- H. Poon and P. Domingos. 2008. Joint unsupervised coreference resolution with Markov Logic. In EMNLP. find
- A.S. Schwartz and M.A. Hearst. 2003. A simple algorithm for identifying abbrevation definitions in biomedical text. In Pacific Symposium on Biocomputing. find
- B.F. Skinner. 1938. The behavior of organisms: An experimental analysis. Appleton-Century-Crofts. find
- V.I. Spitkovsky, H. Alshawi, and D. Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In NAACL. find
- V. Stoyanov, N. Gilbert, C. Cardie, and E. Riloff. 2010. Conundrums in noun phrase coreference resolution: making sense of the state-of-the-art. In ACL-IJCNLP. find our card
- M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In MUC-6. find our card