Elamur Research
☰
home / papers / 2011 / Singletons and Coreference Resolution Evaluation

Singletons and Coreference Resolution Evaluation

What it is about

The paper asks how singletons (mentions that are never coreferent with another mention) change coreference scores. It uses the English data of SemEval-2010 Task 1 and CoNLL-2011, which differ in whether singletons are annotated, and the two official scorers of those tasks. The system is UBIU, a memory-based mention-pair model. Four settings are compared: singletons kept or removed in the gold standard, and kept or removed in the system output. Singletons in the gold standard raise the scores, and the authors show that the metrics behave very differently. MUC ignores singletons; B3, CEAF and BLANC react to them, and the SemEval and CoNLL scorers differ in how they treat singletons in the system output. A trivial baseline that puts every mention in its own chain already reaches high CEAF and B3 scores on the SemEval English data (the paper's Table 1, singleton column: CEAF F1 71.2, B3 F1 83.2, MUC F1 0.0). The conclusion is that scores are not comparable unless the singleton policy is stated and matched.

Ideas

For Mova

Code: coref/src/eval.rs (scorer settings and the baseline rows), coref/src/mention.rs (mention extraction that over-generates NPs). Chapter: languages/en/35-reference-and-coreference.md. Measure: the sieve's CoNLL-style scores under the four singleton settings, with the all-singletons row, run twice to check the output is identical.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/R11-1036.pdf

Summary

Kübler and Zhekova examine how singletons distort the scores used to evaluate coreference resolution. Singletons are mentions that never corefer or that appear only once in a document. They became an issue once shared tasks required systems to detect mentions themselves. The authors compare the English OntoNotes-based data of the SEMEVAL 2010 task 1, which annotates singletons, with the CONLL 2011 shared task data, which does not. They score outputs with both task scorers across MUC, B3, CEAF and BLANC, using a toy example, simple baselines, and their memory-based mention-pair system UBIU. They also vary how singletons, especially pronoun singletons, are filtered from system output, while the actual coreference links stay fixed.

Every metric reacts to singletons in the gold standard. When both gold data and output contain singletons, scores rise artificially. Singletons in the output matter far less, so a good singleton filter is not reliably rewarded. The authors conclude that singletons are useful in training data but inflate scores when kept in evaluation gold standards.

Key points

  • Baselines on SEMEVAL English data:
    • Treating every mention as a singleton gives CEAF F1 71.2 and B3 F1 83.2, but MUC 0.
    • Merging all mentions into one chain gives the best MUC F1 (45.2).
  • Toy example (scorer differences):
    • With the SEMEVAL scorer, a singleton in the system output lowers scores.
    • With the SEMEVAL scorer, an extra gold singleton raises scores even though the output is unchanged.
    • The CONLL scorer ignores output singletons.
  • UBIU setup:
    • Uses TiMBL with IB1, weighted overlap, gain ratio and k = 3.
    • Considers antecedents within a 3-sentence window.
    • Applies a basic feature set, since WordNet and number/gender features did not help.
  • Four output settings tested on CONLL data: AllS, NoS, PronS and AttP (every pronoun forced onto an antecedent).
    • AllS reaches mention recall of 96.55/95.11 but very low precision.
    • Compared with the SEMEVAL scorer, the CONLL scorer raises B3, CEAF_E and BLANC by up to 2.8× in F-score.
  • Training data size: CONLL has 2374 training documents versus 322 for SEMEVAL.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kubler2011singletons,
  title = {{Singletons and Coreference Resolution Evaluation}},
  author = {Kübler, Sandra and Zhekova, Desislava},
  booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing 2011},
  year = {2011},
  url = {https://aclanthology.org/R11-1036/}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, Granada, Spain. find our card
  2. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2007. TiMBL: Tilburg memory based learner – version 6.1 – reference guide. Technical Report ILK 07-07, ILK-CL, Tilburg University. find
  3. Pascal Denis and Jason Baldridge. 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of HLT-NAACL 2007, Rochester, NY. find
  4. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of HLT-NAACL, New York, NY. find
  5. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of ACL, Barcelona, Spain. find
  6. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, Vancouver, Canada. find our card
  7. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL 2011, Portland, OR. find our card
  8. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, Singapore. find
  9. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: A cluster-ranking approach to coreference resolution. JAIR, 40:469–521. find
  10. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering. find
  11. Marta Recasens, Lluı̀s Márquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, Uppsala, Sweden. find
  12. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-AFNLP, Singapore. find our card
  13. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, Columbia, MD. find our card
  14. Holger Wunsch. 2009. Rule-Based and Memory-Based Pronoun Resolution for German: A Comparison and Assessment of Data Sources. Ph.D. thesis, Universität Tübingen. find
  15. Desislava Zhekova and Sandra Kübler. 2010. UBIU: A language-independent system for coreference resolution. In Proceedings of SemEval, Uppsala, Sweden. find
  16. Desislava Zhekova and Sandra Kübler. 2011. UBIU: A Robust System for Resolving Unrestricted Coreference. In Proceedings of CoNLL: Shared Task, Portland, OR. find