Error Analysis for Learning-based Coreference Resolution

Summary

Coreference resolvers trained on MUC-7 tend to plateau at around 60% F-measure. This paper asks why, by closely examining the errors of one strong learning-based system. The author's engine uses a large, linguistically motivated feature set covering surface, syntactic, semantic and salience information. It beats the Soon et al. (2001) baseline with several learners and reaches 65.4% F with SVM, which the paper reports as the best published MUC-7 result at the time.

The missed and spurious links on the MUC-7 test set were categorized by hand. The main sources of trouble are:

markables lost during preprocessing, including those in auxiliary document fields;
full-NP (nominal) anaphora that requires inference;
same-sentence and 1st/2nd person pronouns;
the local, pair-by-pair resolution scheme, which spreads one mistake into others.

The author argues that the feature set is already rich but redundant. Progress should therefore come from more training data, better coupling with preprocessing modules, feature selection, and chain-level resolution. Because most comparable systems score similarly, the analysis is presented as relevant to the field as a whole.

Key points

Features: 351 nominal features (1096 boolean/continuous), split into surface (122), syntactic (64), semantic (29) and salience (136). They were tested with SVM, C4.5, Ripper, Slipper and MaxEnt.
SVM results on 20 MUC-7 "formal test" documents: F 65.4%, recall 63.9%, precision 67.0%. The system missed 469 of 1299 gold links and proposed 408 spurious ones. Learning curves had not yet converged.
Recall errors: missing markables account for about 35%, nominal anaphora for 22%, pronouns for 16% and name-matching for about 7%. ORGANIZATION names cause most of the matching failures.
Precision errors: nominal anaphora (mostly same-head NPs) accounts for about 45%, pronouns for 19% and misleading markables for 19%.
Proposed directions: improve training material, integrate preprocessing modules more closely, and move to global chain-based resolution as in Luo et al. (2004).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{uryupina2008error,
  title = {{Error Analysis for Learning-based Coreference Resolution}},
  author = {Uryupina, Olga},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/38d86t8mhth5},
  url = {https://doi.org/10.63317/38d86t8mhth5}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Breck Baldwin, Tom Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Dimitriadis, Kieran Snyder, and Magdalena Wolska. 1997. Description of the UPenn camp system as used for coreference. In Message Understanding Conference Proceedings. find our card
Catalina Barbu, Richard Evans, and Ruslan Mitkov. 2002. A corpus based investigation of morphological disagreement in anaphoric relations. In Proceedings of the Language Resources and Evaluation Conference. find
Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, pages 132–139. find our card
Dan Cristea, Oana Postolache, and Ruslan Mitkov. 2002. Handling complex anaphora resolution cases. In Proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium. find
James R. Curran and Stephen Clark. 2003. Language independent NER using a maximum entropy tagger. In Proceedings of the Seventh Conference on Natural Language Learning, pages 164–167. find
Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. find our card
George Miller. 1990. Wordnet: An on-line lexical database. International Journal of Lexicography. find
Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 104–111. find our card
Massimo Poesio, Olga Uryupina, Renata Vieira, Mijail Alexandrov-Kabadjov, and Rodrigo Goulart. 2004. Discourse-new detectors for definite description resolution: a survey and preliminary proposal. In Proceedings of the Reference Resolution Workshop at ACL’04. find
Jeffrey C. Reynar and Adwait Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing. find
Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics (Special Issue on Computational Anaphora Resolution), 27(4):521–544. find our card
Olga Uryupina. 2006. Coreference resolution with and without linguistic knowledge. In Proceedings of the Language Resources and Evaluation Conference. find our card
Olga Uryupina. 2007. Knowledge Acquisition for Coreference Resolution. Ph.D. thesis, Saarland University. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/uryupina-2008-error-analysis-learning-based-coreference/

📄 Error Analysis for Learning-based Coreference Resolution