Elamur Research⌕
☰
home / papers / 2010 / Coreference Resolution with Reconcile

Coreference Resolution with Reconcile

Summary

Results for noun phrase coreference resolution are hard to compare across systems. Building a complete pipeline is costly, so many studies replace subtasks such as mention detection, anaphoricity determination, named entity recognition or semantic class identification with gold-standard annotations. Benchmarks also differ in task definitions, formats and scoring metrics. Together these factors make published numbers inconsistent and often unrealistic.

The authors present Reconcile, a modular Java platform that follows the standard supervised, pairwise design of learning-based resolvers in the tradition of Soon et al. (2001). Its five stages are preprocessing, feature generation, classification, clustering and scoring. Every stage can be swapped or extended, and the platform handles the format and scoring quirks of the MUC and ACE benchmarks.

As a demonstration, they build one configuration, Reconcile 2010, and test it on six data sets with the MUC and B3 metrics. Its scores are on par with comparable published supervised systems. This gives researchers a realistic, reproducible baseline and gives developers a ready-to-use resolver.

Key points

  • Architecture and components:
    • Preprocessing wraps external tools: OpenNLP, Stanford parser/NER, Berkeley parser and UIUC.
    • In-house NP detectors follow either MUC or ACE definitions and recover about 95% of gold NPs.
    • There are over 80 pairwise features.
    • Learners come from Weka, libSVM and SVM light.
    • Clustering options are Single-link, Best-First and Most Recent First.
    • Scorers are MUC, B3 and CEAF.
    • The code base exceeds 38,000 lines of Java.
  • Data: MUC-6, MUC-7 and four ACE sets (ACE-2, ACE03, ACE04, ACE05, newswire only). Standard splits are used where they exist; otherwise the data is split randomly 70/30.
  • Reconcile 2010 setup:
    • OpenNLP for sentence splitting, tokenization and POS tagging.
    • Berkeley parser and Stanford NER.
    • 60 hand-selected features drawn from Soon et al., Ng and Cardie, and Bengtson and Roth.
    • Averaged Perceptron classifier.
    • Single-link clustering with a threshold tuned by cross-validation.
  • Results:
    • MUC scores: 68.50 (MUC6), 62.80 (MUC7) and 62.03–67.87 on the ACE sets.
    • B3 scores: 70.88 (MUC6), 65.86 (MUC7) and 73.71–79.39 on the ACE sets.
    • B3 is higher than MUC on every data set.
  • Comparison and availability:
    • On MUC6, Reconcile 2010 beats Soon et al. (62.6) but trails Ng and Cardie (70.4) and Yang et al. (71.3).
    • On MUC7, it exceeds Soon et al. (60.4) and Yang et al. (60.2) and is slightly below Ng and Cardie (63.4).
    • The release is available at cs.utah.edu/nlp/reconcile.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition · Coreference resolution

Cited by

30 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stoyanov2010coreference,
  title = {{Coreference Resolution with Reconcile}},
  author = {Stoyanov, Veselin and Cardie, Claire and Gilbert, Nathan and Riloff, Ellen and Buttler, David and Hysom, David},
  booktitle = {Proceedings of the ACL 2010 Conference Short Papers},
  year = {2010},
  url = {https://aclanthology.org/P10-2029/}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Bagga and B. Baldwin. 1998. Algorithms for scoring coreference chains. In Linguistic Coreference Workshop at the Language Resources and Evaluation Conference. find our card
  2. Baldridge, J. 2005. The OpenNLP project. http://opennlp.sourceforge.net/. ↗
  3. E. Bengtson and D. Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  4. CC Group. 2009. Sentence Segmentation Tool. http://l2r.cs.uiuc.edu/ cogcomp/atool.php?tkey=SS. ↗
  5. C. Chang and C. Lin. 2001. LIBSVM: a Library for Support Vector Machines. Available at http://www.csie.ntu.edu.tw/cjlin/libsvm. ↗
  6. J. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL. find
  7. T. Finley and T. Joachims. 2005. Supervised clustering with support vector machines. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). find
  8. A. Haghighi and D. Klein. 2007. Unsupervised Coreference Resolution in a Nonparametric Bayesian Model. In Proceedings of the 45th Annual Meeting of the ACL. find our card
  9. T. Joachims. 2002. SVM Light , http://svmlight.joachims.org. ↗
  10. D. Klein and C. Manning. 2003. Fast Exact Inference with a Factored Model for Natural Language Parsing. In Advances in Neural Information Processing (NIPS 2003). find
  11. Data sets ACE-2 ACE03 65.99 67.87 78.29 79.39 – – – – – – MUC7 62.80 65.86 60.4 63.4 60.2 ACE04 62.03 76.50 – – – ACE05 67.41 73.71 – – – S. Lappin and H. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535–561. find our card
  12. X. Luo, A. Ittycheriah, H. Jing, N. Kambhatla, and S. Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the 42nd Annual Meeting of the ACL. find our card
  13. X. Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP). find our card
  14. A. McCallum and B. Wellner. 2004. Conditional Models of Identity Uncertainty with Application to Noun Coreference. In Advances in Neural Information Processing (NIPS 2004). find
  15. MUC-6. 1995. Coreference Task Definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6). find
  16. MUC-7. 1997. Coreference Task Definition. In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
  17. V. Ng and C. Cardie. 2002. Improving Machine Learning Approaches to Coreference Resolution. In Proceedings of the 40th Annual Meeting of the ACL. find our card
  18. NIST. 2004. The ACE Evaluation Plan. NIST. find
  19. S. Petrov and D. Klein. 2007. Improved Inference for Unlexicalized Parsing. In Proceedings of the Joint Meeting of the Human Language Technology Conference and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2007). find
  20. M. Poesio and M. Kabadjov. 2004. A general-purpose, off-the-shelf anaphora resolution module: implementation and preliminary evaluation. In Proceedings of the Language Resources and Evaluation Conference. find our card
  21. L. Qiu, M.-Y. Kan, and T.-S. Chua. 2004. A public reference implementation of the rap anaphora resolution algorithm. In Proceedings of the Language Resources and Evaluation Conference. find our card
  22. W. Soon, H. Ng, and D. Lim. 2001. A Machine Learning Approach to Coreference of Noun Phrases. Computational Linguistics, 27(4):521–541. find
  23. V. Stoyanov, N. Gilbert, C. Cardie, and E. Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL/IJCNLP. find our card
  24. V. Stoyanov, C. Cardie, N. Gilbert, E. Riloff, D. Buttler, and D. Hysom. 2010. Reconcile: A coreference resolution research platform. Technical report, Cornell University. find
  25. Y. Versley, S. Ponzetto, M. Poesio, V. Eidelman, A. Jern, J. Smith, X. Yang, and A. Moschitti. 2008. BART: A modular toolkit for coreference resolution. In Proceedings of the Language Resources and Evaluation Conference. find our card
  26. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A Model-Theoretic Coreference Scoring Theme. In Proceedings of the Sixth Message Understanding Conference (MUC-6). find
  27. X. Yang, G. Zhou, J. Su, and C. Tan. 2003. Coreference resolution using competition learning approach. In Proceedings of the 41st Annual Meeting of the ACL. find our card

↑ back to top

This page in Markdown →